By default, Spacewalk creates and uses self-signed certificates for its SSL configuration. This requires the end-user to accept and trust the CA certificiate created by Spacewalk in each browser. This post details how to replace the self-signed certificates with CA-signed certificates from Let’s Encrypt.

Prerequisites

Before you begin, ensure your Spacewalk server is subscribed to the appropriate EPEL channel, either directly or via a Spacewalk-delivered Software Channel.

We will use the certbot utility that is provided by EPEL to obtain and renew certificates from Let’s Encrypt.

This process requires that your Spacewalk server is accessible via the Internet on HTTP (port 80). Let’s Encrypt uses HTTP to validate the request for a certificate. If you cannot connect your Spacewalk instance to the Internet, review alternative methods for obtaining a certificate using DNS.

DNS configuration

As Let’s Encrypt uses HTTP by default to validate the certificate request, ensure that the fully-qualified domain name (FQDN) of the Spacewalk server is resolvable on the Internet and can be used to connect to your Spacewalk instance remotely.

Step 1: Obtain the certificate

First, install the certbot tool:

1

# yum install certbot

Next, use certbot to request a certificate for your Spacewalk server’s FQDN:

1

# certbot certonly -n --webroot -w /var/www/html -d spacewalk.domain.com

Replace spacewalk.domain.com with the actual FQDN of your server. The certbot tool will create an HTML file in /var/www/html/.well-known/ and then attempt to connect to http://spacewalk.domain.com/.well-known/ to retrieve the content of that file. This verifies that the request came from spacewalk.domain.com.

This will fail if:

Let’s Encrypt’s servers cannot resolve spacewalk.domain.com using DNS.
Let’s Encrypt’s servers cannot connect to spacewalk.domain.com via HTTP.

Assuming it succeeds, certbot will create certificates in /etc/letsencrypt/live for your server. We will use those certificates to configure Spacewalk.

Step 2: Setup automatic renewal of certificates

Let’s Encrypt creates certificates that are only valid for 90 days. We need to enable a cron job or systemd timer to check for certificate expiry on a regular basis and renew the certificate before it expires.

Let’s Encrypt recommends checking for expiry twice daily.

Create a script that triggers on renew

jabberd requires a full chain certificate that includes the private key as well as the public key chain. This is not created automatically by certbot so we need a script that runs on renewal to create it automatically.

Edit /usr/local/bin/certbot-renew.sh and add the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#!/bin/bash
LE_LIVE="/etc/letsencrypt/live"
FQDN="spacewalk.domain.com"
EMAIL="your@email.com"

/sbin/spacewalk-service stop >/dev/null 2>&1
cat ${LE_LIVE}/${FQDN}/fullchain.pem ${LE_LIVE}/${FQDN}/privkey.pem > /etc/pki/spacewalk/jabberd/server.pem
/sbin/spacewalk-service start >/dev/null 2>&1

echo "Let's Encrypt certificate renewed on: $(date)" | mailx -s "Certificate Renewal" ${EMAIL}

And then make it executable by running:

# chmod +x /usr/local/bin/certbot-renew.sh

We will configure certbot to run this script whenever the certificates are successfully renewed.

Using Cron (Oracle Linux 6 or 7)

A sample cron script to check for renewals:

43 0,12 * * * certbot renew --renew-hook "/usr/local/bin/certbot-renew.sh"

This job will run twice a day and if the certificate is renewed, will restart all the Spacewalk services. No action is taken if the certificate is not renewed.

Using Systemd (Oracle Linux 7)

The latest certbot package ships with a systemd timer that can be used to automatically renew the certificate.

First, enable and start the timer:

1
2


# systemctl enable certbot-renew.timer
# systemctl start certbot-renew.timer

You can view the status of the timer as well:

1
2
3
4
5
6
7


# systemctl status certbot-renew.timer
* certbot-renew.timer - This is the timer to set the schedule for automated renewals
   Loaded: loaded (/usr/lib/systemd/system/certbot-renew.timer; enabled; vendor preset: disabled)
   Active: active (waiting) since Fri 2017-04-07 08:49:44 AEST; 55min ago

Apr 07 08:49:44 spacewalk.domain.com systemd[1]: Started This is the timer to set the schedule for automated renewals.
Apr 07 08:49:44 spacewalk.domain.com systemd[1]: Starting This is the timer to set the schedule for automated renewals.

You should then edit /etc/sysconfig/certbot to restart Spacewalk if the certificate is renewed by the timer. Add the following command to the RENEW_HOOK directive:

1

RENEW_HOOK="--renew-hook '/usr/local/bin/certbot-renew.sh'"

Step 3: Build the CA Certificate Chain

While the certbot tool will create and renew the client-side certificates, it doesn’t automatically save the full CA chain. This is not an issue for standard HTTPS sites, as the chain is embedded in most browsers. However, Spacewalk distributes the CA chain to all clients so that the Spacewalk client software can verify the certificate presented by the browser.

To build the Let’s Encrypt CA chain, we need two different certificates.

IdenTrust DST Root CA X3

Copy and paste the IdenTrust DST Root CA X3 certificate into a text file from https://www.identrust.com/certificates/trustid/root-download-x3.html and save it locally on your Spacewalk server.

You should add -----BEGIN CERTIFICATE----- at the top of the file and -----END CERTIFICATE----- at the end of the file so that it looks like this:

Next, grab the [Let’s Encrypt Authority X3 (IdenTrust cross-signed)] (https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt) from https://letsencrypt.org/certificates/ and save that to a text file too:

Finally, combine the two text files to create the chain, with the IdenTrust certificate first and the Let’s Encrypt certificate second. The final file should look like this:

You should save that as /root/ca-chain.pem on your Spacewalk server.

You can then validate the chain using the openssl tool:

1
2


# openssl verify -CAfile /root/ca-chain.pem /etc/letsencrypt/live/spacewalk.domain.com/fullchain.pem
/etc/letsencrypt/live/spacewalk.domain.com/fullchain.pem: OK

If you get the OK, you’re OK to reconfigure Spcaewalk.

Step 4: Update ssl-build with the Let’s Encrypt Certificate

First, backup the SSL configuration currently in use by Spacewalk:

1
2
3
4
5


tar -cvf SSLconfig.tar \
   /etc/httpd/conf/ssl.* \
   /etc/pki/spacewalk/jabberd/server.pem \
   /root/ssl-build \
   /var/www/html/pub

Then, replace the self-signed certificates created by Spacewalk with links to the live Let’s Encrypt certificates.

In the following commands, replace <hostname> with the hostname of your Spacewalk server and <fqdn> with the fully-qualified domain name of your Spacewalk server.

# cd /root/ssl-build/<hostname>/
# mv server.crt server.crt.self-signed
# mv server.key server.key.self-signed
# ln -s /etc/letsencrypt/live/<fqdn>/fullchain.pem server.crt
# ln -s /etc/letsencrypt/live/<fqdn>/privkey.pem server.key

Next, replace the existing CA certificate with the Let’s Encrypt chain:

1

# cp /root/ca-chain.pem /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT

You should now be able to validate the certificate in-place using the same openssl command we used before:

1
2


# openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/<hostname>/server.crt
/root/ssl-build/<hostname>/server.crt: OK

If you get the OK, it’s time to tell Spacewalk about your new certificate.

Step 5: Add the CA to Spacewalk and create the certificate RPMs

We need to add the CA certificate to the Spacewalk database so that it is automatically installed on provisioned systems during kickstart:

1
2


# rhn-ssl-dbstore -v --ca-cert=/root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
Public CA SSL certificate:  /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT

Generate the web server SSL package

Take care to note the actual filename of the RPMs created. You’ll need to use that filename in subsequent commands.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# rhn-ssl-tool --gen-server --rpm-only --dir /root/ssl-build

...working...

Generating web server's SSL key pair/set RPM:
    /root/ssl-build/swksvr/rhn-org-httpd-ssl-key-pair-spacewalk-1.0-rev.src.rpm
    /root/ssl-build/swksvr/rhn-org-httpd-ssl-key-pair-spacewalk-1.0-rev.noarch.rpm

The most current Spacewalk Proxy Server installation process against RHN hosted
requires the upload of an SSL tar archive that contains the CA SSL public
certificate and the web server's key set.

Generating the web server's SSL key set and CA SSL public certificate archive:
    /root/ssl-build/swksvr/rhn-org-httpd-ssl-archive-swksvr-1.0-rev.tar

Deploy the server's SSL key pair/set RPM:
    (NOTE: the Red Hat Satellite or Proxy installers may do this step for you.)
    The "noarch" RPM needs to be deployed to the machine working as a
    web server, or Red Hat Satellite, or Spacewalk Proxy.
    Presumably 'spacwalk.domain.com'.

Then, install it:

1

# rpm -Uvh /root/ssl-build/swksvr/rhn-org-httpd-ssl-key-pair-spacewalk-1.0-rev.noarch.rpm

Generate the CA certificate package

Take care to note the actual filename of the RPMs created. You’ll need to use that filename in subsequent commands.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# rhn-ssl-tool --gen-ca --dir=/root/ssl-build --rpm-only

...working...
Generating CA public certificate RPM:
    /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-rev.src.rpm
    /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-rev.noarch.rpm

Make the public CA certificate publicly available:
    (NOTE: the Red Hat Satellite or Proxy installers may do this step for you.)
    The "noarch" RPM and raw CA certificate can be made publicly accessible
    by copying it to the /var/www/html/pub directory of your Red Hat Satellite or
    Proxy server.

Then, install it:

1

# rpm -Uvh /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-rev.noarch.rpm

Make the CA certificates available to clients

We need to copy the CA certificate and the CA RPM to /var/www/html/pub so that they’re accessible to client machines.

1
2


# cp /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-rev.noarch.rpm /var/www/html/pub
# cp /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /var/www/html/pub

Step 6: Link Let’s Encrypt Certificates

The Let’s Encrypt certificates are updated every 90 days by the renew timer. We don’t want to have to manually build a new rhn-org-httpd-ssl-key-pair-spacewalk RPM every time they’re updated, so let’s just create symlinks instead.

First, move the old Apache/httpd certificates and keys out of the way

1
2


# mv /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.self-signed
# mv /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.self-signed

Then, link the live Let’s Encrypt certificates in their place

1
2


# ln -s /etc/letsencrypt/live/<fqdn>/privkey.pem /etc/httpd/conf/ssl.key/server.key
# ln -s /etc/letsencrypt/live/<fqdn>/fullchain.pem /etc/httpd/conf/ssl.crt/server.crt

Finally, move and link the jabberd certificate

1
2


# mv /etc/pki/spacewalk/jabberd/server.pem /etc/pki/spacewalk/jabberd/server.pem.self-signed
# ln -s /etc/letsencrypt/live/<fqdn>/fullchain.pem /etc/pki/spacewalk/jabberd/server.pem

Now all the services will update their certificates automatically when the certbot renewal timer fires the renew-hook on a successful renewal.

Step 7: Restart Spacewalk

This is pretty simple:

1

# spacewalk-service restart

At this point, you should be able to visit https://spacewalk.domain.com and see the Let’s Encrypt certificate trusted by your browser.

Step 8: Install the updated CA on your clients

Remember to use the actual filenames of the RPMs created in the previous steps and the actual FQDN of your Spacewalk server.

In Step 5, we copied rhn-org-trusted-ssl-cert-1.0-rev.noarch.rpm to /var/www/html/pub so it will be available at http://spacewalk.domain.com/pub/rhn-org-trusted-ssl-cert-1.0-rev.noarch.rpm. You can use that URL to install it on each client.

However, as the SSL certificate has already changed, yum on the client will not be able to connect to Spacewalk until the new CA has been installed.

Therefore, we have to update the RPM manually:

1

# yum --noplugins -y localinstall http://spacewalk.domain.com/pub/rhn-org-trusted-ssl-cert-1.0-rev.noarch.rpm

This will disable all plugins (including the Spacewalk client) to bypass the connection error. It will then grab the new RPM from your server and install it, replacing the old certificate. At this point, yum should resume working.

Finally, restart osad (if enabled) using service (OL6):

1

# service osad restart

or systemctl (OL7):

1

# systemctl restart osad

Using Let's Encrypt SSL Certificates with Spacewalk