Certbot

Certbot by LETSENCRYPT is a fabulous tool used all over the internet for getting certificates for TLS/SSL encryption, so everybody can have HTTPS on their servers, especially smaller departments or individuals running servers with no IT budget.

They have had some roadbumps recently however, both in their software and in their expired widely used certificcate.

For instance, on a home server I have 5 services (httpd, webmail, imap, smtp, sftp) that all use the same shared LETSENCRYPT certs and they all broke recently due to changes there.

After learning from that upgrade for my home server, I used the information to do a quick and successful upgrade of certbot for Tuatara the HCA project which is now running well with fresh certbot.

Sadly, the folks at LETSENCRYPT (LE) have not made the upgrade particularly easy.

CERTBOT-AUTO DISCONTINUED

First off, they widely promoted that people should use certbot-auto, and now that certbot-auto has been discontinued with no upgrade path. certbot-auto was constructed so that it could update itself automatically, but they ran short of staff and funding and gave up on it. The is not the fault of the LE group, but they did tell everybody on their website to use certbot-auto, the recommended method.

Errors began appearing in logs around the world, often not noticed that certbot-auto was discontinued and that they need find another solution to update certbot.

NEW CERTBOT METHODS

Instead you have to use their new approach which uses Snap containers for easy automatic upgrading and sandboxing.

Snap

They claim Snap is more powerful than Docker. Docker certbot could renew certs, but not hack the apache config files, for example. Unfortunately, snap is not a redhat technology, but it can often be made to work on it.

https://snapcraft.io/docs/installing-snap-on-centos

Hooks

They also have changed the way that it handles hooks, now supporting a nice directory for hooks like git has. Just stick an executable script in here and it gets run whenever new certs were successfully created by renew.

/etc/letsencrypt/renewal-hooks/deploy/

Triggering cert renewal

They do not use cron-jobs any more, they use systemctl timers that call the certbot renew function. This is the default renew setup automatically by snap certbot.

systemctl list-timers Tue 2021-10-05 10:51:00 PDT 12h left Mon 2021-10-04 13:08:00 PDT 9h ago snap.certbot.renew.timer snap.certbot.renew.service

ISSUES WITH NEW LE CERTS AFTER THEIR FIRST MAJOR CA CERT EXPIRES

Here are some more links on the topic and problems:

LetsEncryptRootCertExpire

Expiration-september-2021

Certificate-compatibility

unable-to-verify-the-first-certificate

Your-system-is-not-supported-by-certbot-auto-anymore

certbot-auto-deprecated-explanation-and-solutions

So these will fail on hgwdev (and probably the RR too) with the error Verify return code: 10 (certificate has expired):

openssl s_client -connect g-60ce76.a78b8.36fe.data.globus.org:443 -servername g-60ce76.a78b8.36fe.data.globus.org</WRAP>

openssl s_client -connect www.folkplanet.com:443 -servername www.folkplanet.com

But if you add the command line option "-trusted_first" it will work on hgwdev.

openssl s_client -trusted_first -connect g-60ce76.a78b8.36fe.data.globus.org:443 -servername g-60ce76.a78b8.36fe.data.globus.org

openssl s_client -trusted_first -connect www.folkplanet.com:443 -servername www.folkplanet.com

Trusted-first is now the default behavior in openssl 1.1.

QUOTE from LE website:

---

This means that clients verifying certificate chains can find the alternative non-expired path to the ISRG Root X1 self-signed certificate in their trust store.

Unfortunately this does not apply to OpenSSL 1.0.2 which always prefers the untrusted chain and if that chain contains a path that leads to an expired trusted root certificate (DST Root CA X3), it will be selected for the certificate verification and the expiration will be reported.

---

So it looks like we have to upgrade to openssl on hgwdev and RR etc in order to be compatible with the LETSENcrypt certs which are widely used everywhere.

In addition to upgrading certbot and getting fresh certs, I still had two other problems.

IMAP

Dovecot imap service was configured to use whatever security level the OS had, and since the OS had been upgraded to Centos 8.4, this left it with an obscure error in the logs about "dh key too small." That problem was fixed pretty easily be running

openssl dhparam -out /etc/dovecot/dh.pem 4096

then added this line to /etc/dovecot/conf.d/10-ssl.conf:

ssl_dh = </etc/dovecot/dh.pem

(yes that weird unmatched "<" is the correct syntax.)

The Diffie Hellman (dh) is a neat algorithm that lets a two parties exchange information and create a shared secret (key) that they both know. DH is used sometimes instead of RSA, it is faster and has better forward security. It lacks authentication of the parties and man-in-the-middle protections that RSA and certs have. If your exchange is recorded and someday your private key is discovered, RSA is vulnerable. DH can quickly make a new key for each conversation and it does not require storage anywhere secure like a private key.

The reason that it makes sense for imap email exchange to use DH which lacks authentication is because you are going to login by sending your user and password (on an encrypted channel), so the imap server will know who you are from its own protocol.

SENDMAIL The other issue I ran into was that previously sendmail smtp used a single cert that worked, but it failed with the new LETSENCRYPT cert that wants a fullchain sent. So that means I tweaked sendmail paths for client and server certs:

old now bad fails:

define(`confSERVER_CERT', `CERT_DIR/cert.pem')
define(`confCLIENT_CERT', `CERT_DIR/cert.pem')

new good works:

define(`confSERVER_CERT', `CERT_DIR/fullchain.pem')
define(`confCLIENT_CERT', `CERT_DIR/fullchain.pem')

Then sendmail was happy too. LETSENCRYPT explained that they liked the simplicity of configuration for users with the single cert, but for future use, it makes more sense to use fullchain.

BOULDER

Boulder is the name of the software that serves up the LE service. It is the CA or certificate authority for LE. It's software is available on github. You can even run your own site that grants certs to others, and they want to make sure that certificate revocation works since it is an important part of the process.

CERTBOT DOCUMENTATION

https://certbot.eff.org/docs/using.html

CERTBOT COMMANDS

Some example certbot commands that are useful.

certbot --version

certbot certificates Found the following certs:

 Certificate Name: www.folkplanet.com
   Serial Number: 368bdeaf9d358db4049a69e927bb387114c
   Key Type: RSA
   Domains: www.folkplanet.com folkplanet.com ftp.folkplanet.com imap.folkplanet.com ipv6.folkplanet.com mail.folkplanet.com smtp.folkplanet.com
   Expiry Date: 2021-11-29 11:05:06+00:00 (VALID: 56 days)
   Certificate Path: /etc/letsencrypt/live/www.folkplanet.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/www.folkplanet.com/privkey.pem

--certname option is very handy when you have multiple sites you are supporting.

--cert-name www.folkplanet.com # limit the command to the "www.folkplanet.com" site only.

You can during install hope that certbot will correctly recogize your situation, and not give any parameters. However, if you have any complexity at all it is better to just explicitly list the sites and aliases that you want.

I have 4 websites, folkplanet.com, lastingvoices.com, zambra.org and redwooddulcimer.com and each of them have multiple hostname aliases in apache config. I only have email and ftp services on folkplanet.com. certbot did NOT automatically update dovecot and sendmail and my webmail software. Plug-ins for certbot to allow it to handle these could be created, and may exist written by somebody. I put my custom script in the deploy hook mentioned above to restart the servers after a cert renewal takes place.

certbot run --cert-name=www.folkplanet.com -d www.folkplanet.com -d folkplanet.com -d ipv6.folkplanet.com -d smtp.folkplanet.com -d imap.folkplanet.com

certbot run --cert-name=www.lastingvoices.com -d www.lastingvoices.com -d lastingvoices.com -d ipv6.lastingvoices.com

certbot run --cert-name=www.zambra.org -d www.zambra.org -d zambra.org -d ipv6.zambra.org

certbot run --cert-name=www.redwooddulcimer.com -d www.redwooddulcimer.com -d redwooddulcimer.com -d ipv6.redwooddulcimer.com

If you have already installed on apache and just need to update the certs, you can use the certonly command instead of run.

certbot certonly

To check if cert expired and renew if they have, run this command.

certbot renew

If you need to force a renewal in order to fully test your system, you can use the --force-renewal option, however, do NOT use this in production as it will renew over and over abusing the LE system.

certbot renew --force-renewal