Certbot
Quick Summary
Openssl 1.0 does not have trusted_first flag set and breaks with most software using it. It fails with LETSENCRYPT certs after Fall 2021 expiration. This is actually an openssl bug.
Openssl 1.1 does have trusted_first flag by default.
CURL
The trusted_first flag has been backported by curl!
Most versions of curl should work with LETSENCRYPT, especially if their servers have updated their LE certbot software and certs.
WGET
wget has NO backport.
Ubuntu
Ubuntu has backport to old versions with openssl 1.0.
RH7 (Centos7)
Redhat has NO backport so fails for much software with openssl 1.0 and LE certs.
kent C programs CGIs and Utilities
I have added the trusted_first flag setting to kent/src/lib/https.c
so we get cert verifications in net.c that work with LE certs even with openssl 1.0.
Other methods that could work:
Go to each server and remove a certain expired LE cert.
This works fine, but a bit of a pain to do for administrators and mirror site operators.
Upgrade to openssl 1.1. Probably have to install from source and recompile.
may have to do install it on many machines, so not fun for administrators. Administrators are understandably reluctant to use direct source install.
Upgrade to Centos8. A bit of a pain, and nobody feels too certain about Centos8 future,
RedHat who controls Centos, got bought by IBM and has more or less discontinued Centos, it gets reduced to developer level stability rather than production level stability.
Upgrade to some other Linux/Unix?
--
Certbot by LETSENCRYPT is a fabulous tool used all over the internet for getting certificates for TLS/SSL encryption, so everybody can have HTTPS on their servers, especially smaller departments or individuals running servers with no IT budget.
They have had some roadbumps recently however, both in their software and in their expired widely used certificcate.
For instance, on a home server I have 5 services (httpd, webmail, imap, smtp, sftp) that all use the same shared LETSENCRYPT certs and they all broke recently due to changes there.
After learning from that upgrade for my home server, I used the information to do a quick and successful upgrade of certbot for Tuatara the HCA project which is now running well with fresh certbot.
Sadly, the folks at LETSENCRYPT (LE) have not made the upgrade particularly easy.
CERTBOT-AUTO DISCONTINUED
First off, they widely promoted that people should use certbot-auto, and now that certbot-auto has been discontinued with no upgrade path. certbot-auto was constructed so that it could update itself automatically, but they ran short of staff and funding and gave up on it. The is not the fault of the LE group, but they did tell everybody on their website to use certbot-auto, the recommended method.
Errors began appearing in logs around the world, often not noticed that certbot-auto was discontinued and that they need find another solution to update certbot.
NEW CERTBOT METHODS
Instead you have to use their new approach which uses Snap containers for easy automatic upgrading and sandboxing.
Snap
They claim Snap is more powerful than Docker. Docker certbot could renew certs, but not hack the apache config files, for example. Unfortunately, snap is not a redhat technology, but it can often be made to work on it.
snap install core snap refresh core
snap install --classic certbot sudo ln -s /snap/bin/certbot /usr/bin/certbot
Hooks
They also have changed the way that it handles hooks, now supporting a nice directory for hooks like git has. Just stick an executable script in here and it gets run whenever new certs were successfully created by renew.
/etc/letsencrypt/renewal-hooks/deploy/
Triggering cert renewal
They do not use cron-jobs any more, they use systemctl timers that call the certbot renew function. This is the default renew setup automatically by snap certbot.
systemctl list-timers
Tue 2021-10-05 10:51:00 PDT 12h left Mon 2021-10-04 13:08:00 PDT 9h ago snap.certbot.renew.timer snap.certbot.renew.service
ISSUES WITH NEW LE CERTS AFTER THEIR FIRST MAJOR CA CERT EXPIRES
Here are some more links on the topic and problems:
So these will fail on hgwdev (and probably the RR too) with the error Verify return code: 10 (certificate has expired):
openssl s_client -connect g-60ce76.a78b8.36fe.data.globus.org:443 -servername g-60ce76.a78b8.36fe.data.globus.org</WRAP>
openssl s_client -connect www.folkplanet.com:443 -servername www.folkplanet.com
But if you add the command line option "-trusted_first" it will work on hgwdev.
openssl s_client -trusted_first -connect g-60ce76.a78b8.36fe.data.globus.org:443 -servername g-60ce76.a78b8.36fe.data.globus.org
openssl s_client -trusted_first -connect www.folkplanet.com:443 -servername www.folkplanet.com
Trusted-first is now the default behavior in openssl 1.1.
Our https.c code now sets the trusted-first flag before validating certs.
QUOTE from LE website:
This means that clients verifying certificate chains can find the alternative non-expired path to the ISRG Root X1 self-signed certificate in their trust store.
Unfortunately this does not apply to OpenSSL 1.0.2 which always prefers the untrusted chain and if that chain contains a path that leads to an expired trusted root certificate (DST Root CA X3), it will be selected for the certificate verification and the expiration will be reported.
So it looks like we have to upgrade to openssl on hgwdev and RR etc in order to be compatible with the LETSENCRYPT certs which are widely used everywhere.
In addition to upgrading certbot and getting fresh certs, I still had two other problems.
IMAP
Dovecot imap service was configured to use whatever security level the OS had, and since the OS had been upgraded to Centos 8.4, this left it with an obscure error in the logs about "dh key too small." That problem was fixed pretty easily be running
openssl dhparam -out /etc/dovecot/dh.pem 4096
then added this line to /etc/dovecot/conf.d/10-ssl.conf:
ssl_dh = </etc/dovecot/dh.pem
Yes, that weird unmatched "<" is the correct syntax.
The Diffie Hellman (dh) is a neat algorithm that lets a two parties exchange information and create a shared secret (key) that they both know. DH is used sometimes instead of RSA, it is faster and has better forward security. It lacks authentication of the parties and man-in-the-middle protections that RSA and certs have. If your exchange is recorded and someday your private key is discovered, RSA is vulnerable. DH can quickly make a new key for each conversation and it does not require storage anywhere secure like a private key.
The reason that it makes sense for imap email exchange to use DH which lacks authentication is because you are going to login by sending your user and password (on an encrypted channel), so the imap server will know who you are from its own protocol.
SENDMAIL
The other issue I ran into was that previously sendmail smtp used a single cert that worked, but it failed with the new LETSENCRYPT cert that wants a fullchain sent. So that means I tweaked sendmail paths for client and server certs:
old now bad fails:
define(`confSERVER_CERT', `CERT_DIR/cert.pem') define(`confCLIENT_CERT', `CERT_DIR/cert.pem')
new good works:
define(`confSERVER_CERT', `CERT_DIR/fullchain.pem') define(`confCLIENT_CERT', `CERT_DIR/fullchain.pem')
Then sendmail was happy too.
LETSENCRYPT explained that they liked the simplicity of configuration for users with the single cert, but for future use, it makes more sense to use fullchain.
BOULDER
Boulder is the name of the software that serves up the LE service. It is the CA or certificate authority for LE. It's software is available on github. You can even run your own site that grants certs to others, and they want to make sure that certificate revocation works since it is an important part of the process.
CERTBOT DOCUMENTATION
Install Instructions Using Certbot DETAILS YOU MAY NEED Uninstall
CERTBOT COMMANDS
Some example certbot commands that are useful.
certbot --version
certbot certificates
Found the following certs: Certificate Name: www.folkplanet.com Serial Number: 368bdeaf9d358db4049a69e927bb387114c Key Type: RSA Domains: www.folkplanet.com folkplanet.com ftp.folkplanet.com imap.folkplanet.com ipv6.folkplanet.com mail.folkplanet.com smtp.folkplanet.com Expiry Date: 2021-11-29 11:05:06+00:00 (VALID: 56 days) Certificate Path: /etc/letsencrypt/live/www.folkplanet.com/fullchain.pem Private Key Path: /etc/letsencrypt/live/www.folkplanet.com/privkey.pem
--certname option is very handy when you have multiple sites you are supporting. Limit the command to the "www.folkplanet.com" site only:
--cert-name www.folkplanet.com
You can during install hope that certbot will correctly recogize your situation, and not give any parameters. However, if you have any complexity at all it is better to just explicitly list the sites and aliases that you want.
I have 4 websites, folkplanet.com, lastingvoices.com, zambra.org and redwooddulcimer.com and each of them have multiple hostname aliases in apache config. I only have email and ftp services on folkplanet.com. certbot did NOT automatically update dovecot and sendmail and my webmail software. Plug-ins for certbot to allow it to handle these could be created, and may exist written by somebody. I put my custom script in the deploy hook mentioned above to restart the servers after a cert renewal takes place. Note that if you do not specify --certname then certbot uses the first domain listed with -d.
CANNOT USE IP ADDRESSES. Your site must have a registered domain. specify -d domainname, it cannot be an IP address. DO NOT USE some ip address like -d 152.274.95.234, because it will fail.
certbot run --cert-name=www.folkplanet.com -d www.folkplanet.com -d folkplanet.com -d ipv6.folkplanet.com -d smtp.folkplanet.com -d imap.folkplanet.com
certbot run --cert-name=www.lastingvoices.com -d www.lastingvoices.com -d lastingvoices.com -d ipv6.lastingvoices.com
certbot run --cert-name=www.zambra.org -d www.zambra.org -d zambra.org -d ipv6.zambra.org
certbot run --cert-name=www.redwooddulcimer.com -d www.redwooddulcimer.com -d redwooddulcimer.com -d ipv6.redwooddulcimer.com
If you have already installed on apache and just need to update the certs, you can use the certonly command instead of run.
certbot certonly
DO NOT run certonly or run commands WITHOUT specifying --cert-name, otherwise certbot can get confused and build a new sub cert with -0001 suffix, but terrible since it is never behavior wanted by anybody, and then you have to try to delete it, and recovering is a pain.
To check if cert expired and renew if they have, run this command.
certbot renew
If you need to force a renewal in order to fully test your system, you can use the --force-renewal option, however, do NOT use this in production as it will renew over and over abusing the LE system.
certbot renew --force-renewal
For detailed documentation on the commands and plugins:
Using Certbot
