On September 30, 2021, the Let’s Encrypt DST Root CA X3 certificate expired. This certificate was used to sign all SSL certificates generated by Let’s Encrypt.
To avoid serious problems, Let’s Encrypt created a new ISRG Root X1 certificate that should be used from there on to generate certificates.
Renewing SSL Certificates by forcing ISRG Root X1
Trying to renew the certificates by forcing the use of the new Let’s Encrypt ISRG Root X1 did not seem to work:
certbot --force-renewal --preferred-chain "ISRG Root X1" renew usage: certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ... Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. certbot: error: unrecognized arguments: --preferred-chain ISRG Root X1
Install correct version of Certbot
The version of Certbot I was using was already a bit outdated, and did not support the --preferred-chain argument, so it had to be replaced with a newer version. The EFF (Electronic Frontier Foundation) recommends using the Snap version:
sudo snap install --classic certbot mv /usr/bin/certbot /usr/bin/certbot-old sudo ln -s /snap/bin/certbot /usr/bin/certbot
I then tried to renew the certificate again:
certbot --force-renewal --preferred-chain "ISRG Root X1" renew Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/mail.DOMAIN.com.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Renewing an existing certificate for mail.DOMAIN.com and 4 more domains - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Congratulations, all renewals succeeded: /etc/letsencrypt/live/mail.DOMAIN.com/fullchain.pem (success) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No problems this time!
Implementing new certificates in Zimbra
In my case, I use a script called Certbot-Zimbra that automates the process of deploying certificates in Zimbra, since doing this manually is a tedious process, and it is very easy to make mistakes that later become headaches.
root@mail:/home/mqc/certbot-zimbra# ./certbot_zimbra.sh -d -H mail.DOMAIN.com
certbot-zimbra v0.7.12 - https://github.com/YetOpen/certbot-zimbra
Checking for dependencies...
Detected Zimbra 8.8.15 on UBUNTU16_64
Using domain mail.DOMAIN.com (as certificate DN)
Preparing certificates for deployment.
Testing with zmcertmgr.
** Verifying '/run/certbot-zimbra/certs-MXMcy89c/cert.pem' against '/run/certbot-zimbra/certs-MXMcy89c/privkey.pem'
Certificate '/run/certbot-zimbra/certs-MXMcy89c/cert.pem' and private key '/run/certbot-zimbra/certs-MXMcy89c/privkey.pem' match.
** Verifying '/run/certbot-zimbra/certs-MXMcy89c/cert.pem' against '/run/certbot-zimbra/certs-MXMcy89c/zimbra_chain.pem'
Valid certificate chain: /run/certbot-zimbra/certs-MXMcy89c/cert.pem: OK
Deploying certificates.
** Verifying '/run/certbot-zimbra/certs-MXMcy89c/cert.pem' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate '/run/certbot-zimbra/certs-MXMcy89c/cert.pem' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying '/run/certbot-zimbra/certs-MXMcy89c/cert.pem' against '/run/certbot-zimbra/certs-MXMcy89c/zimbra_chain.pem'
Valid certificate chain: /run/certbot-zimbra/certs-MXMcy89c/cert.pem: OK
** Copying '/run/certbot-zimbra/certs-MXMcy89c/cert.pem' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Copying '/run/certbot-zimbra/certs-MXMcy89c/zimbra_chain.pem' to '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
** Appending ca chain '/run/certbot-zimbra/certs-MXMcy89c/zimbra_chain.pem' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Importing cert '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt' as 'zcs-user-commercial_ca' into cacerts '/opt/zimbra/common/lib/jvm/java/lib/security/cacerts'
** NOTE: restart mailboxd to use the imported certificate.
** Saving config key 'zimbraSSLCertificate' via zmprov modifyServer mail.DOMAIN.com...ok
** Saving config key 'zimbraSSLPrivateKey' via zmprov modifyServer mail.DOMAIN.com...ok
** Installing imapd certificate '/opt/zimbra/conf/imapd.crt' and key '/opt/zimbra/conf/imapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/imapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/imapd.key'
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/conf/imapd.keystore'
** Installing ldap certificate '/opt/zimbra/conf/slapd.crt' and key '/opt/zimbra/conf/slapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/slapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/slapd.key'
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/mailboxd/etc/keystore'
** Installing mta certificate '/opt/zimbra/conf/smtpd.crt' and key '/opt/zimbra/conf/smtpd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/smtpd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/smtpd.key'
** Installing proxy certificate '/opt/zimbra/conf/nginx.crt' and key '/opt/zimbra/conf/nginx.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/nginx.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/nginx.key'
** NOTE: restart services to use the new certificates.
** Cleaning up 7 files from '/opt/zimbra/conf/ca'
** Removing /opt/zimbra/conf/ca/4042bcee.0
** Removing /opt/zimbra/conf/ca/8d33f237.0
** Removing /opt/zimbra/conf/ca/commercial_ca_2.crt
** Removing /opt/zimbra/conf/ca/commercial_ca_1.crt
** Removing /opt/zimbra/conf/ca/ca.key
** Removing /opt/zimbra/conf/ca/a6222139.0
** Removing /opt/zimbra/conf/ca/ca.pem
** Copying CA to /opt/zimbra/conf/ca
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.key' to '/opt/zimbra/conf/ca/ca.key'
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.pem' to '/opt/zimbra/conf/ca/ca.pem'
** Creating CA hash symlink 'a6222139.0' -> 'ca.pem'
** Creating /opt/zimbra/conf/ca/commercial_ca_1.crt
** Creating CA hash symlink '8d33f237.0' -> 'commercial_ca_1.crt'
** Creating /opt/zimbra/conf/ca/commercial_ca_2.crt
** Creating CA hash symlink '4042bcee.0' -> 'commercial_ca_2.crt'
Removing temporary files in /run/certbot-zimbra/certs-MXMcy89c
Restarting Zimbra.
Host mail.DOMAIN.com
Stopping zmconfigd...Done.
Stopping zimlet webapp...Done.
Stopping zimbraAdmin webapp...Done.
Stopping zimbra webapp...Done.
Stopping service webapp...Done.
Stopping stats...Done.
Stopping mta...Done.
Stopping spell...Done.
Stopping snmp...Done.
Stopping cbpolicyd...Done.
Stopping archiving...Done.
Stopping opendkim...Done.
Stopping amavis...Done.
Stopping antivirus...Done.
Stopping antispam...Done.
Stopping proxy...Done.
Stopping memcached...Done.
Stopping mailbox...Done.
Stopping logger...Done.
Stopping dnscache...Done.
Stopping ldap...Done.
Host mail.DOMAIN.com
Starting ldap...Done.
Starting zmconfigd...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting memcached...Done.
Starting proxy...Done.
Starting amavis...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting opendkim...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
Starting service webapp...Done.
Starting zimbra webapp...Done.
Starting zimbraAdmin webapp...Done.
Starting zimlet webapp...Done.
As you can see, it takes care of deploying the certificates and restarting the Zimbra server.
Possible problems
At first, when I launched the process, I received this result:
certbot-zimbra v0.7.12 - https://github.com/YetOpen/certbot-zimbra Checking for dependencies... Detected Zimbra 8.8.15 on UBUNTU16_64 Using domain mail.DOMAIN.com (as certificate DN) Preparing certificates for deployment. cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory
To fix this, first verify that you have the ca-certificates installed:
root@mail:/# apt-cache policy ca-certificates ca-certificates: Installed: 20210119~16.04.1ubuntu0.1~esm1 Candidate: 20210119~16.04.1ubuntu0.1~esm1
If they are not installed, install them using the following command:
apt-get install ca-certificates
Then follow the steps above to install the Snap version of Certbot, and force a certificate renewal using the ISRG Root X1.