webroot_map entries missing for re-issues with added domains

[criwayc]

My operating system is (include version):

Debian 12.5, x64

I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):

OS package (certbot 2.1.0-4)

I ran these commands:

1. certbot certonly --webroot -w /var/www/letsencrypt -m mail@example.com --agree-tos --keep-until-expiring --expand --preferred-chain "ISRG Root X1" -d primarydomain.tld
2. certbot delete --cert-name primarydomain.tld
3. certbot certonly --webroot -w /var/www/letsencrypt -m mail@example.com --agree-tos --keep-until-expiring --expand --preferred-chain "ISRG Root X1" -d primarydomain.tld -d secondarydomain.tld

Certbot's behavior differed from what I expected because:

The file /etc/letsencrypt/renewal/primarydomain.tld is missing the [[webroot_map]] entry for primarydomain.tld and only contains the one for secondarydomain.tld

This happens when the delete and "reissue" of a certificate happens within the timeframe (~7 days for LetsEncrypt?) where the previously already validated domains of the certificate do not need a re-validation.
In these cases certbot will only add the newly validated domain(s) to the webroot_map, causing the next auto renewal of this certificate to fail.

[osirisinferi]

I can't reproduce the failure of the renewal (with the most recent Certbot, not sure if that matters). Even with a completely empty [[webroot_map]] the certificate renews just fine because only a single webroot path is being used and that's configured in the webroot_path variable in the configuration file?

Can you perhaps show the failure? Preferably with the entire Certbot log, which was also one of the headers of the initial template which you conveniently removed 😉

[criwayc]

The renewal will result in this error:

---

Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for primarydomain.tld
http-01 challenge for secondarydomain.tld
Cleaning up challenges
Attempting to renew cert (primarydomain.tld) from /etc/letsencrypt/renewal/primarydomain.tld.conf produced an unexpected error: Missing command line flag or config entry for this setting:
Select the webroot for secondarydomain.tld:
Choices: ['Enter a new webroot', '/var/www/letsencrypt']

(You can set this with the --webroot-path flag). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/primarydomain.tld/fullchain.pem (failure)

---

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/primarydomain.tld/fullchain.pem (failure)

---

If your [[webroot_map]] is empty or has domain(s) missing, the renewal only works within the first 7? days, where LetsEncrypt does not actively revalidate the domain.

[osirisinferi]

Let's Encrypt caches valid authorizations for 30 days, but I tested using --dry-run which deactivates any prior valid authorization thus forcing a new one.

My Certbot does not produce your error when I test your setup.. While making sure no valid authz were present on the staging server, I ran:

certbot certonly --staging --cert-name foo --webroot -w /var/www/vhosts/foo.example.com/htdocs/ -d foo.example.com
certbot delete --cert-name foo
certbot certonly --staging --cert-name foo --webroot -w /var/www/vhosts/foo.example.com/htdocs/ -d foo.example.com -d bar.example.com
certbot renew --cert-name foo --dry-run

And the renewal worked flawlessly (with new authorizations due to the --dry-run; I can see them in the Apache log). The letsencrypt.log shows:

2024-05-09 20:46:50,194:INFO:certbot._internal.plugins.webroot:Using the webroot path /var/www/vhosts/foo.example.com/htdocs for all unmatched domains.

Thus not requiring any separate webroot paths in the webroot map..

Can you perhaps provide the log and the renewal configuration file?

That said I do notice while testing what you mean: when an authz is already valid, it won't be present in the webroot map. Whether this actually a problem or not, I'm not sure, because even without a complete webroot map, Certbot seems to be able to figure out the correct webroot paths with the partial webroot map and the webroot_path variable which contains both the webroot paths, separated by a comma 🤷‍♂️

Thus, how hard I try, I'm unable to reproduce your issue with Certbot 2.10.0.

[criwayc]

Seems you are right, after some more testing I can confirm that on a Debian 12 it will renew the certificate despite missing [[webroot_map]] entries. (only older versions of certbot fail in such cases)