[Urgent] Let's Encrypt revocations affecting your TLS certificates

[hferradj]

I just received this email from LetsEncrypt, informing me that all the certificates issued by my caddy server will be revoked on the 28th of January, at 16:00 UTC.

Please immediately renew your TLS certificate(s) that were issued from
Let's Encrypt using the TLS-ALPN-01 validation method and the following
ACME registration (account) ID(s):

 11*****3

We've determined that an error made it possible for TLS-ALPN-01
challenges, completed before today, to not comply with certificate
issuance requirements. We have remediated this problem and will revoke
all unexpired certificates that used this validation method at 16:00 UTC
on 28 January 2022. Please renew your certificates now to ensure an
uninterrupted experience for your site visitors.

We apologize for any inconvenience this may cause. If you need support
in the renewal process, please comment on our forum post. Our staff and
community members are available to help:

https://community.letsencrypt.org/t/170449

Thank you,

The Let's Encrypt Team

I have caddy version 2.2.1 and the certificates are stored with the dynamodb module.

I am unsure how to re-issue the certificates and am a bit worried as it's affecting clients' custom domains in production.

Here is what my caddy config file looks like:

{
  "storage": {
    "module": "dynamodb",
    "table": "odjo_caddy_ssl_certificates"
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "match": [
                {
                  "path": [
                    "/health"
                  ]
                }
              ],
              "handle": [
                {
                  "body": "Im healthy!",
                  "handler": "static_response",
                  "status_code": 200
                }
              ]
            }
          ]
        },
        "srv1": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "headers": {
                    "request": {
                      "set": {
                        "Host": [
                          "example.com"
                        ],
                        "X-Forwarded-For": [
                          "{http.request.remote.host}"
                        ],
                        "X-Forwarded-Host": [
                          "{http.request.host}"
                        ],
                        "X-Forwarded-Port": [
                          "{server_port}"
                        ],
                        "X-Forwarded-Proto": [
                          "{http.request.scheme}"
                        ]
                      }
                    }
                  },
                  "health_checks": {
                    "active": {
                      "timeout": 5000000000
                    }
                  },
                  "transport": {
                    "protocol": "http",
                    "tls": {}
                  },
                  "upstreams": [
                    {
                      "dial": "example.com:443"
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuer": {
              "email": "me@example.com",
              "module": "acme",
              "challenges": {
                "http": {
                  "disabled": true
                }
              }
            },
            "on_demand": true
          }
        ],
        "on_demand": {
          "ask": "https://example.com/system/caddy-domain-authorization-path"
        }
      }
    }
  },
  "logging": {
    "logs": {
      "": {
        "level": "INFO"
      }
    }
  }
}

Thank you

[mholt]

Caddy v2.4.2 and newer will automatically replace certificates that get revoked.

Easiest thing for you to do is just upgrade Caddy, and keep it up-to-date (after proper testing of course)! Especially if you have customers relying on it. I'd also recommend getting a sponsorship to support your business needs and support the project so we can support you better.

[francislavoie]

I have caddy version 2.2.1 and the certificates are stored with the dynamodb module.

Upgrade Caddy ASAP, versions of Caddy before 2.4.2 would not correctly automatically renew revoked certificates. See here for more details: https://community.letsencrypt.org/t/questions-about-renewing-before-tls-alpn-01-revocations/170449/21

Once you've updated, you won't need to take any additional action.

Edit: Dang, @mholt 🥷'd me 🤣

[hferradj]

Thank you so much :)

[sphr2k]

Unfortunately, this does not seem to work reliably. Two servers (Caddy v2.4.6) did not correctly recognize revoked certificates that need to be renewed.

[gbhrdt]

Same for me. Caddy 2.4.6 did not automatically renew revoked certificates. We are currently manually removing the affected certificates. @mholt

[francislavoie]

@sphr2k @gbhrdt We'll need more details. Could you share your logs? How did you notice the problem?

[mholt]

It can take up to 4 days for the revocation to be recognized as it has to wait for the OCSP Staple to be halfway through its validity period. The cert will be acceptable until the Good OCSP response expires.

[b-reich]

@sphr2k @gbhrdt We'll need more details. Could you share your logs? How did you notice the problem?

Firefox give you an error, that die cert is not longer valid.

[bmichotte]

As a workaround, I had to delete the files in /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory and restart caddy

[francislavoie]

@b-reich We need more details than that. What's your config, what's in your logs?

[gbhrdt]

@sphr2k @gbhrdt We'll need more details. Could you share your logs? How did you notice the problem?

Unfortunately I can not provide any logs as the Kubernetes Pod has already been removed. When it appears next time I will provide logs.

We noticed the problem when customers complained that opening their sites in Chrome, Firefox etc. yields SSL errors (cert revoked).

We solved it by removing the certificates from letsencrypt, so they are regenerated. They temporarily increased our rate limit after asked in the Community forums.
Also it's very good that you have the integration with ZeroSSL so even if there would be rate limits, everything still works.

[francislavoie]

Unfortunately I can not provide any logs as the Kubernetes Pod has already been removed.

You should probably persist/roll your logs somewhere in that case. That's not great.

[b-reich]

Here is my config. (Just randomize some private info)

{
    email somemail@example.com
}

tld1.example.com {
    reverse_proxy http://somehosta {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote.host}
        header_up X-Forwarded-For {http.request.remote.host}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
            }
    header {
        Strict-Transport-Security max-age=31536000;
    }
    encode zstd gzip
}
tld2.example.com {
    reverse_proxy http://somehostb {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote.host}
        header_up X-Forwarded-For {http.request.remote.host}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
            }
    header {
        Strict-Transport-Security max-age=31536000;
    }
    encode zstd gzip
}
tld3.example.com {
    reverse_proxy http://somehostc {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote.host}
        header_up X-Forwarded-For {http.request.remote.host}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
            }
    header {
        Strict-Transport-Security max-age=31536000;
    }
    encode zstd gzip
}
tld.4.example.com {
    reverse_proxy http://somehostd {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote.host}
        header_up X-Forwarded-For {http.request.remote.host}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
            }
    header {
        Strict-Transport-Security max-age=31536000;
    }
    encode zstd gzip
}
tld5.example.com {
    reverse_proxy http://somehoste {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote.host}
        header_up X-Forwarded-For {http.request.remote.host}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
            }
    header {
        Strict-Transport-Security max-age=31536000;
    }
    encode zstd gzip
}
tld6.example.com {
    reverse_proxy http://somehostf {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote.host}
        header_up X-Forwarded-For {http.request.remote.host}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
            }
    header {
        Strict-Transport-Security max-age=31536000;
    }
    encode zstd gzip
}

I dont have any logs for this machine. Caddy was not configured to save the logs. (I have already generated new certificates)

[mholt]

Caddy always emits logs by default. Without the logs, all we can do is guess, unfortunately.

[gbhrdt]

I just saw the latest CertMagic upgrade, as we are also Using OnDemand mode, I guess it might solve our issue in the next release: 599c81d

[mholt]

@gbhrdt Maybe. The comments on the commit suggest that there's still a bug (because I wrote the patch around midnight when I was way too tired, but I tried): caddyserver/certmagic@9245be5

[mholt]

This PR might be better: caddyserver/certmagic#166

[sphr2k]

@sphr2k @gbhrdt We'll need more details. Could you share your logs? How did you notice the problem?

I checked all our LE certificates which use tls-apln-01. Here are our logs:

caddy.log

As you can see, the cert for the mta-sts subdomain was automatically replaced; for the send subdomain, I had to delete the existing cert to force caddy to issue a new one.

[mholt]

All known revocation-related issues with certs not getting renewed should be fixed in CertMagic v0.15.3: https://github.com/caddyserver/certmagic/releases/tag/v0.15.3

This has been tested in production with thousands of domains and is known to work.

Honestly, this feature was just a party trick (I still don't know of any other server that even tries to auto-renew revoked certificates) and it was just tricky to get it right to work in more intensive use cases.

[owenconti]

Hi @mholt, please forgive my dumb question:

What can I do to get CertMagic 0.15.3 running on my Caddy servers?

[mholt]

@owenconti Just build Caddy from the latest commit. You might find https://github.com/caddyserver/xcaddy helpful.

[owenconti]

@mholt ah, I didn't realize it was already updated in the Caddy repo. I'll try out the custom build now.

Is there an ETA for 2.4.5 2.5?

[mholt]

I hope to tag a pre-release of 2.5 this month 🤞