Implement OCSP responder

[asyd]

Can you please thinking to add an OSCP responder in vault pki backend?

Thanks

[jefferai]

Hi Bruno,

We currently have no plans to do so. Sorry!

[danilobuerger]

See https://github.com/T-Systems-MMS/vault-ocsp

[jefferai]

@danilobuerger that's awesome! Would love to get that directly integrated into Vault.

[danilobuerger]

@jefferai lets ping @jandd, he wrote the thing.

[asyd]

👍 🍰

[jefferai]

@danilobuerger Ah, I saw you had commits on the repo so thought you were directly involved.

@jandd if you're interested in pushing this upstream I can help out.

[jandd]

@jefferai vault-ocsp depends heavily on Cloudflare's cfssl (BSD-2 clause licensed). If this is no hurdle for integration into vault I would be happy to get it integrated.

[jefferai]

@jandd Yep, that's no hurdle.

The main issue will be getting non-JSON OCSP requests handled as Vault has no mechanism for doing that now. I have some thoughts on how to approach it though. Do you prefer to discuss over GH or do you want me to add you to a Slack channel?

[jandd]

@jefferai Slack would be fine

[danilobuerger]

@jandd if you need any assistance, I am willing to pitch in.

[jefferai]

@jandd Cool, I will send an invite, however, please know that due to HsshiDays I will have no time for this until later next week.

[nwwells]

What's the status of this?

[jefferai]

The original author disappeared, and since we consider OCSP an anti-pattern the Vault team won't be spending time on this personally.

[danlsgiga]

+1 Having this would be super awesome!

[gautaz]

The original author disappeared, and since we consider OCSP an anti-pattern the Vault team won't be spending time on this personally.

@jefferai Would you mind explaining why the Vault team is considering OCSP an anti-pattern?

[oboukili]

The original author disappeared, and since we consider OCSP an anti-pattern the Vault team won't be spending time on this personally.

@jefferai I was really looking forward to it to perform OCSP stapling on the resources publishing those generated certificates, as simple CRL (simple OCSP as well tbh) just does not scale and can be stale.

imo, using short TTLs may not sufficient in case of a security breach.

[jefferai]

We actually consider simple CRLs to be an antipattern too. If you must use CRLs we recommend a CRLSets approach.

Some really good reading on this topic is a series of posts over multiple years from Adam Langley at Google, who wrote Go's crypto libraries and is one of those rare people on earth that you can trust to implement crypto properly. See, chronologically, https://www.imperialviolet.org/2011/03/18/revocation.html | https://www.imperialviolet.org/2012/02/05/crlsets.html | https://www.imperialviolet.org/2014/04/19/revchecking.html | https://www.imperialviolet.org/2014/04/29/revocationagain.html

OCSP stapling removes one leg of the conversation but it still has a soft-fail problem and as a result it's still cached -- the whole point is that the signature is valid for some period of time. Traditionally a CRL would be cached about 72 hours before refreshing and in a soft-fail scenario on refresh you'd give it another 72 hours. Let's say that with OCSP stapling you cache for 24 hours. Why not just make your certificate lifetimes 4 hours max? Vault makes it easy to do this. We know clients literally generating a new cert with Vault per client connection.

If in the case of a revocation you want to remove the cached OCSP response from a server you can do so -- but clients may still have cached it, and you may have very little control over that if it's happening across languages/libraries/OSes. Certificate lifetimes are treated the same way everywhere; they're simple and predictable, which are really good things in security. And they can be as short as you want to make them.

FWIW, we've heard from a number of businesses that CRLs and/or OCSP are required because of auditors and compliance. Every time we've suggested this approach, we've never had them come back about any issue that an auditor had with this approach, because the auditors see it as going above and beyond the compliance requirements due to the significantly shortened window.

[danilobuerger]

@jefferai The soft-fail problem could be solved with OCSP must staple.

[jefferai]

Must-staple only indicates that there must be a stapled response. How fresh that response has to be is still something that is up to the user/implementer/server/etc., which means you can still be returning cached responses for revoked certificates until whatever is doing the stapling refreshes its cache. Must-staple simply makes it an error if the upstream responder is broken and the cache has expired and the server decides to then not return a stapled response. If the server behaves that way, and many are horribly broken.

Revocation has always been broken and is still broken, and even if must-staple were implemented properly you're still dealing with a potentially stale cache. Make your certificate lifetimes short, between <shorter than that caching period> and <significantly shorter than that caching period> and revocation becomes entirely moot and every client will do the right thing automatically.

[danilobuerger]

I only said it would solve the soft-fail problem, not the caching one. However, the caching problem can also be solved by drastically reducing the validity interval of the OCSP response.

In the end, I think your conclusion is right. Reducing the certificate lifetime to hours is good enough. However, I don't agree with your arguments. As shown, you can handle the soft-fail and caching problems. An OCSP responder integrated into vault would surely do that.

[jefferai]

Integrating an OCSP responder into Vault means you have an authoritative source for OCSP information, but that doesn't protect you from downstream servers performing stapling but with with poorly-implemented caching or stapling behavior.

Can soft-fail and caching be mitigated? To some extent, yes, but it mostly comes down to making caching intervals shorter. A normal/common OCSP response window is 6 days with a cache refresh of 3 days, so if you drastically reduce the validity to 4 hours, why not just make your certificate lifetimes 2 hours instead? You don't have to deal with any CRL or OCSP infrastructure, have no soft-fail concerns, have no caching concerns, behave predictably with every client, and reduce network traffic.

I understand that short certificate lifetimes present their own challenges but they're often relatively straightforward to work around and they have tangible and straightforward benefits. But, so far as the original question of why the Vault team considers CRLs and OCSP to be an anti-pattern -- now you know.

[gautaz]

@jefferai Thanks for all the insight and explanation.

You may also consider that short certificate lifetimes and hence certificate rotation might be hard to manage if they are used by a large fleet of IoT objects dispatched all over a facility.

I am far from being a security expert and as far as I understand, having these objects being non responsive or also being stored temporarily without any access to the infrastructure means that you will not be able to rotate their certificate automatically. Having short certificate lifetime means more occurrences of "manual" operations to get back online these objects.

So it seems to me that this is all about making trade-offs. How would you manage this type of situation? For now I am more inclined to use CRLs to deal with this after what was just said on OCSP.

[jefferai]

If you need some kind of revocation I would suggest the CRLSets approach. Regularly push the current CRL(s) to the devices and/or have them fetch the latest CRL(s) when they regain connectivity, then use them locally for validation.

[trinitronx]

Sorry for reviving this issue, but I found some important relevant info!

After some testing of CRL & browser behavior, it appears that a CRL alone may be ineffective in revoking PKI & Certificate trust! Therefore, OCSP or low TTL certs may be the only way in the future to ensure certificate revocation is working as expected.

Tested on macOS 10.12.6 with browsers:

• Google Chrome Version 66.0.3359.139 (Official Build) (64-bit)
• Firefox Quantum 59.0.2 (64-bit)

Test Case Walkthrough

For example, if I set up a PKI + Intermediary CA & CRLs via:

$ export VAULT_ADDR=https://vault.example.net
$ export VAULT_TOKEN=$(cat ~/.vault-token)
$ export PKI_NAME=pki_example_ca
$ export CA_NAME=ca/example_ca
$ mkdir -p ~/vault_pki_test/generated/
$ vault mount -path=$PKI_NAME -description="Example Root CA for Intranet" -max-lease-ttl=87600h pki
$ vault write $PKI_NAME/root/generate/internal common_name="*.example.net" ttl=87600h key_bits=4096 key_type=rsa max_path_length=1 ou="Example Intranet CA" organization="ACME Corp" country="US" province="Foobar" locality="Baz" street_address="123 Fake Street" postal_code="55555"

$ vault mount -path=$CA_NAME -description="Example Intermediate Authority for Intranet Websites" -max-lease-ttl=87600h pki
$ vault write -field=csr $CA_NAME/intermediate/generate/internal common_name="*.web.example.net" ttl=87600h key_bits=4096 key_type=rsa max_path_length=0 ou="Example Intranet Intermediary CA" organization="ACME Corp" country="US" province="Foobar" locality="Baz" street_address="123 Fake Street" postal_code="55555" > ~/vault_pki_test/generated/web_int.csr

$ vault write -format=json ${PKI_NAME}/root/sign-intermediate csr=@${HOME}/vault_pki_test/generated/web_int.csr format=pem_bundle > ~/vault_pki_test/generated/sign-intermediate-result.json
$ cat ~/vault_pki_test/generated/sign-intermediate-result.json | \
  jq -r '"\(.data.certificate)\n\(.data.issuing_ca)"' > ~/vault_pki_test/generated/web_ca_bundle.pem
$ vault write $CA_NAME/intermediate/set-signed certificate=@${HOME}/vault_pki_test/generated/web_ca_bundle.pem

$ vault write $CA_NAME/config/urls \
      issuing_certificates="${VAULT_ADDR}/v1/${CA_NAME}/cert/ca" \
      crl_distribution_points="${VAULT_ADDR}/v1/${CA_NAME}/crl"

Next, create a role & generate some certs:

$ export ROLE_NAME=web-role
$ vault write $CA_NAME/roles/${ROLE_NAME} \
    allowed_domains=web.example.net \
    allow_subdomains=true \
    allow_bare_domains=false \
    allow_glob_domains=false \
    allow_any_name=false \
    enforce_hostnames=true \
    allow_ip_sans=true \
    key_type=rsa \
    key_bits=4096 \
    max_ttl=72h
$ echo "$(cat <<POLICY
path "${CA_NAME}/issue/${ROLE_NAME}" {
  capabilities = ["create", "update"]
}
POLICY
)" | \
  vault policy-write ${CA_NAME}/${ROLE_NAME} -

$ vault write ${CA_NAME}/issue/${ROLE_NAME} \
    common_name=foo.web.example.net
$ vault write ${CA_NAME}/issue/${ROLE_NAME} \
    common_name=bar.web.example.net

Copy those certs to a web server host (e.g.: nginx or Apache), then access them in your browser.

To fully test "green lock" web of trust, you will have to install the CA bundle we just created into your
OS or browser. Instructions depend on OS & browser used. These are basic instructions:

• Download the CA cert chain bundle for your environment from Vault Server:
  • curl -kv -o "/tmp/$(basename $CA_NAME).pem" ${VAULT_ADDR}/v1/${CA_NAME}/ca_chain
  • Check CA contents: openssl x509 -in "/tmp/$(basename $CA_NAME).pem" -noout -text
• Depending on your browser and OS:

  • Linux:

      # Note: Check your Distro's docs for OpenSSL / ca-certificates file location!
      # For CentOS 7 / RHEL:
      cp /tmp/$(basename $CA_NAME).pem /etc/pki/ca-trust/source/anchors/
      sudo update-ca-trust
      # For CoreOS
      sudo cp /tmp/$(basename $CA_NAME).pem /etc/ssl/certs
      sudo update-ca-certificates
    

  • OSX:

    • Simply open $(basename $CA_NAME).pem which will load it into the "Keychain Access" App. Under Certificates double click the certificate to open it, then expand Trust and switch it to Always Trust.

  • Windows:

    • Start Menu, click Run… (or Windows key + R) and type mmc
    • In MMC, File->Add/Remove Snap-in… and click the Add button
    • Select Certificates from the list of snap-ins and click Add.
    • Trusted CA certificates should go in the Local Computer store so choose the Computer Account radio button.
    • Click Next and then Finish.
    • Install the CA certificate chain $(basename $CA_NAME).pem and select place all certificates in the following store:
    • Trusted root certification authority store
    • Right click on the Trusted Root Certification Authorities folder and choose All Tasks -> Import… to bring up the Certificate Import Wizard.
    • The Certificate Import Wizard will walk you through the process of selecting a certificate file and adding it to the store.
    • You can then repeat this process as needed to add more trusted CA certificates to the store.

  • Firefox:

    • Open the about:preferences#privacy page (Just type in url bar, or Click Preferences -> Privacy & Security)
    • Under the Certificates section heading, Click the View Certificates button
    • Click the Authorities Tab heading
    • Click the Import… button
    • Locate the $(basename $CA_NAME).pem file and Click Open
    • *.web.example.net CA cert should now be imported and trusted by Firefox browser

Finally, after you have installed this go ahead and access your web server URLs in your web browser. Check for green lock, look at cert details & check it's valid.

Ok, so if this all worked, great! Time to check for CRL functionality.

Now, let's revoke all certs in this PKI!

$ export VAULT_ADDR=https://vault.example.net
$ export VAULT_TOKEN=$(cat ~/.vault-token)
$ export CA_NAME=ca/example_ca
$ curl -s -H "X-Vault-Token: $VAULT_TOKEN" -XLIST  ${VAULT_ADDR}/v1/${CA_NAME}/certs | jq -r '.data.keys[]' | xargs -n1 -I{} /bin/bash -c "export SERIAL_NUMBER='{}'; curl  -v --header \"X-Vault-Token: $VAULT_TOKEN\" --request POST --data \"{\\\"serial_number\\\": \\\"\$SERIAL_NUMBER\\\"}\"  ${VAULT_ADDR}/v1/${CA_NAME}/revoke ;"

# Check the CRL contents:
curl -vk ${VAULT_ADDR}/v1/${CA_NAME}/crl  -o web.crl
openssl crl -inform DER -text -noout -in web.crl
# You should see the revoked serials here!

Ok, if you got this far, you should have a CRL full of these revoked cert serials. You should have a web server with previously valid & trusted "green lock" TLS cert & chain.

But now we've revoked it, right?

Wrong! Check your browser again... refresh, clear cache & refresh again... still green lock! Ok, so what's going on? The web browsers themselves are not actually checking CRLs! D'oh!

So, according to these tests CRLs are definitely an "anti-pattern"... or at least they don't work as expected 😞 😅

Maybe OCSP would work instead (untested), but probably low TTL certs are safest.

[jefferai]

Chrome does not normally check CRLs or OCSP. See https://dev.chromium.org/Home/chromium-security/crlsets. I don't know about Firefox but assume it's similar.

Both are considered antipatterns in modern PKI. Vault currently supports CRLs so that you can implement CRLSets style checks. But you're better off with short TTLs. If your cert lifetime is less than the time for which you'd normally cache a CRL or OCSP response, you're on average at least as well off.

[jkirschner-hashicorp]

I realize the last activity on this closed issue was years ago, but for the sake of anyone who comes across this issue now and is still interested in the functionality:

Vault's PKI engine recently added OCSP responder support: #16723