Use right fulcio certificate when verifying bundles

[flavio]

Description

Fulcio has updated its certificate. Right now, inside of Sigstore's TUF repository there are 2 certificates:

• fulcio.crt.pem
• fulcio_v1.crt.pem

Keyless signatures produced some months ago have been signed with fulcio.crt.pem, while more recent ones have been signed with fulcio_v1.crt.pem.

Right now the cosign::Client uses just one certificate, which causes some signatures to fail verification.

Apparently the cosign client isn't affected by this issue. We should understand how the upstream client handles that, and replicate the behaviour here.

[flavio]

I found the root cause, to solve that we need to verify the certificate against a set of root CAs.

After a lot of searching, I found out that webpki can help us there. Unfortunately it needs some minor changes that are implemented via briansmith/webpki#254

[asraa]

I found the root cause, to solve that we need to verify the certificate against a set of root CAs.

Yep!

Apparently the cosign client isn't affected by this issue. We should understand how the upstream client handles that, and replicate the behaviour here.

Also -- Hayden and I scoped how this would look like in the future as more Fulcio and rekor logs are rotated. It will end up being that we add all targets that have an annotation `"usage": Fulcio" attached to them. Hayden made a PR here sigstore/cosign#1423

I wonder if awslabs has support for parsing/reading custom metadata attached to targets? I can try to look into that

[flavio]

Also -- Hayden and I scoped how this would look like in the future as more Fulcio and rekor logs are rotated. It will end up being that we add all targets that have an annotation `"usage": Fulcio" attached to them. Hayden made a PR here sigstore/cosign#1423

Thanks for pointing this out! In the meantime my plan is to just load all the files that are matching a certain pattern.
We can change the code once the TUF metadata is updated. Can you ping me once this is out?

I wonder if awslabs has support for parsing/reading custom metadata attached to targets? I can try to look into that

Yes, it seems this is handled: https://docs.rs/tough/latest/tough/schema/struct.Target.html - I guess they will show up inside of the custom attribute, if not, they will be inside of the _extra one, which is populated by "serde flatten".

There's another question I have. What is your plan regarding rekor's public key? What do you plan to do if this key is suddenly changed?

[flavio]

We might have a "problem". We need to create a certificate pool composed of all the Fulcio certificates found inside of the TUF repository, and then use it to validate the certificates issued by Fulcio.

Currently there are two Rust libraries that allow that:

• webpki
• picky

Unfortunately there isn't a clear winner and, most important of all, there's nothing that works out of the box. Let's take a closer look at both libraries.

webpki

Pros:

• Based on ring. We already depend on that, this is a widely used and trusted crypto library.
• Wide adoption: the webpki crate is also used by rustls.
• Supports many type of encryption algorithms, all the ones we need are handled.

Cons:

• It's being designed to validate certificates that are meant to be used by a server or by a client. Fulcio certificates do not have the right attributes for the current checks.
• Maintainers seems to be busy, lots of PR are pending feedback/approval. That includes my PR that addresses the limitation described above.

picky

Pros:

• Easy to use
• Pure Rust library, including its dependencies
• Seems to be an active project

Cons:

• Smaller adoption
• Elliptic curve keys are not supported yet. Unfortunately we need that in order to process Fulcio's certificates.

Proposed solutions

These are the possible solutions I see. I would personally go with the first one, the one that leverages webpki.

Leverage webpki

I propose we build a solution based on webpki, using the patch that hasn't been accepted upstream. The patch is really small and not invasive. Hopefully we will find an agreement with upstream about that, and sort things out for their next release.

We however have an issue. Given we want to release sigstore-rs on crates.io, all our dependencies must be published on crates.io. This would not be possible for webpki, until the patch is officially merged and released.

A common solution is to vendor the whole patched webpki library inside of our Git repository.

Perform a small change to the current code

Currently, the code is verifying the validity of the certificates issued by Fulcio in this way:

• Extract the public key of Fulcio's CA from one single certificate that we trust. This would be either the certificate we found inside of the TUF repository, or one provided by the user.
• Parse the certificate found inside of a signature using the x509-parser library, and then leverage the X509Certificate::verify_signature method to ensure this certificate has been issued by the Fulcio CA we know about.

We could change the code to:

• Have a vector of Fulcio certificates, instead of a single object like we currently do
• Iterate over the list of these trusted certificates:
  • Extract the public key from the certificate
  • Use the X509Certificate::verify_signature method like we are already doing.
  • If the verification is successful, mark the certificate found inside of the certificate as trusted
  • If none of the trusted Fulcio certificates satisfies the verification check, mark the signature certificate as untrusted

Pros:

• No need to vendor code, everything we need is already public (*)

Cons:

• This wouldn't work when intermediate certificates are used. This is not the case for the official Fulcio instance, but it could be for the self-hosted Fulcio instances.

(*) Yesterday a new version of x509-parser got released: v0.13.0. This version has a regression that breaks the verification of Fulcio issued certificates. I've already created this PR to address that. In the meantime we could just use the previous v0.12.0 release, which is not affected by this issue.

[lukehinds]

This wouldn't work when intermediate certificates are used. This is not the case for the official Fulcio instance, but it could be for the self-hosted Fulcio instances.

Is this likely to always be the case or temporary, e.g. is it possible upstream code changes provide a method of handling intermediates?

This would not be possible for webpki, until the patch is officially merged and released.

How certain are we it will get merged (do you have a link to the PR?)

[flavio]

Is this likely to always be the case or temporary, e.g. is it possible upstream code changes provide a method of handling intermediates?

Both webpki and picky handle intermediate certificates. It's our current naive implementation that doesn't.

How certain are we it will get merged (do you have a link to the PR?)

To be honest, I'm not certain at all. There are lots of pending PRs. Mine is briansmith/webpki#254, which is pretty small and building on top of already existing code. That should give it higher chances to be merged 🤞

Another option would be to go ahead and implement EC support inside of picky. I've already worked with other EC libraries that are not ring, that wouldn't be a problem for me. But I'm not sure this is worth the efforts due to the limited popularity (compared to webpki) of picky (no offense for the maintainers intended!).

[flavio]

I've worked on this PR Devolutions/picky-rs#132 that adds EC support to picky.

[flavio]

Good news, the PR has been accepted and merged. A new version of picky which includes this fix is going to be released in the next days.

I'll start working on the code that consumes the library, so that everything is going to be ready by the time a picky release is available.

[flavio]

Also -- Hayden and I scoped how this would look like in the future as more Fulcio and rekor logs are rotated. It will end up being that we add all targets that have an annotation `"usage": Fulcio" attached to them. Hayden made a PR here sigstore/cosign#1423

@asraa I just checked, it looks like this extra information is not yet added to the TUF repository (unless tough is not parsing it - which would be a bug - or I'm doing something wrong).

Do you know if this additional metadata is already added to the targets definition? Thanks!

[lukehinds]

great, they were evidently the good pick of the two if they reviewed that quickly.

[asraa]

@asraa I just checked, it looks like this extra information is not yet added to the TUF repository (unless tough is not parsing it - which would be a bug - or I'm doing something wrong).

Hey! I just saw this. That's correct, we're waiting for version 3 root signing. The targets will looks like sigstore/root-signing#139 (comment) but we fallback on the hardcoded names

[flavio]

@asraa thanks, would you be so kind to ping me when this change is finally in place (meaning the TUF metadata is updated) 🙏