add --ca-roots and --ca-intermediates flags to 'cosign verify' #3464

dmitris · 2024-01-02T19:25:00Z

Summary

Add new --ca-roots and --ca-intermediates flags to allow pass a certificate bundle PEM file with multiple CA roots and optionally a file with the intermediate certificates. Related to issue #3462. Current commit adds the two flags to verify the CLI options. There is a new functional test file test/e2e_tsa_certbundle.sh for the new flags to cosign verify.

Release Note

New features and improvements, including behavioural changes, UI changes and CLI changes
added --ca-roots and --ca-intermediates flags to take a certificate bundle PEM file with one or multiple CA roots and optionally a PEM file with intermediate certificates. Both flags are exclusive with --certificate-chain.

Documentation

sigstore/docs#291

codecov · 2024-01-02T19:27:54Z

Codecov Report

Attention: Patch coverage is 6.89655% with 54 lines in your changes are missing coverage. Please review.

Project coverage is 40.25%. Comparing base (2ef6022) to head (f7157af).
Report is 67 commits behind head on main.

Files	Patch %	Lines
cmd/cosign/cli/verify/verify.go	0.00%	38 Missing ⚠️
cmd/cosign/cli/options/certificate.go	0.00%	14 Missing ⚠️
cmd/cosign/cli/verify.go	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3464      +/-   ##
==========================================
+ Coverage   40.10%   40.25%   +0.15%     
==========================================
  Files         155      155              
  Lines       10044    10137      +93     
==========================================
+ Hits         4028     4081      +53     
- Misses       5530     5563      +33     
- Partials      486      493       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

haydentherapper · 2024-01-22T18:44:39Z

@dmitris Is this ready for review? Or were you looking for more initial feedback for moving it out from draft?

dmitris · 2024-01-23T17:00:54Z

@dmitris Is this ready for review? Or were you looking for more initial feedback for moving it out from draft?

I was waiting for the initial feedback - I think I have it now (in #3462), thanks! - I'll try to add the unit tests shortly and mark it as "ready to review".

dmitris · 2024-01-29T08:51:22Z

@haydentherapper marked as "ready for review" now. I thought about adding unit tests - but I believe the current implementation with the "big" VerifyCommand.Exec() method https://github.com/sigstore/cosign/blob/main/cmd/cosign/cli/verify/verify.go#L85-L316 doesn't make it easy to cover that code with unit tests, could be a good idea to refactor and split that. I added a new functional test test/e2e_tsa_certbundle.sh modelled after and expanding on the test/e2e_tsa_mtls.sh code - it works with the PR code and (unsurprisingly) fails with the trunk version (since it tests the newly added --ca-roots command-line flag). We also are running this version internally in some CI pipelines.

cmd/cosign/cli/options/certificate.go

haydentherapper

Given this is a more sensitive change, cc'ing a few others for thoughts - @woodruffw @kommendorkapten PTAL.

cmd/cosign/cli/verify/verify.go

cmd/cosign/cli/options/certificate.go

cmd/cosign/cli/verify/verify.go

haydentherapper · 2024-02-06T07:33:04Z

pkg/cosign/verify.go

+// ValidateAndUnpackCertWithCertPools creates a Verifier from a certificate. Verifies that the certificate
+// chains up to the provided root. CheckOpts should contain a pool of CA Roots and optionally the Intermediates.
+// Optionally verifies the subject and issuer of the certificate.
+func ValidateAndUnpackCertWithCertPools(cert *x509.Certificate, co *CheckOpts) (signature.Verifier, error) {


Given this is only used in one place, can we just call ValidateAndUnpackCert directly? I believe if no roots are provided, verification will fail so we don't need an explicit check, though it would be good to have a unit test to confirm.

[FYI] I tried to "combine" the two functions but was getting test failures with cosign attach flows - likely I have missed some code paths. Backed up to the state that you reviewed for now to get into the green build zone - will try again "with smaller steps."

I got rid of the "special" function ValidateAndUnpackCertWithCertPools in 44e0ff5.
Do you think I should remove this check?
https://github.com/sigstore/cosign/pull/3464/files#diff-fc9a26a4caa1d8684a4bdd9102f187f44e1f7fce7c06a19f4185035f13317cc8R296-R298

if co.RootCerts == nil { return errors.New("no CA roots provided to validate certificate") }

I would think it can be good to fail earlier with possibly more explicit error what have gone wrong, to make for easier troubleshooting (if it ever gets triggered). Let me know if you prefer this yanked @haydentherapper. I will also look into adding or expanding the unit tests, as you mentioned above.

cmd/cosign/cli/options/certificate.go

haydentherapper · 2024-02-06T07:36:18Z

cmd/cosign/cli/verify/verify.go

@@ -177,29 +179,59 @@ func (c *VerifyCommand) Exec(ctx context.Context, images []string) (err error) {
 		}
 	}
 	if keylessVerification(c.KeyRef, c.Sk) {


Is there a test we could add to e2e_test.go to exercise this new code?

@haydentherapper added functional tests in 62de169

haydentherapper · 2024-02-06T07:39:12Z

cmd/cosign/cli/verify/verify.go

@@ -60,6 +60,8 @@ type VerifyCommand struct {
 	CertGithubWorkflowName       string
 	CertGithubWorkflowRepository string
 	CertGithubWorkflowRef        string
+	CAIntermediates              string


Unfortunately, we have a lot of duplication in cosign right now. We'll also need to add this to verify-blob, verify-attestation, and verify-blob-attestation. Before you add that, let's wrap up all comments here, and then it should be a straightforward copy-paste.

yes, I know - wanted to do it in a follow-up PR to make it easier to review. Can make an issue not to forget - or add it to this PR.

(TODO - add this to verify-blob, verify-attestation, and verify-blob-attestation, after settling on the initial cosign verify case)

haydentherapper

LGTM overall, just two small comments.

So what's left is:

Add reading the chain from the new flags when a certificate is provided
Refactor into switch statement if you agree
Copy-paste this for all other verification functions and add tests

If you can add these as individual commits, that should make reviewing go fast.

The last nice-to-have would be removing the testing script and having these tests as code in e2e_test.go, because we now have fully hermetic testing with its own TSA, Fulcio, Rekor, and registry. I know this might be a bigger change so happy to circle back to this in a later PR.

haydentherapper · 2024-02-16T18:50:48Z

cmd/cosign/cli/verify/verify.go

-			if err != nil {
-				return err
+			if c.CARoots == "" {
+				// Verify certificate with chain


Do we also want to read in the chain from ca-roots and ca-intermediates to verify the cert?

@haydentherapper not sure if this is still relevant - I added in 96c2cd1 a comment to the co.CARoots != nil case: // Verify certificate with root (and if given, intermediate) certificate. Both with the chain and the ca-roots / ca-intermediates, the validation is done with the same function ValidateAndUnpackCert , only with the certificate chain there is a step of parsing the file and loading the CheckOpts. Let me know or resolve if this is ok or you would like some changes, please.

@haydentherapper also, regarding "what's left is: * Add reading the chain from the new flags when a certificate is provided" - I think we do read the chain - parse the certificates into CheckOpts which are then used for verifying certificates. Or do you mean something different? please let me know 😄

haydentherapper · 2024-02-16T18:52:12Z

cmd/cosign/cli/verify/verify.go

+				}
+			} else {
+				if co.RootCerts == nil {
+					return errors.New("no CA roots provided to validate certificate")


I would restructure this as a switch statement though, for "ca roots and cert chain == nil", "ca roots provided", "cert chain provided". This last check should not be possible then, correct? Either you've provided no root and we fetch it automatically, or you provide a root.

agree - done in dcefda7 (if looks good to you, please let me know, thumb up or resolve)

.github/workflows/e2e-tests.yml

woodruffw · 2024-02-16T22:10:53Z

cmd/cosign/cli/options/certificate.go

+	cmd.Flags().StringVar(&o.CAIntermediates, "ca-intermediates", "",
+		"path to a file of intermediate CA certificates in PEM format which will be needed "+
+			"when building the certificate chains for the signing certificate. "+
+			"The flag is optional and must be used together with --ca-roots, conflicts with "+
+			"--certificate-chain.")
+	_ = cmd.Flags().SetAnnotation("ca-intermediates", cobra.BashCompFilenameExt, []string{"cert"})
+	cmd.Flags().StringVar(&o.CARoots, "ca-roots", "",
+		"path to a bundle file of CA certificates in PEM format which will be needed "+
+			"when building the certificate chains for the signing certificate. Conflicts with --certificate-chain.")
+	_ = cmd.Flags().SetAnnotation("ca-roots", cobra.BashCompFilenameExt, []string{"cert"})
+


I'm less familiar with cosign so please take this with a grain of salt, but: I'm personally -1 on separate state flags for intermediate and root CA certificates.

My rationale:

Technically speaking, there is no real distinction between a "root" and an "intermediate" certificate: some people use "root" to mean "self-signed," but it's common to have both non-self-signed and cross-issued roots in real PKIs. In other contexts "intermediate" means "part of the untrusted set", but it's not immediately clear to me whether intermediates are trusted or untrusted in cosign (CC @haydentherapper)

Long term, I believe the plan is to have most Sigstore clients use a common --trusted-root or similar flag, which will load a shared format for the roots of trust for an "entire" Sigstore deployment (including the Rekor key, CTFE key, etc.). While this doesn't preclude that effort, it does add another set of knobs that will eventually need to be deprecated (or treated as another source of state).

Similarly long-term, I believe the plan is to have most Sigstore clients (including cosign) use Sigstore bundles as their interchange format. Bundles in turn support untrusted intermediates for the BYO PKI case, and in an unambiguous fashion (everything in the bundle is presumed untrusted).

So, TL;DR: if the idea behind --ca-intermediates is for it to pass trusted intermediates, then IMO it should be removed and collapsed into --ca-roots (which IMO could be called --trusted-certs to minimize confusion).

On the other hand, if these certificates are meant to be untrusted I can see the value of this flag, but I think it might be somewhat short-lived given the trustroot + bundle progression. But maybe that isn't as far along as I think 😅

For public good Sigstore, the intermediate is "trusted", as it is provided via the TUF trust root, and it's part of the trusted_root.json, and for public good we do not allow intermediates not provided direct via the trust root.

But if cosign is to be used with other deployments, they can be untrusted. So with that in mind, and the bundle format is not yet supported in cosign (to carry untrusted intermediates) I think the current proposal makes sense, but it can feel a bit awkward for the public good instance.

@woodruffw thanks for your review and suggestions!

Technically speaking, there is no real distinction between a "root" and an "intermediate" certificate

Well, for example looking at the Fulcio Certificate Specification, it has subsections for Root, Intermediate, and Issued Certificates, with specifications of what each type of certificate MUST / MUST NOT / SHOULD have. Also, for better or worth, a search for intermediate brings up over 140 hits in the sigstore/cosign codebase, including in the existing options for cosign verify:

doc/cosign_verify.md: --certificate-chain string path to a list of CA certificates in PEM format which will be needed when building the certificate chain for the signing certificate. Must start with the parent intermediate CA certificate of the signing certificate and end with the root certificate

also over 60 matches have both intermediate and root in the same line so these terms seems both well established and useful in the cosign repo and code. Maybe it is due to my 'surrounding environment' but I don't remember any cases where I or people around would have difficulties distinguishing which (CA) certificates are root ones, and which - intermediates.

In other contexts "intermediate" means "part of the untrusted set", but it's not immediately clear to me whether intermediates are trusted or untrusted in cosign
[...]
But if cosign is to be used with other deployments, they can be untrusted.

I have to admit having difficulties understanding which certificates are (or should be) considered "trusted" and "untrusted" in cosign (there is only one search hit on untrusted in sigstore/cosign and it is to a CHANGELOG in #2124; the only search hit on untrusted in https://docs.sigstore.dev also is not very helpful). I think the certificate trust is so much context-dependant. For example, I'm used to think of the certificates issues by our TheCompanyCA as both fully trusted and more trustworthy than those from Digicert & Co. 😄 - while in an environment oriented to the public websites and Certificate Authorities it would likely be considered as "Self-Signed" and untrusted.

Regarding the points 2. and 3. ("use a common --trusted-root flag" and Sigstore bundles as the interchange format) - in the long term, I have some reservations regarding this in terms of the ease of use and Developer / DevOps Experience. I worry that they would become custom requirements and make hurdles to using Sigstore for signing artifacts. PEM certificate bundles are "universal", ubiquitous and extremely familiar to everyone working with them and also deeply interwined with the existing scripts, processes etc. The TUF roots / Sigstore bundles are (IMO) in comparison relatively new, more complex (PEM -> JSON -> ...), "special purpose" and require custom handling, adoption steps / timeframe and increase the adoption and operational friction, especially for large users. Undoubtedly, the change will make life easier for us as the sigstore developers and the codebase probably nicer ("just one file/format to rule them all!") but possibly at the expense of placing some extra burden on the end users ("in addition to distributing the ca-bundle.pem to 100,000 endpoints that you are doing now and must continue to do for many years, you also need to create the file and distribute it to all the endpoints that will or might need to verify cosign signatures"). So I would recommend and plead 😄 to consider the trade-off carefully and if decided, let the users know well in advance about the upcoming change(s) and provide for an ample transition period (maybe something like one major version during which one could use both old and new ways of supplying certificates) 🙏 /cc @haydentherapper

But as you said, this would be long-term changes - and currently we have an "acute" issue of not being able to (properly) verify cosign signatures with cosign verify and through the cosign Go library if there is more than one CA authority issuing certificates - namely using a certificate bundle PEM file that contains multiple CA Root certificates. (And we would like to start issuing and verifying lots of Sigstore/Cosign signatures in Q1 '24! 😁).

The suggested cosign verify --ca-roots <certbundle.pem> and the expanded CertVerifyOptions and VerifyCommand in the Go library would solve that problem. If you have other ideas how we could do this (in the near term), please let me know! (I thought about expanding the current --certificate-chain format or adding --certificate-chains [plural], but couldn't figure out how to handle the file format - with the single chain, it is easy to parse the file [<Intermediate>*<Root>] but if there are multiple roots, it gets very messy.... I also didn't want to get into the business of building multiple certificate chains from a set of certificates - the Go stdlib already does it very well but it requires providing Roots and optionally Intermediates in the VerifyOptions mentioned above - to which the two new suggested flags would correspond.)

Related to issue sigstore#3462. Current commit adds the flag to verify the CLI options. The new flag doesn't have any effect yet (will add in follow-up PRs). Signed-off-by: Dmitry S <dsavints@gmail.com>

Add --ca-roots command-line flag for 'cosign verify' to enable verifying cosign signatures using PEM bundles of CA roots. Whether to also add --ca-intermediates flag is TBD. Unit tests will be added in the next commit(s). Fixes sigstore#3462. Signed-off-by: Dmitry S <dsavints@gmail.com>

Signed-off-by: Dmitry S <dsavints@gmail.com>

Add --ca-intermediates flag to enable to pass a PEM file with intermediate CA certificates. One can use either --ca-roots, optionally together with --ca-intermediates - or --certificate-chain, which contains zero, one or several intermediate CA certificate followed by the root CA certificate. Expand the helper Go program test/gencert/main.go to allow to generate root and intermediate CA certificates, and a certificate signed by the intermediate CA. Expand the functional test e2e_tsa_certbundle.sh to test the --ca-intermediates flag (together with --ca-roots). Fixed sigstore#3462. Signed-off-by: Dmitry S <dsavints@gmail.com>

* simplify switch statements (remove unnecessary brackets) * reword help text for '--ca-roots' and '--ca-intermediates' flags for clarity Signed-off-by: Dmitry S <dsavints@gmail.com>

Signed-off-by: Dmitry S <dsavints@gmail.com>

dmitris force-pushed the certbundle branch from 4e053fe to 88a2079 Compare January 2, 2024 19:25

dmitris changed the title ~~add --certificate-bundle flag to 'cosign verify'~~ add --ca-roots flag to 'cosign verify' Jan 9, 2024

dmitris mentioned this pull request Jan 9, 2024

feature: 'cosign verify' add flags --ca-roots and --ca-intermediates to allow multiple CA roots #3462

Open

dmitris force-pushed the certbundle branch from b6c572c to 3943fef Compare January 11, 2024 16:56

dmitris force-pushed the certbundle branch from f012234 to 2f4d7a8 Compare January 22, 2024 18:16

dmitris force-pushed the certbundle branch 3 times, most recently from 8fb3141 to fca88dc Compare January 29, 2024 08:41

dmitris marked this pull request as ready for review January 29, 2024 08:42

dmitris force-pushed the certbundle branch 2 times, most recently from 323e003 to 9954fe2 Compare January 29, 2024 18:59

dmitris mentioned this pull request Jan 29, 2024

document --ca-roots and --ca-intermediates flags for 'cosign verify' sigstore/docs#291

Merged

dmitris force-pushed the certbundle branch 4 times, most recently from 8e49130 to 22350e9 Compare January 30, 2024 15:02

haydentherapper reviewed Jan 31, 2024

View reviewed changes

cmd/cosign/cli/options/certificate.go Show resolved Hide resolved

dmitris force-pushed the certbundle branch 7 times, most recently from e770d38 to e7375fa Compare February 5, 2024 17:52

haydentherapper requested changes Feb 6, 2024

View reviewed changes

dmitris force-pushed the certbundle branch from e7375fa to 449147a Compare February 6, 2024 17:28

dmitris force-pushed the certbundle branch 4 times, most recently from 44e0ff5 to 6451fc4 Compare February 16, 2024 12:28

haydentherapper reviewed Feb 16, 2024

View reviewed changes

woodruffw reviewed Feb 16, 2024

View reviewed changes

dmitris changed the title ~~add --ca-roots flag to 'cosign verify'~~ add --ca-roots and --ca-intermediates flags to 'cosign verify' Feb 19, 2024

dmitris force-pushed the certbundle branch 8 times, most recently from bd10952 to 62de169 Compare February 27, 2024 08:15

dmitris force-pushed the certbundle branch from 62de169 to 4006e62 Compare March 1, 2024 17:51

dmitris force-pushed the certbundle branch from 4006e62 to 06690fd Compare March 13, 2024 13:59

haydentherapper mentioned this pull request Mar 20, 2024

Request to support non-identity based cert as verifier sigstore/policy-controller#1318

Open

dmitris force-pushed the certbundle branch from 06690fd to 6d3ff2e Compare March 22, 2024 13:24

dmitris added 10 commits April 3, 2024 09:25

add --certificate-bundle flag to 'cosign verify'

4fa0491

Related to issue sigstore#3462. Current commit adds the flag to verify the CLI options. The new flag doesn't have any effect yet (will add in follow-up PRs). Signed-off-by: Dmitry S <dsavints@gmail.com>

add functional tests for --ca-roots flag

e98ab7a

Signed-off-by: Dmitry S <dsavints@gmail.com>

address PR feedback

cb0225e

* simplify switch statements (remove unnecessary brackets) * reword help text for '--ca-roots' and '--ca-intermediates' flags for clarity Signed-off-by: Dmitry S <dsavints@gmail.com>

setup-crane action for e2e_test_pkcs11.sh

667ad6e

Signed-off-by: Dmitry S <dsavints@gmail.com>

get rid of ValidateAndUnpackCertWithCertPools

66b7245

Signed-off-by: Dmitry S <dsavints@gmail.com>

refactor CertChain/RootCerts check as switch

4c80f04

Signed-off-by: Dmitry S <dsavints@gmail.com>

add comment for co.RootCerts non-nil case

7ba741d

Signed-off-by: Dmitry S <dsavints@gmail.com>

add functional tests for CARoots handling

f7157af

Signed-off-by: Dmitry S <dsavints@gmail.com>

dmitris force-pushed the certbundle branch from 6d3ff2e to f7157af Compare April 3, 2024 07:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add --ca-roots and --ca-intermediates flags to 'cosign verify' #3464

add --ca-roots and --ca-intermediates flags to 'cosign verify' #3464

dmitris commented Jan 2, 2024 •

edited

codecov bot commented Jan 2, 2024 •

edited

haydentherapper commented Jan 22, 2024

dmitris commented Jan 23, 2024

dmitris commented Jan 29, 2024

haydentherapper left a comment

haydentherapper Feb 6, 2024

dmitris Feb 13, 2024

dmitris Feb 14, 2024

haydentherapper Feb 6, 2024

dmitris Feb 26, 2024 •

edited

haydentherapper Feb 6, 2024

dmitris Feb 6, 2024 •

edited

dmitris Feb 19, 2024

haydentherapper left a comment

haydentherapper Feb 16, 2024

dmitris Feb 20, 2024 •

edited

dmitris Feb 20, 2024

haydentherapper Feb 16, 2024

dmitris Feb 19, 2024 •

edited

woodruffw Feb 16, 2024

kommendorkapten Feb 19, 2024

dmitris Feb 19, 2024

add --ca-roots and --ca-intermediates flags to 'cosign verify' #3464

Are you sure you want to change the base?

add --ca-roots and --ca-intermediates flags to 'cosign verify' #3464

Conversation

dmitris commented Jan 2, 2024 • edited

Summary

Release Note

Documentation

codecov bot commented Jan 2, 2024 • edited

Codecov Report

haydentherapper commented Jan 22, 2024

dmitris commented Jan 23, 2024

dmitris commented Jan 29, 2024

haydentherapper left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitris Feb 26, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitris Feb 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haydentherapper left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitris Feb 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitris Feb 19, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitris commented Jan 2, 2024 •

edited

codecov bot commented Jan 2, 2024 •

edited

dmitris Feb 26, 2024 •

edited

dmitris Feb 6, 2024 •

edited

dmitris Feb 20, 2024 •

edited

dmitris Feb 19, 2024 •

edited