Support trusted root TUF target

[haydentherapper]

Description

Copied from https://sigstore.slack.com/archives/C049ALX6Q83/p1709072587850229

tl;dr - Sigstore TUF metadata has evolved, but Cosign and Scaffolding are lagging behind. We need to start ingesting the new format so we can fully deprecate the old format.

For some context, a history of Sigstore TUF targets:

• Originally, TUF targets had to match expected file names, like "fulcio.crt.pem" or "ctfe.pub". You can see examples of this for Cosign. This was a problem for a lot of reasons: What happens when we need to rotate the key or cert? What happens if I want to support both a public and private deployment?
• To deal with that, we added "custom" metadata on each TUF target metadata, like https://github.com/sigstore/root-signing/blob/9f2561e6d5a8035ed2e742d3fc47907d5164baf2/repository/repository/targets.json#L41-L47. This conveyed a) what service it should be used with (Fulcio, Rekor or the CT log), b) whether it was the latest Active key or Expired (but still trusted, if untrusted it would be removed from the TUF targets), and c) the URI of the service (to support multiple instances of a log or CA in one trust root)
  • We do handle that in Cosign.
  • But this was not standardized across clients. No other Sigstore client supports custom metadata. And even within the Go clients, it's inconsistent. In Cosign, we ignore the URI, whereas in Scaffolding, we generate metadata with the URI
• To standardize across clients, we created a Sigstore TUF client specification, and defined a spec for a trust root. Each target in a trust root would include its URI, its validity period, and any additional metadata. This also let us define additional target services, like timestamp authorities, and can easily be extended since the metadata is versioned.
  • Here's the public good instance trust root target

Note that Cosign used to be the only Sigstore client, but now we've got Sigstore clients, and now also TUF clients, in a bunch of languages. Each of these newer clients has carefully followed the spec. There is a rewrite of the Go TUF client which aims to fix any spec deviations and API issues. We now have a Go Sigstore TUF client that uses the new API that has support for the trusted root metadata - https://github.com/sigstore/sigstore-go/tree/main/pkg/tuf.

Also note that all other clients besides Cosign (namely sigstore-python, -js, and -java, and -rs soon) are reading the trusted root file already.

So what do we need to do to get Golang Sigstore repos, namely Cosign and Scaffolding, up to date?

• Update Cosign to use the new Sigstore TUF client, which requires a trusted root target
  • Once we stop using the old TUF client, we will stop supporting private deployments without the trusted root target. In the short term, we could have Cosign use both TUF clients, preferring the newer one when the trusted root target is available.
• Update Scaffolding to generate the trusted root file.
  • I don't know if any changes are needed in the TUF helm chart too. Probably not? It depends on what is ingesting the trusted root file and outputting the per-service target files.
• Update the public TUF repo and Scaffolding to no longer distribute the individual targets, which will require the updated Cosign. This will happen after a deprecation period.

cc @sabre1041 @lkatalin

[haydentherapper]

sigstore/cosign#3548 tracks the change in Cosign

This issue can track what's needed in Scaffolding. https://github.com/kommendorkapten/trtool/blob/main/cmd/trtool/app/initroot.go might be a place to start for code to generate a trusted root. We need to update target generation to generate a trusted root in addition to the individual targets. Note that if we jump straight to only generating the trusted root file, we'll need to have Cosign updated first.