copy: consider support cosign: verify image while pulling from source & sign image while copying to destination

[Dentrax]

Currently, we can pass a --sign-by flag by giving a GPG key.

--sign-by FINGERPRINT                   Sign the image using a GPG key with the specified FINGERPRINT

Just thinking how cosign fit in this project, so drew the following architecture:

We probably have to introduce some new flags to achieve this goal:

--dest-public-key <PATH TO PUBLIC KEY for SOURCE>
--src-private-key <PATH TO PRIVATE KEY for DESTINATION>
--sign-by <GPG,COSING> # to make it backward compatible

Eventually, the final copy command will look like this:

$ skopeo copy docker://src/foo:1.0 docker://dest/foo:1.0 --sign-by cosign --dest-public-key </path/to/public.key> --src-private-key </path/to/private.key>

It's optional to sign the image again with a different key. The motivation behind this is we might want to ensure all the images are signed with our own private key in the internal registry. So we can easily validate it before the deployment.

Any thoughts?

cc @developer-guy @dlorenc

[mtrmac]

Thanks for your report.

Most of that infrastructure needs to be developed in containers/image (and in particular, the public keys or other roots of trust need to be configured in policy.json, so that users are secure without having to provide them manually on every single command and never forgetting). Only after it is implemented and designed there would we discuss the details of how to expose it in a Skopeo UI.

For reference, some very partial work on that has happened in containers/image#1364 .

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

@mtrmac and I are currently working on integrating cosign into the toolchain.

[developer-guy]

that's awesome, please let us know if anything we can help 🙋🏻‍♂️

[developer-guy]

@vrothberg @mtrmac, is there anything that we might get involved in it? We're willing to work on this 🤩

[vrothberg]

@vrothberg @mtrmac, is there anything that we might get involved in it? We're willing to work on this star_struck

Thanks for offering help, @developer-guy, much appreciated! At the time of writing, I don't see any work items that we can distribute. We are currently consolidating parts of sigstore/cosign into sigstore/sigstore; the cosign code entails a lot dependencies that we need to prune.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Dentrax]

the cosign code entails a lot dependencies that we need to prune

Maybe we can just do call cosign binary directly instead of bring it as a new dependency.

[vrothberg]

the cosign code entails a lot dependencies that we need to prune

Maybe we can just do call cosign binary directly instead of bring it as a new dependency.

I don't think that is sustainable solution long term. The binary size of Skopeo is currently ~20MB, Cosign is currently at ~90MB. Cosign can do much more than just signing and verifying images.

[Dentrax]

Dropping Sanitize cosign's dependency tree issue as a reference.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[dmesser]

@Dentrax As a side, what's the perceived value of signing something while copying? Wouldn't you rather want to copy images that are signed already, verifying and copying the signature with the image in the process? See containers/image#1575

It seems like the assumption here is that we want to establish trust on a copy operation. Talking to users and customers taught me that they don't want to trust the machine / registry that's doing the copy but rather the source. Establishing a supply chain security posture that they can attest to along the pipeline.

[mtrmac]

To update the status, as of #1701 and the just-released Skopeo 1.9.0, Skopeo can enforce signatures via policy.json, and create new signatures on copy; both only using public/private key pairs, with no Fulcio/Rekor features.

[mtrmac]

@Dentrax As a side, what's the perceived value of signing something while copying? Wouldn't you rather want to copy images that are signed already, verifying and copying the signature with the image in the process?

A typical use case is promotion from a testing build to an approved public release:

• Enforcing that the original build is signed for testing (with a testing repo, or with a production repo and a signedIdentity remapping in policy.json)
• Removing the testing signature
• Adding a production signature with the production repo.

Alternatively, some large companies like to have an internal “use in $company is approved” signature added to every third-party image, and enforce their presence.

[Dentrax]

It seems like the assumption here is that we want to establish trust on a copy operation.

Yes, this is the idea, but also I see the your concern in next statement:

Talking to users and customers taught me that they don't want to trust the machine / registry that's doing the copy but rather the source.

Miloslav already covers the exact use-case and why we need:

some large companies like to have an internal “use in $company's approved” signature added to every third-party image, and enforce their presence.

Definitely what we were looking for!

---

I tried the new v1.9.0 release and couldn't make it work in the first place and filed an issue: containers/image#1613

After played a while, I am finally figured it out and make it work!

This feature is exactly why we filed this issue is for. I think we can close this one. 👏

[mtrmac]

If you don’t mind, I’d like to keep this open to track the planned Fulcio/Rekor support/interoperability. (Compare also #1704 .)

[dmesser]

A typical use case is promotion from a testing build to an approved public release:

• Enforcing that the original build is signed for testing (with a testing repo, or with a production repo and a signedIdentity remapping in policy.json)
• Removing the testing signature
• Adding a production signature with the production repo.

And users are looking to do that all with a tool that actually copies images? Seems like scope creep to me. Validating signatures as part of copying makes a lot of sense. Signing not so much (IMHO). Users will likely want to use a specific signing tool (cosgin, podman).

Alternatively, some large companies like to have an internal “use in $company's approved” signature added to every third-party image, and enforce their presence.

That makes sense, but again, why would someone do that with skopeo as part of the copy? Chances are that the decision whether to bless something with a company signature is based on some introspection, so the process likely involves downloading the image first and doing something with it. It's very likely that the user wants to sign the image in a specific way, which requires skopeo to know and support all the possible options by which you can sign an image (keyless, KMS-based, etc). Do we really want to carry all that into skopeo?

[mtrmac]

And users are looking to do that all with a tool that actually copies images?

I mean, what other tool in the GitHub.com/containers ecosystem would be more appropriate? The “promotion of a testing build” use case is literally copying an unmodified image from an internal testing repo to the public repo, which is the primary job of skopeo copy. Signing it in the process is a fairly small addition on top of the core copy intent.

(Users that don’t have a testing repo and just want build and publish an image would not need Skopeo; they can just podman build && podman push --sign-by*, and that’s better.)

I could totally buy an argument that this should be an integrated part of a large-scale build system application, and such an application should not shell out to a separate CLI. That’s perfectly fine; Skopeo is a thin wrapper over c/image and to an extent intentionally an example for building such large applications.

Users will likely want to use a specific signing tool (cosgin, podman).

cosign sign, and podman image sign as they exist right now, are notably less secure because the image is first copied to the destination, and only then signed there; so, the destination registry, if it is malicious, can modify the image in between.

In contrast, podman build && podman push --sign-by* is the best workflow in the simple case: It guarantees end-to-end-protection from the very start of the existence of the image, and has no external infrastructure dependencies that need to be trusted.

skopeo copy --sign-by* is useful in the rarer steps of adding a signature to an image, where the trust in the original image is somehow established; for unsigned images, it’s at least a bit more useful than plain cosign sign / podman image sign, because it assumes trust in the source registry, not in the destination registry. (If we fully trusted the destination registry to not be malicious, we wouldn’t need to sign the images to ensure authenticity; the signatures would only provide nonrepudiation, which is … not quite in the interest of the party publishing the image, if the image turns out to be backdoored.)

---

Chances are that the decision whether to bless something with a company signature is based on some introspection, so the process likely involves downloading the image first and doing something with it.

(I’m skeptical that an introspection of a few hundreds of megabytes of binary code can actually ensure much about the behavior, but sure, it can help with checking boxes like “Was this built in one of a $dozen approved systems? Are all the RPMs upgraded as of today?”. In my thinking, an introspection of an already-built image is a distinct second best to enforcing policy, and signing, as an integral part of a build pipeline.)

That said, for an introspection workflow, skopeo copy is actually the correct tool:

• (If the input is a tagged and not a digested reference, use skopeo inspect to determine a specific digest; from now on, refer to the input image only by digest, to ensure everything refers exactly to the same artifact.)
• Inspect the image, however necessary.
• Then use skopeo copy --sign-by* $source@$digest $destination:$tag. This
  • Ensures that exactly the same binary artifact that was inspected was signed (unlike podman pull && podman inspect && $otherTool inspect && podman push)
  • As a special property of that, because the original binary artifact is used, this can preserve the original vendor’s signatures, maintaining nonrepudiation and an audit trail. (Unlike podman pull && podman inspect && $otherTool inspect && podman push)
  • By using a separate repo, only publishes the image to $destination after it is known to be usable, reducing user confusion.

And again, this could certainly be an integrated part of a larger application that calls a Go library, not necessarily invoking the skopeo command.

It's very likely that the user wants to sign the image in a specific way, which requires skopeo to know and support all the possible options by which you can sign an image (keyless, KMS-based, etc). Do we really want to carry all that into skopeo?

The set of features is definitely going to be limited to something we can understand, maintain, test against, and support. Other use cases are going to be better served with specialized tools, maybe built on top of c/image, maybe on top of github.com/sigstore/sigstore, maybe a CLI like Cosign.

[Dentrax]

why would someone do that with skopeo as part of the copy?

Calling cosign binary right after copy of an image, leads to a slight time window in which it could have tampered. I think signing should be inherent to the image copy process.

One use-case example is that companies runs tons of images in the production. If you want to enforce signed image policy in your clusters using a Policy Engine, you have to find all the public keys to verify against with. Instead we could mark the image simply one signature to ensure: "yes, we copied that image and validated it's signature, we trust this image to run in clusters. (Its could be scanned, audited, etc.)

Do we really want to carry all that into skopeo?

I completely understand the concern here. We don't want to make skopeo an image signing tool by adding bunch of features. So we should be careful here.

[dmesser]

I get the argument that currently the gap in cosign to sign local images is a problem from a security point of view. However the use case where this is important IMHO is not where skopeo copy would be used but where the image is built locally, tested, attested to and then signed ideally before pushed to a registry.

In your example @mtrmac you speak about an internal testing repo. So cosgin could totally be used to sign that image but not upload the signature (--upload=false) and then push the signature and image to the target registry. I appreciate the correctness of skopeo copy in relying on the digest, same as cosign.
But it the latter has way more options that the limited support of skopeo's --sign-by-sigstore-private-key, e.g. KMS which is likely in place in larger environments because local file-based key management is extremely cumbersome. I would expect the majority of users to rely on that rather on skopeo with likely outdated imports of cosign. Hence I am wondering why even accept all the maintainer responsibility to keep cosign support updates and all the troubles that come with this heavy-weight import. But I can also see how you would need to do that anyway for the verification part of images signed in Sigstore style (which I would still +1), so I guess I can rest my case 😄

@Dentrax This interesting, because I heard exactly the opposite. In a zero-trust environment customers don't want to trust the machine that does the copy. They want all the images to be signed by the vendor already and trust the vendors keys for full end2end accountability and integrity. I think cosign's effort to move towards OIDC identity based trust

[mtrmac]

However the use case where this is important IMHO is not where skopeo copy would be used but where the image is built locally, tested, attested to and then signed ideally before pushed to a registry.

That’s just not how it works in the GitHub.com/containers ecosystem. Locally-built images in c/storage don’t have a compressed representation that can be signed and then pushed; they are compressed during the push, and that’s the earliest reasonable time to sign them (before that, an uncompressed representation would be signed, and it would have to also be published uncompressed in order not to break the signature).

Sure, there’s some nuance to this (a single-purpose local registry, or pushing to a staging dir: with compression), but the base case most users should be thinking about is “build an image, and sign as a part of the first push to a registry”. Experts to signing and attestation might have a different design but they would also know why they want to deviate from the base case.

---

But it the latter has way more options that the limited support of skopeo's --sign-by-sigstore-private-key, e.g. KMS which is likely in place in larger environments because local file-based key management is extremely cumbersome. I would expect the majority of users to rely on that rather on skopeo with likely outdated imports of cosign.

(Again, I expect the majority of users to rely on Podman rather than Skopeo, which makes all this focus on Skopeo a bit surprising.)

Hence I am wondering why even accept all the maintainer responsibility to keep cosign support updates and all the troubles that come with this heavy-weight import.

*shrug* The software exists. I don’t see arguing now that it shouldn’t have been written as all that productive.

• We need a signing workflow that is simple to understand and secure by default. I argue that podman push --sign-by, and skopeo copy, can be that workflow; cosign sign seems to be one but in practice almost never is (because the registry becomes a risk, and because the simplest way to use it exposes an unsigned image to users, causing a disruption to consumers that enforce signatures). Of course that could be fixed in Cosign.
• “Outdated imports of cosign” are not really a concern. First, the signatures are a long-term artifact and they must be stable over time. WRT keeping up with security vulnerabilities, the much smaller scale actually makes that much more manageable.
• The scope of support/maintenance any team, or vendor, is always smaller than the whole universe of software that exists. The alternative of Podman somehow fully supporting all 80 MB of /usr/bin/cosign, and all the features that will ever be added, just isn’t what would happen. There will certainly be users that will want to use /usr/bin/cosign directly, rather than Podman, and we’re perfectly fine with that.

Specific RFEs and pointing out functionality that is critical to support (both for signing and verification) could be useful (in separate issues, in the c/image/ repo, please). “This feature shouldn’t exist” is much less so.

[mtrmac]

@Dentrax This interesting, because I heard exactly the opposite. In a zero-trust environment customers don't want to trust the machine that does the copy. They want all the images to be signed by the vendor already and trust the vendors keys for full end2end accountability and integrity.

policy.json allows users (if they want to take all that extra effort) to have both.

[dmesser]

To be honest I didn't know that skopeo had the sign-by variant for SigStore style signature until today. I assumed it was in the works but it seems like this issue hasn't been updated with that progress. So yeah, it's a moot point now.

For what it's worth, since cosign signs the digest it can't currently not sign before it is in the registry, for the reasons you say. So I guess what podman/skopeo are doing is simply cutting the time where an unsigned image exists in the registry down vs. the time it takes for a pipeline to run podman/skopeo and then cosign. Let me know if this understanding is incorrect. The thinking on the cosign side seems to go towards a solution that uses a local OCI layout for this: sigstore/cosign#596 (comment)

[mtrmac]

So I guess what podman/skopeo are doing is simply cutting the time where an unsigned image exists in the registry down vs. the time it takes for a pipeline to run podman/skopeo and then cosign.

Thanks, that’s actually a good point: the c/image/docker destination implementation does write the manifest before the signatures, so there is a time where the image is visible on the destination unsigned. But that’s not actually inherent to the design, let’s track fixing it in containers/image#1623 .

[mtrmac]

The last update for this issue is #1533 (comment) .

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mtrmac]

Skopeo 1.11 now supports

• Creating signatures using Fulcio-generated certificates
• Uploading created signatures to Rekor
• In policy.json, enforcing Rekor log presence (off-line only, based on a SET in the signature)
• In policy.json accepting Fulcio-generated certificates (with a very restricted issuer/subject matching capability).

That’s about as far as I currently want to take this; any future enhancements would depend on user feedback and production experience. (#1704 is still mostly outstanding.)