Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify base image signatures #356

Open
imjasonh opened this issue May 7, 2021 · 7 comments 路 May be fixed by #918
Open

Verify base image signatures #356

imjasonh opened this issue May 7, 2021 · 7 comments 路 May be fixed by #918

Comments

@imjasonh
Copy link
Member

imjasonh commented May 7, 2021

If ko is instructed to build an image based on another image, it would be nice to verify signatures on those base images.

It just so happens the default base image, distroless, is signed. 馃帀

  • What should the CLI surface look like for enabling/disabling signing base images?
  • Should we start by only signing (edit) verifying the default distroless base image?
  • How should we get the keys to verify the signatures?
  • How should users configure the keys to verify arbitrary images' signatures?

cc @dlorenc

@mattmoor
Copy link
Collaborator

mattmoor commented Aug 6, 2021

I'd propose a similar two-release strategy to things we've done previously (e.g. requiring ko://):

  1. Start to verify things by default, be noisy when they are NOT (link to how), and have an option to fail,
  2. Require verification by default, fail closed, and have an option to allow unsigned images.

@imjasonh
Copy link
Member Author

imjasonh commented Aug 6, 2021

I don't think there are enough signed base images in the wild yet to justify failing closed. A big scary warning sgtm though.

Since we control the default and the default is signed, I do think it's safe to fail closed if we detect distroless isn't signed.

@mattmoor
Copy link
Collaborator

mattmoor commented Aug 6, 2021

@imjasonh Yeah 2 doesn't have to be the next release, but we should indicate in our scary warning that this is the direction we intend to head. We can always wake up smarter and walk that back, but the more people we can scare into taking action the better IMO

@mattmoor
Copy link
Collaborator

mattmoor commented Aug 6, 2021

To pontificate a bit. At least for me, ko and other "last mile" build tools exist to try to realize the best practices (e.g. nonroot, multi-arch, minimal base).

While the breaking nature of this sort of against the idea of back-compatibility, I think the north star we want to remain compatible with is that we leave folks with containers that are as good as we can possibly produce given the evolution of best practices over time.

I wonder if it would be worth including a bit about that as a sort of "mission statement" or "philosophy" for the tool in the main README.md? Sort of like we added the principles section to GGCR.

</pontification>

@mattmoor
Copy link
Collaborator

mattmoor commented Aug 6, 2021

This applies similarly to: #357

@imjasonh
Copy link
Member Author

imjasonh commented Aug 6, 2021

Taking a stab at the latter two points:

.ko.yaml:

verifySignatures:
  "alpine:1.23":
  - publicKey: https://some.url/path/to/key.pub
    annotations:
      key: value
      another: value2

If any image is based on alpine:1.23 then ko will verify that it has a signature signed using the private key it can find at the URL, and has those annotations.

The default distroless base image has an assumed - publicKey: https://github.com/GoogleContainerTools/distroless/blob/main/cosign.pub as documented here.

Users can configure other signatures that need to be verified for distroless, or any base image they use.

Q: how should ephemeral OIDC keys be configured to be verified?

If all base images involved in an invocation of ko are signed and verified, no warning. If any are missing a signature, warn. If any have signatures that can't be verified as configured, fail loudly.

wdyt?

mattmoor added a commit to mattmoor/ko that referenced this issue Sep 12, 2021
When `COSIGN_EXPERIMENTAL=true`, this will verify base images, and sign produced images using the KEYLESS flow.

Fixes: ko-build#357
Fixes: ko-build#356
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 13, 2021
When `COSIGN_EXPERIMENTAL=true`, this will verify base images, and sign produced images using the KEYLESS flow.

Fixes: ko-build#357
Fixes: ko-build#356
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 13, 2021
When `COSIGN_EXPERIMENTAL=true`, this will verify base images, and sign produced images using the KEYLESS flow.

Fixes: ko-build#357
Fixes: ko-build#356
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 14, 2021
This change enables folks to start verifying their base images are signed against Fulcio by specifying `COSIGN_EXPERIMENTAL=true`, which will ensure each digest we intend to base an image on is verified.

Unless `COSIGN_EXPERIMENTAL=true` is enabled, this change will have no effect.

When `COSIGN_EXPERIMENTAL=true` is enabled, this change will verify base image digests, and emit a `WARNING:` when they are not signed (the `gcr.io/distroless` images are now signed!).

When `COSIGN_EXPERIMENTAL=true` is enabled *AND* `--strict` is passed, then failure to verify a base image digest will fail the `ko` command.

Related: ko-build#356
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 14, 2021
This change enables folks to start verifying their base images are signed against Fulcio by specifying `COSIGN_EXPERIMENTAL=true`, which will ensure each digest we intend to base an image on is verified.

Unless `COSIGN_EXPERIMENTAL=true` is enabled, this change will have no effect.

When `COSIGN_EXPERIMENTAL=true` is enabled, this change will verify base image digests, and emit a `WARNING:` when they are not signed (the `gcr.io/distroless` images are now signed!).

When `COSIGN_EXPERIMENTAL=true` is enabled *AND* `--strict` is passed, then failure to verify a base image digest will fail the `ko` command.

Related: ko-build#356
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 14, 2021
This change enables folks to start verifying their base images are signed against Fulcio by specifying `COSIGN_EXPERIMENTAL=true`, which will ensure each digest we intend to base an image on is verified.

Unless `COSIGN_EXPERIMENTAL=true` is enabled, this change will have no effect.

When `COSIGN_EXPERIMENTAL=true` is enabled, this change will verify base image digests, and emit a `WARNING:` when they are not signed (the `gcr.io/distroless` images are now signed!).

When `COSIGN_EXPERIMENTAL=true` is enabled *AND* `--strict` is passed, then failure to verify a base image digest will fail the `ko` command.

Related: ko-build#356
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 14, 2021
This change enables folks to start verifying their base images are signed against Fulcio by specifying `COSIGN_EXPERIMENTAL=true`, which will ensure each digest we intend to base an image on is verified.

Unless `COSIGN_EXPERIMENTAL=true` is enabled, this change will have no effect.

When `COSIGN_EXPERIMENTAL=true` is enabled, this change will verify base image digests, and emit a `WARNING:` when they are not signed (the `gcr.io/distroless` images are now signed!).

When `COSIGN_EXPERIMENTAL=true` is enabled *AND* `--strict` is passed, then failure to verify a base image digest will fail the `ko` command.

Related: ko-build#356
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 20, 2021
This change enables folks to start verifying their base images are signed against Fulcio by specifying `COSIGN_EXPERIMENTAL=true`, which will ensure each digest we intend to base an image on is verified.

Unless `COSIGN_EXPERIMENTAL=true` is enabled, this change will have no effect.

When `COSIGN_EXPERIMENTAL=true` is enabled, this change will verify base image digests, and emit a `WARNING:` when they are not signed (the `gcr.io/distroless` images are now signed!).

When `COSIGN_EXPERIMENTAL=true` is enabled *AND* `--strict` is passed, then failure to verify a base image digest will fail the `ko` command.

Related: ko-build#356
@mattmoor
Copy link
Collaborator

Revisiting this and adding a few thoughts after mulling this over a bit this morning.

I think that rather than inventing our own ~API for verification, we should (similar to the Go releaser precedent) align around the sigstore/policy-controller API for verification, which closes a lot of open questions (including rollout).

I think there are a few key elements of configuration:

  1. Global configuration (e.g. what to do for uncovered images?)
  2. Policy configuration (e.g. what to check for which images?)

For 1. I think we want a little bit of configuration that exposes the knobs that policy-controller defines within its configmap here. For example, the no-match-policy setting can warn on uncovered images (probably what we should start with as the default!), but folks can configure this to allow (match current behavior) or deny (to block things uncovered by policy).

For 2. I think it makes sense to align with the ClusterImagePolicy API shape (and TrustRoot coming soon!) rather than inventing our own thing. This allows us access to the following authorities properties:

  1. Statically trust certain images with static: {action: pass},
  2. Statically distrust certain images with static: {action: fail},
  3. Verify signatures with fixed keys using key: {data: ... }
  4. Verify signatures with KMS keys using key: {kms: gcpkms://... }
  5. Verify signatures with CAs using keyless: { ... }
  6. Verify signature with TSAs using rfc3161timestamp: { ... }
  7. Verify transparency log inclusion with ctlog: { ... }
  8. Verify attestation signatures in all of the above ways
  9. Verify individual attestation predicates using Cue or Rego
  10. Verify N of M-like properties with top-level Cue or Rego policies.

This is significantly more capable that what you can easily do via cosign verify directly today (benefits of API over CLI!), will only grow in capability over time, and will always exceed the capabilities of anything homegrown we try to define ourselves. 馃槄

Before foraying into the next section, let me ACK the dependency tree nightmare we have inherited from sigstore/cosign, which @imjasonh and others are working to fix with sigstore/sigstore-go.

sigstore/policy-controller is also intended to be used as a library, and exposes a ValidatePolicy method to allow folks to validate policies against image digests directly. It currently has something we call the policy-tester which is a toy CLI for playing with policies, and demonstrates some of how you can call this method directly with a policy today.


As a total strawperson proposal for this:

verification:
  no-match-policy: ...
  policies:
  - data: # inline policy
  - url: # URL for a policy
  - path: # Path to policy (support directories as well, like kubectl/ko -f)

I think we should default this to something like:

verification:
  no-match-policy: warn
  policies:
  # a default policy covering our default base image (cgr.dev/chainguard/static*)
  - data: ...
  # Users can put CIPs/Secrets/TrustRoots under .ko/ (alongside .ko.yaml)
  - path: .ko/

Folks can get the current behavior with:

verification:
  # Allow uncovered images
  no-match-policy: allow
  # Specify no policies
  policies: []

mattmoor added a commit to mattmoor/ko that referenced this issue Jan 1, 2023
mattmoor added a commit to mattmoor/ko that referenced this issue Jan 1, 2023
mattmoor added a commit to mattmoor/ko that referenced this issue Jan 1, 2023
mattmoor added a commit to mattmoor/ko that referenced this issue Jan 2, 2023
馃巵 This implements the very basics from my recent comment on ko-build#356.

This still needs a bunch of work, but some playing with it and `go run` seem to demonstrate what I'd hope for.

Things I'd like to do still:
 - Add validation that warns/errors if aspects of CIP irrelevant outside K8s are used (e.g. `match:`, `includeSpec:`),
 - Expand unit test coverage significantly,
 - Move `pkg/policy` upstream to `sigstore/policy-controller` to facilitate using this in other projects with similar ease (e.g. kaniko, buildpacks, ...)

/kind feature
@mattmoor mattmoor linked a pull request Jan 2, 2023 that will close this issue
4 tasks
mattmoor added a commit to mattmoor/ko that referenced this issue Jan 2, 2023
馃巵 This implements the very basics from my recent comment on ko-build#356.

This still needs a bunch of work, but some playing with it and `go run` seem to demonstrate what I'd hope for.

Things I'd like to do still:
 - Add validation that warns/errors if aspects of CIP irrelevant outside K8s are used (e.g. `match:`, `includeSpec:`),
 - Move `pkg/policy` upstream to `sigstore/policy-controller` to facilitate using this in other projects with similar ease (e.g. kaniko, buildpacks, ...)

/kind feature
mattmoor added a commit to mattmoor/policy-controller that referenced this issue Jan 2, 2023
馃巵 This change factors a new small library `./pkg/policy` which is intended to streamline incorporating CIP validation into downstream tooling.

For a (much) more verbose explanation see [here](ko-build/ko#356 (comment)), but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base
images in build tools such as `ko` or `kaniko`.  The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior
to consumption.

For example, I might write the following `.ko.yaml`:
```yaml
verification:
  noMatchPolicy: deny
  policies:
  - data: |
      # inline policy
  - url: https://github.com/foo/bar/blobs/main/POLICY.yaml
```

With this library, it is likely <100 LoC to add base image policy verification to `ko`, and significantly simplifies our own `policy-tester` which has spaghetti code replicating some of this functionality.

/kind feature

Signed-off-by: Matt Moore <mattmoor@chainguard.dev>
mattmoor added a commit to mattmoor/policy-controller that referenced this issue Jan 2, 2023
馃巵 This change factors a new small library `./pkg/policy` which is intended to streamline incorporating CIP validation into downstream tooling.

For a (much) more verbose explanation see [here](ko-build/ko#356 (comment)), but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base
images in build tools such as `ko` or `kaniko`.  The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior
to consumption.

For example, I might write the following `.ko.yaml`:
```yaml
verification:
  noMatchPolicy: deny
  policies:
  - data: |
      # inline policy
  - url: https://github.com/foo/bar/blobs/main/POLICY.yaml
```

With this library, it is likely <100 LoC to add base image policy verification to `ko`, and significantly simplifies our own `policy-tester` which has spaghetti code replicating some of this functionality.

/kind feature

Signed-off-by: Matt Moore <mattmoor@chainguard.dev>
mattmoor added a commit to mattmoor/policy-controller that referenced this issue Jan 3, 2023
馃巵 This change factors a new small library `./pkg/policy` which is intended to streamline incorporating CIP validation into downstream tooling.

For a (much) more verbose explanation see [here](ko-build/ko#356 (comment)), but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base
images in build tools such as `ko` or `kaniko`.  The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior
to consumption.

For example, I might write the following `.ko.yaml`:
```yaml
verification:
  noMatchPolicy: deny
  policies:
  - data: |
      # inline policy
  - url: https://github.com/foo/bar/blobs/main/POLICY.yaml
```

With this library, it is likely <100 LoC to add base image policy verification to `ko`, and significantly simplifies our own `policy-tester` which has spaghetti code replicating some of this functionality.

/kind feature

Signed-off-by: Matt Moore <mattmoor@chainguard.dev>
mattmoor added a commit to mattmoor/policy-controller that referenced this issue Jan 3, 2023
馃巵 This change factors a new small library `./pkg/policy` which is intended to streamline incorporating CIP validation into downstream tooling.

For a (much) more verbose explanation see [here](ko-build/ko#356 (comment)), but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base
images in build tools such as `ko` or `kaniko`.  The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior
to consumption.

For example, I might write the following `.ko.yaml`:
```yaml
verification:
  noMatchPolicy: deny
  policies:
  - data: |
      # inline policy
  - url: https://github.com/foo/bar/blobs/main/POLICY.yaml
```

With this library, it is likely <100 LoC to add base image policy verification to `ko`, and significantly simplifies our own `policy-tester` which has spaghetti code replicating some of this functionality.

/kind feature

Signed-off-by: Matt Moore <mattmoor@chainguard.dev>
mattmoor added a commit to mattmoor/policy-controller that referenced this issue Jan 3, 2023
馃巵 This change factors a new small library `./pkg/policy` which is intended to streamline incorporating CIP validation into downstream tooling.

For a (much) more verbose explanation see [here](ko-build/ko#356 (comment)), but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base
images in build tools such as `ko` or `kaniko`.  The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior
to consumption.

For example, I might write the following `.ko.yaml`:
```yaml
verification:
  noMatchPolicy: deny
  policies:
  - data: |
      # inline policy
  - url: https://github.com/foo/bar/blobs/main/POLICY.yaml
```

With this library, it is likely <100 LoC to add base image policy verification to `ko`, and significantly simplifies our own `policy-tester` which has spaghetti code replicating some of this functionality.

/kind feature

Signed-off-by: Matt Moore <mattmoor@chainguard.dev>
mattmoor added a commit to mattmoor/policy-controller that referenced this issue Jan 3, 2023
馃巵 This change factors a new small library `./pkg/policy` which is intended to streamline incorporating CIP validation into downstream tooling.

For a (much) more verbose explanation see [here](ko-build/ko#356 (comment)), but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base
images in build tools such as `ko` or `kaniko`.  The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior
to consumption.

For example, I might write the following `.ko.yaml`:
```yaml
verification:
  noMatchPolicy: deny
  policies:
  - data: |
      # inline policy
  - url: https://github.com/foo/bar/blobs/main/POLICY.yaml
```

With this library, it is likely <100 LoC to add base image policy verification to `ko`, and significantly simplifies our own `policy-tester` which has spaghetti code replicating some of this functionality.

/kind feature

Signed-off-by: Matt Moore <mattmoor@chainguard.dev>
mattmoor added a commit to mattmoor/ko that referenced this issue Jan 3, 2023
馃巵 This implements the very basics from my recent comment on ko-build#356.

This still needs the upstream changes to land in a release, but illustrates how the integration will work.

/kind feature
mattmoor added a commit to mattmoor/ko that referenced this issue Jan 3, 2023
馃巵 This implements the very basics from my recent comment on ko-build#356.

This still needs the upstream changes to land in a release, but illustrates how the integration will work.

/kind feature
mattmoor added a commit to mattmoor/ko that referenced this issue Jan 3, 2023
馃巵 This implements the very basics from my recent comment on ko-build#356.

This still needs the upstream changes to land in a release, but illustrates how the integration will work.

/kind feature
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants