Error uploading artifact to container registry when using Google Artifact Registry

[capri-xiyue]

Does actions/attest-build-provenance@v1 support Google Artifact Registry?

The build-push step successfully pushed image to Google Artifact Registry with this workflow at Build the integration test server container and push to the registry. It's failing at the attestation step with Error: OCIError: Error uploading artifact to container registry Error: Error fetching https://us-central1-docker.pkg.dev/v2/<repoA>/<imageA>/manifests/sha256:xxxxxxx - expected 200, received 404

Here's my job step.

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      attestations: write
      packages: write
    env:
      file: './python-hello-world/Dockerfile'
      run_from: './python-hello-world'

    steps:
      - name: 'Checkout'
        uses: 'actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11' # ratchet:actions/checkout@v4

      # Authenticate to GCP with WIF
      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@55bd3a7c6e2ae7cf1877fd1ccb9d54c0503c457c' # ratchet:google-github-actions/auth@v2
        with:
          workload_identity_provider: '${{ env.WIF_PROVIDER }}'
          service_account: '${{ env.WIF_SERVICE_ACCOUNT }}'
          token_format: 'access_token'

      - name: 'Authenticate to Artifact Registry'
        uses: 'docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d' 
        with:
          username: 'oauth2accesstoken'
          password: '${{ steps.auth.outputs.access_token }}'
          registry: 'us-central1-docker.pkg.dev'

      # Build and Push Image
      - name: 'Build the integration test server container and push to the registry'
        id: push
        uses: 'docker/build-push-action@4a13e500e55cf31b7a5d59a38ab2040ab0f42f56'
        with:
          push: true
          tags: '${{ env.DOCKER_REPO }}/${{ env.IMAGE_NAME }}:${{env.DOCKER_TAG}}'
          file: '${{ env.file }}'
          context: '${{ env.run_from }}'

      - name: Generate artifact attestation
        uses: actions/attest-build-provenance@v1
        with:
          subject-name: '${{ env.DOCKER_REPO }}/${{ env.IMAGE_NAME }}' // example is `us-central1-docker.pkg.dev/repoA/imageA`
          subject-digest: '${{ steps.push.outputs.digest }}'
          push-to-registry: true

Looks like actions/attest-build-provenance@v1 doesn't use the auth I provided in previous step automatically and it doesn't expose a way to configure the auth needed for Google Artifact Registry. The previous steppush which push the image to the Google Artifact Registry works, so I'm pretty sure I'm providing the correct auth for Google Artifact Registry.

At the same time, I also run curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://us-central1-docker.pkg.dev/v2/<repoA>/<imageA>/manifests/sha256:xxxxxxx for the url throw in the error message Error fetching https://us-central1-docker.pkg.dev/v2/<repoA>/<imageA>/manifests/sha256:xxxxxxx - expected 200, received 404, and the curl works which shows the https://us-central1-docker.pkg.dev/v2/<repoA>/<imageA>/manifests/sha256:xxxxxxx is a valid url.

Thank you for your help!

[bdehamer]

@capri-xiyue We have done some testing w/ Google Artifact Registry (here's a sample run you can look at).

I suspect that you are correct -- the underlying issue is probably auth related. The one difference I see between your workflow and my sample is that you are using the google-github-actions/auth action. I'll incorporate that into my sample and do some testing to see if I can figure out what is going on.

[capri-xiyue]

@bdehamer
Looks like the main difference is that you are using Service account based authentication(https://github.com/docker/login-action?tab=readme-ov-file#service-account-based-authentication-1) and I'm using Workload identity federation(https://github.com/docker/login-action?tab=readme-ov-file#google-artifact-registry-gar)

[capri-xiyue]

@bdehamer
I made it work after adding such actions

 - name: Set up QEMU
    uses: docker/setup-qemu-action@v3
 - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v3

But I have some questions, looks like it will generate 5 images for one run. Could you explain what each image is for?

[bdehamer]

@capri-xiyue I think I know why this was failing for you before adding the docker/setup-buildx-action . . . but first lemme discuss the "5 images".

I assume you're talking about the the 5 entries you see if you do something like the following:

$ gcloud artifacts docker images list docker.pkg.dev/foo/bar --include-tags
Listing items under project XXXX, location us-west1, repository foo.

IMAGE                   DIGEST                                                                   TAGS                                                                     CREATE_TIME          UPDATE_TIME          SIZE
docker.pkg.dev/foo/bar  sha256:329a93ba4c9b1911ed73d8815fba2bebfc102bfcbf0ef95b1f5feca08a1ee3a9  latest                                                                   2024-05-14T12:44:13  2024-05-14T12:44:13
docker.pkg.dev/foo/bar  sha256:57fa8d2dcfc3929aadb784fe60463751727e9a6dfd61a3a1c3aa04bb4f8b635a                                                                           2024-05-14T12:44:16  2024-05-14T12:44:16  10538
docker.pkg.dev/foo/bar  sha256:d531c7f8b5eb0051c9cad575c3195b6a809ce933b091ab9405a921954071bd5f  sha256-329a93ba4c9b1911ed73d8815fba2bebfc102bfcbf0ef95b1f5feca08a1ee3a9  2024-05-14T12:44:16  2024-05-14T12:44:16
docker.pkg.dev/foo/bar  sha256:d67e45b3a36ee3d13f64dafa2a69c4762c3f82d195c4226bf72613e4c6fb9ddd                                                                           2024-05-14T12:44:12  2024-05-14T12:44:12  1050
docker.pkg.dev/foo/bar  sha256:e7276394149582ad4cd184fec5cb65b5cee36108ca0b275ce5c5279b04ca8be3  

or something similar in the Google Cloud Console:

What you're seeing here are not 5 images, but rather 5 different manifests which have been stored in the repository. Only one of these actually represents your image, but let's walk through each of them:

1. The entry shown above with the tag latest is the index for the image which was built. If we crack it open we'll see two entries listed in the index:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "manifests": [
         {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:e7276394149582ad4cd184fec5cb65b5cee36108ca0b275ce5c5279b04ca8be3",
            "size": 476,
            "platform": {
               "architecture": "amd64",
               "os": "linux"
            }
         },
         {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:d67e45b3a36ee3d13f64dafa2a69c4762c3f82d195c4226bf72613e4c6fb9ddd",
            "size": 565,
            "annotations": {
               "vnd.docker.reference.digest": "sha256:e7276394149582ad4cd184fec5cb65b5cee36108ca0b275ce5c5279b04ca8be3",
               "vnd.docker.reference.type": "attestation-manifest"
            },
            "platform": {
               "architecture": "unknown",
               "os": "unknown"
            }
         }
      ]
   }
   

   The first listed manifest is for the image itself (note the architecture/os information which is attached. The second manifest points to a provenance statement which was generated by the docker build command and associated with this image (note the "attestation-manifest" annotation which hints at the purpose for this manifest).

   This index serves to bind these two entities together -- the image and the provenance statement for that image.

2. Retrieving the first manifest listed in the index above (sha256:e72763) yields the following:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "config": {
         "mediaType": "application/vnd.oci.image.config.v1+json",
         "digest": "sha256:355a3507efda559fc3d75033a3b444af9a18690da2eb01b8bface7a0d890ad40",
         "size": 436
      },
      "layers": [
         {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:5f06459796339119f39b30d9a4dbcda7d50ab0bc3d5b1a1c1fb5c0c070328ac4",
            "size": 137
         }
      ]
   }

   This image manifest contains all the information about the OCI image itself -- this is the information that a client would use to actually pull the bytes of the image.

3. The second manifest listed in the index (sha:d67e45) contains the following:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "config": {
         "mediaType": "application/vnd.oci.image.config.v1+json",
         "digest": "sha256:8b77ce151e3e9738811376727e094b907df17beea97a57cf96733c2e47ae93dc",
         "size": 167
      },
      "layers": [
         {
            "mediaType": "application/vnd.in-toto+json",
            "digest": "sha256:56d3363720aa0e22487996db0f4ee7adb9e22f2e13b672119a59c0a24565636d",
            "size": 883,
            "annotations": {
               "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
            }
         }
      ]
   }

   The layer information here points to the contents of the provenance statement (again, this is the provenance statement generated by docker build). If you retrieve the blob at sha:56d336 you'd get the JSON-encoded provenance statement.

4. The first three manifests were all generated by the docker build command itself, while the last two come from the actions/attest-build-provenance action.

   The manifest tagged with sha256-329a93... is the attestation index created by attest-build-provenance. Since we're generating attestations for an already-existing image, we need to push them in a separate manifest. The sha256-... tag is how we relate our attestation index back to the original image (note that the digest in the tag here matches the digest associated with the latest tag).

   The content of this index (sha:d531c7) looks something like this:

   {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "schemaVersion": 2,
      "manifests": [
         {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:57fa8d2dcfc3929aadb784fe60463751727e9a6dfd61a3a1c3aa04bb4f8b635a",
            "size": 812,
            "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json",
            "annotations": {
               "org.opencontainers.image.created": "2024-05-14T19:44:14.206Z",
               "dev.sigstore.bundle.content": "dsse-envelope",
               "dev.sigstore.bundle.predicateType": "https://slsa.dev/provenance/v1"
            }
         }
      ]
   }

   In this case, the index contains reference to a single attestation. If we had generated multiple attestations for this image each of them would be listed here -- providing a mechanism for discovering all of the externally-generated attestations (worth noting that this wasn't a scheme we invented, it's part of the OCI specification describing how to associate arbitrary artifacts with container images).

5. The final manifest (sha:57fa8d) is the one identified in the attestation index:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json",
      "config": {
         "mediaType": "application/vnd.oci.empty.v1+json",
         "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
         "size": 2
      },
      "layers": [
         {
            "mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
            "digest": "sha256:51301ee43b449df2bd645addcab872c28759706d1caaa4b8bcc581dcd89fa3d7",
            "size": 10536
         }
      ],
      "subject": {
         "mediaType": "application/vnd.oci.image.index.v1+json",
         "digest": "sha256:329a93ba4c9b1911ed73d8815fba2bebfc102bfcbf0ef95b1f5feca08a1ee3a9",
         "size": 855
      },
      "annotations": {
         "org.opencontainers.image.created": "2024-05-14T19:44:14.206Z",
         "dev.sigstore.bundle.content": "dsse-envelope",
         "dev.sigstore.bundle.predicateType": "https://slsa.dev/provenance/v1"
      }
   }

   This points to the blob containing the JSON-encoded build provenance attestation generated by attest-build-provenance.

[bdehamer]

Regarding the original error you reported . . .

Error: OCIError: Error uploading artifact to container registry Error: Error fetching https://us-central1-docker.pkg.dev/v2///manifests/sha256:xxxxxxx - expected 200, received 404

I'm pretty sure that this is a bug in the attest-build-provenance code.

When you add the docker/setup-buildx-action action to your workflow it changes the behavior of the docker/build-push-action -- when these two actions are paired, build-push-action will automatically generate/push an provenance statement for the image build steps. As described above, this means that the tagged image will point to an index (w/ media type application/vnd.oci.image.index.v1+json).

When setup-buildx-action is NOT present, build-push-action does NOT generate a provenance statement and the repository tag points directly to an image manifest (w/ media type application/vnd.docker.distribution.manifest.v2+json).

Currently, the attest-build-provenance code does not identify application/vnd.docker.distribution.manifest.v2+json as an acceptable media type when attempting to resolve the image tag which results in the 404 error you're seeing.

We simply need to update our code to identify application/vnd.docker.distribution.manifest.v2+json as an expected manifest type.

[capri-xiyue]

@capri-xiyue I think I know why this was failing for you before adding the docker/setup-buildx-action . . . but first lemme discuss the "5 images".

I assume you're talking about the the 5 entries you see if you do something like the following:

$ gcloud artifacts docker images list docker.pkg.dev/foo/bar --include-tags
Listing items under project XXXX, location us-west1, repository foo.

IMAGE                   DIGEST                                                                   TAGS                                                                     CREATE_TIME          UPDATE_TIME          SIZE
docker.pkg.dev/foo/bar  sha256:329a93ba4c9b1911ed73d8815fba2bebfc102bfcbf0ef95b1f5feca08a1ee3a9  latest                                                                   2024-05-14T12:44:13  2024-05-14T12:44:13
docker.pkg.dev/foo/bar  sha256:57fa8d2dcfc3929aadb784fe60463751727e9a6dfd61a3a1c3aa04bb4f8b635a                                                                           2024-05-14T12:44:16  2024-05-14T12:44:16  10538
docker.pkg.dev/foo/bar  sha256:d531c7f8b5eb0051c9cad575c3195b6a809ce933b091ab9405a921954071bd5f  sha256-329a93ba4c9b1911ed73d8815fba2bebfc102bfcbf0ef95b1f5feca08a1ee3a9  2024-05-14T12:44:16  2024-05-14T12:44:16
docker.pkg.dev/foo/bar  sha256:d67e45b3a36ee3d13f64dafa2a69c4762c3f82d195c4226bf72613e4c6fb9ddd                                                                           2024-05-14T12:44:12  2024-05-14T12:44:12  1050
docker.pkg.dev/foo/bar  sha256:e7276394149582ad4cd184fec5cb65b5cee36108ca0b275ce5c5279b04ca8be3  

or something similar in the Google Cloud Console:

What you're seeing here are not 5 images, but rather 5 different manifests which have been stored in the repository. Only one of these actually represents your image, but let's walk through each of them:

1. The entry shown above with the tag latest is the index for the image which was built. If we crack it open we'll see two entries listed in the index:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "manifests": [
         {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:e7276394149582ad4cd184fec5cb65b5cee36108ca0b275ce5c5279b04ca8be3",
            "size": 476,
            "platform": {
               "architecture": "amd64",
               "os": "linux"
            }
         },
         {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:d67e45b3a36ee3d13f64dafa2a69c4762c3f82d195c4226bf72613e4c6fb9ddd",
            "size": 565,
            "annotations": {
               "vnd.docker.reference.digest": "sha256:e7276394149582ad4cd184fec5cb65b5cee36108ca0b275ce5c5279b04ca8be3",
               "vnd.docker.reference.type": "attestation-manifest"
            },
            "platform": {
               "architecture": "unknown",
               "os": "unknown"
            }
         }
      ]
   }

   The first listed manifest is for the image itself (note the architecture/os information which is attached. The second manifest points to a provenance statement which was generated by the docker build command and associated with this image (note the "attestation-manifest" annotation which hints at the purpose for this manifest).
   This index serves to bind these two entities together -- the image and the provenance statement for that image.

2. Retrieving the first manifest listed in the index above (sha256:e72763) yields the following:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "config": {
         "mediaType": "application/vnd.oci.image.config.v1+json",
         "digest": "sha256:355a3507efda559fc3d75033a3b444af9a18690da2eb01b8bface7a0d890ad40",
         "size": 436
      },
      "layers": [
         {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:5f06459796339119f39b30d9a4dbcda7d50ab0bc3d5b1a1c1fb5c0c070328ac4",
            "size": 137
         }
      ]
   }

   This image manifest contains all the information about the OCI image itself -- this is the information that a client would use to actually pull the bytes of the image.

3. The second manifest listed in the index (sha:d67e45) contains the following:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "config": {
         "mediaType": "application/vnd.oci.image.config.v1+json",
         "digest": "sha256:8b77ce151e3e9738811376727e094b907df17beea97a57cf96733c2e47ae93dc",
         "size": 167
      },
      "layers": [
         {
            "mediaType": "application/vnd.in-toto+json",
            "digest": "sha256:56d3363720aa0e22487996db0f4ee7adb9e22f2e13b672119a59c0a24565636d",
            "size": 883,
            "annotations": {
               "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
            }
         }
      ]
   }

   The layer information here points to the contents of the provenance statement (again, this is the provenance statement generated by docker build). If you retrieve the blob at sha:56d336 you'd get the JSON-encoded provenance statement.

4. The first three manifests were all generated by the docker build command itself, while the last two come from the actions/attest-build-provenance action.
   The manifest tagged with sha256-329a93... is the attestation index created by attest-build-provenance. Since we're generating attestations for an already-existing image, we need to push them in a separate manifest. The sha256-... tag is how we relate our attestation index back to the original image (note that the digest in the tag here matches the digest associated with the latest tag).
   The content of this index (sha:d531c7) looks something like this:

   {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "schemaVersion": 2,
      "manifests": [
         {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:57fa8d2dcfc3929aadb784fe60463751727e9a6dfd61a3a1c3aa04bb4f8b635a",
            "size": 812,
            "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json",
            "annotations": {
               "org.opencontainers.image.created": "2024-05-14T19:44:14.206Z",
               "dev.sigstore.bundle.content": "dsse-envelope",
               "dev.sigstore.bundle.predicateType": "https://slsa.dev/provenance/v1"
            }
         }
      ]
   }

   In this case, the index contains reference to a single attestation. If we had generated multiple attestations for this image each of them would be listed here -- providing a mechanism for discovering all of the externally-generated attestations (worth noting that this wasn't a scheme we invented, it's part of the OCI specification describing how to associate arbitrary artifacts with container images).

5. The final manifest (sha:57fa8d) is the one identified in the attestation index:

   {
      "schemaVersion": 2,
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json",
      "config": {
         "mediaType": "application/vnd.oci.empty.v1+json",
         "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
         "size": 2
      },
      "layers": [
         {
            "mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
            "digest": "sha256:51301ee43b449df2bd645addcab872c28759706d1caaa4b8bcc581dcd89fa3d7",
            "size": 10536
         }
      ],
      "subject": {
         "mediaType": "application/vnd.oci.image.index.v1+json",
         "digest": "sha256:329a93ba4c9b1911ed73d8815fba2bebfc102bfcbf0ef95b1f5feca08a1ee3a9",
         "size": 855
      },
      "annotations": {
         "org.opencontainers.image.created": "2024-05-14T19:44:14.206Z",
         "dev.sigstore.bundle.content": "dsse-envelope",
         "dev.sigstore.bundle.predicateType": "https://slsa.dev/provenance/v1"
      }
   }

   This points to the blob containing the JSON-encoded build provenance attestation generated by attest-build-provenance.

Thanks for the details! Curious in such CUJ, do you know how does https://cli.github.com/manual/gh_attestation_verify works here?
I tried it, but I can't make it work. Can I verify the container image using the attestation pushed to registry? I also submitted a issue in cli/cli#9084.

[bdehamer]

When using the gh attestation verify command to verify an attestation associated with a container image, be sure to use the oci:// prefix on the image name (per https://cli.github.com/manual/gh_attestation_verify).

The oci:// prefix is used to signal that the artifact being verified is hosted in an OCI registry (and not a local file).

$ gh attestation verify oci://us-west1-docker.pkg.dev/foo/bar:latest  --owner foo

Loaded digest sha256:fa5feadae892be36eeca65fad5e05f71041242adf9cdafb8fa8be930f223777f for oci://us-west1-docker.pkg.dev/foo/bar:latest
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:fa5feadae892be36eeca65fad5e05f71041242adf9cdafb8fa8be930f223777f was attested by:
REPO     PREDICATE_TYPE                  WORKFLOW                                        
foo/bar  https://slsa.dev/provenance/v1  .github/workflows/oci-google.yml@refs/heads/main

[capri-xiyue]

When using the gh attestation verify command to verify an attestation associated with a container image, be sure to use the oci:// prefix on the image name (per https://cli.github.com/manual/gh_attestation_verify).

The oci:// prefix is used to signal that the artifact being verified is hosted in an OCI registry (and not a local file).

$ gh attestation verify oci://us-west1-docker.pkg.dev/foo/bar:latest  --owner foo

Loaded digest sha256:fa5feadae892be36eeca65fad5e05f71041242adf9cdafb8fa8be930f223777f for oci://us-west1-docker.pkg.dev/foo/bar:latest
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:fa5feadae892be36eeca65fad5e05f71041242adf9cdafb8fa8be930f223777f was attested by:
REPO     PREDICATE_TYPE                  WORKFLOW                                        
foo/bar  https://slsa.dev/provenance/v1  .github/workflows/oci-google.yml@refs/heads/main

I think you are using the attestation in the github api when you specify the --owner. Looks like the verify command only supports fetch attestations associated with the provided artifact from the GitHub API or verify the artifact using attestations stored on disk So what's the usage of the attestation stored in registry when we set pushed to registry: true? Looks like the verify command doesn't support it? Could you share the context of the use case of attestation stored in registry when we set pushed to registry: true

[bdehamer]

@capri-xiyue we may expand the gh attestation verify command in the future to support retrieval of attestations from the registry.

A feature we're currently working on is to update the Sigstore policy-controller with support for attestation bundles. This feature would work with attestations stored in the container registry and allow you to write k8s policies preventing any non-attested images from being deployed.

Bottom line: there is no use case at the moment which takes advantage of registry-hosted attestations, but we're working on a few things which will leverage this capability in the future.

[capri-xiyue]

@capri-xiyue we may expand the gh attestation verify command in the future to support retrieval of attestations from the registry.

A feature we're currently working on is to update the Sigstore policy-controller with support for attestation bundles. This feature would work with attestations stored in the container registry and allow you to write k8s policies preventing any non-attested images from being deployed.

Bottom line: there is no use case at the moment which takes advantage of registry-hosted attestations, but we're working on a few things which will leverage this capability in the future.

Thanks for the all these details!!!

[bdehamer]

Just published v1.1.2 of the action which should address the original issue you were seeing when NOT using the setup-buildx-action with the Google Artifact Registry.