Mysterious Image Digest being pulled by the image-updater

[huynhj93]

Describe the bug
Two Thursday ago we encountered an issue with the image updater. We are using the following for our CI/CD pipeline:

1. github actions to build and push to AWS ECR (using buildX action step)
2. argocd-image-updater with Git write back method and digest strategy

The image-updater started commiting SHAs to git with unknown origins. This resulted in an issue with the following error:

docker pull private-ecr@release-development@sha256:fbe6d
Error response from daemon: manifest unknown

To Reproduce
Use a github action with the buildX action plugin to build your image to ECR. Set argocd image updater to use the digest method.

Expected behavior
The image updater actually pulls in an image that exists.

Additional context
Here is the timeline for our entire build: (redacted private ecr host name and shortened shas)
3:47 PST: Merged PR into main, which kicked off a github action.
3:50 PST: Docker build finished within github action with ImageID: sha256:13b65 and digest: sha256:13b65 . they are the same for some reason, but this happened in earlier build where it succeeded. We are also using buildX as suggested here.
3:56 PST: Got these logs from ArgoCD-image-updater pod:

time="2023-01-19T23:56:35Z" level=debug msg="Processing application experts-store-dev"
time="2023-01-19T23:56:35Z" level=debug msg="Considering this image for update" alias=experts-store application=experts-store-dev image_name=experts-store image_tag="sha256:7bf5" registry=private-ecr
time="2023-01-19T23:56:35Z" level=debug msg="Using version constraint 'release-development' when looking for a new tag" alias=experts-store application=experts-store-dev image_name=experts-store image_tag="sha256:7bf5" registry=private-ecr
time="2023-01-19T23:56:37Z" level=debug msg="found 1 from 1 tags eligible for consideration" image="private-ecr/experts-store@sha256:7bf5"
time="2023-01-19T23:56:37Z" level=info msg="Setting new image to private-ecr/experts-store@sha256:fbe6d alias=experts-store application=experts-store-dev image_name=experts-store image_tag="sha256:7bf51" registry=private-ecr
time="2023-01-19T23:56:37Z" level=debug msg="target parameters: image-spec= image-name=image.repository, image-tag=image.tag" application=experts-store-dev image=private-ecr/experts-store
time="2023-01-19T23:56:37Z" level=info msg="Successfully updated image 'private-ecr/experts-store@sha256:7bf51' to 'private-ecr@sha256:fbe6d5', but pending spec update (dry run=false)" alias=experts-store application=experts-store-dev image_name=experts-store image_tag="sha256:7bf5" registry=private-ecr
time="2023-01-19T23:56:37Z" level=debug msg="Using commit message: build: automatic update of experts-store-dev\n\nupdates image experts-store tag 'sha256:7bf51' to 'sha256:fbe6d'\n"

3:56 PST : Double checked above within github and the argocd image updater file that the image sources and it shows that image updater commited the unknown fbe6d digest to VCS, which makes the pod spin up with “manifest unknown”
3:56 pst:Double checked with ECR and we see the actual digest being the one shown in the github action: sha256:13b65

I’m wondering if anyone has encountered this issue? I expected argo to have commited the SHA from the github action: `asha256:13b65 but instead we see this mysterious sha that can’t be found anywhere inside our ECR fbe6d. I noticed that the source code sorts the digest strategy by alphabetically instead of by date, is this intended? https://github.com/argoproj-labs/argocd-image-updater/blob/master/pkg/image/version.go#L95

Monkey Patch fix for others encountering this issue

We have reverted to using the latest strategy in combination with the allowed_tags annotation. The latest strategy actually pulls in an image that exists.

** Other user notes **

1. Brandon Helms in the slack channel has stated that this only happens on his github actions where he uses the buildX step. Other repositories are working just fine.
2. I've attempted to downgrade my buildx action to fix the problem but to no avail. Downgraded to v0.9.1 but issue still occured.
3. Brandon helms in the slack channel has said that the buildx action is behaving weirdly as well where it commits untagged images, which is happening to us as well.

For other users statements, see slack thread here. https://cloud-native.slack.com/archives/C0296T47CHY/p1674584570398149

[subourbonite]

I see this behavior sometimes when querying ECR - images that don't really "exist" get picked up by the updater, causing cascading failures on our end when it tries to apply the update.

[Cr0n1c]

I am experiencing this exact issue and it started att same time as OP.

https://cloud-native.slack.com/archives/C0296T47CHY/p1674588511908689?thread_ts=1674584570.398149&channel=C0296T47CHY&message_ts=1674588511.908689

[ralf-berger]

Same here. Maybe this is caching data or related to multiarch manifests? But strangely enough, there doesn't seem to be any problem when using write-back-method: argocd. 🤔

Has anybody tried provenance: false as the build-push-action release notes suggest (to not create OCI images)?

[andremetzen]

@huynhj93 @Cr0n1c @ralf-berger Disabling provenance fixes the issue!