Can someone help to understand why image digest (SHA) gets changed when imported using ctr

[rgaduput]

Hi,
We have container images which are saved using docker save and then are loaded into Kubernetes workers using ctr import.
After the import we observed that image digest gets changed and SHA is completely different than what we see in the docker inspect
Aren't image digest supposed to be unique for image ? Any idea why is this happening and is there a way to prevent it ?

We have vulnerability scanner which is sensitive of the SHA, It looks at the SHA of running pod image and then it tries to pull image from docker hub. But it fails since the local image SHA does not match with one in remote even though they are same images.

Docker pull and load

admin@VM-00044:~$ docker pull bitnami/postgresql:13.9.0-debian-11-r21
13.9.0-debian-11-r21: Pulling from bitnami/postgresql
f8c1c832ce65: Already exists 
1ce94ecd4ef6: Pull complete 
Digest: sha256:6f57d7212b34ed17dc148f08269b1cf15ed278e1395a047e82c713868b0337a1
Status: Downloaded newer image for bitnami/postgresql:13.9.0-debian-11-r21
docker.io/bitnami/postgresql:13.9.0-debian-11-r21
admin@ASEC-VM-00044:~$

admin@VM-00044:~$ docker inspect bitnami/postgresql:13.9.0-debian-11-r21 | jq -r '.[].RepoDigests'
[
  "bitnami/postgresql@sha256:6f57d7212b34ed17dc148f08269b1cf15ed278e1395a047e82c713868b0337a1"
]

ctr image import

admin@VM-00044:~$ sudo ctr image import postgres.tar 
unpacking docker.io/bitnami/postgresql:13.9.0-debian-11-r21 (sha256:e39e92c0611d8975131c69768e0af9601dad16a0c88c57e21887ee018e21ea34)...done
admin@ASEC-VM-00044:~$ 

admin@VM-00044:~$ sudo ctr image ls | grep postgres
docker.io/bitnami/postgresql:13.9.0-debian-11-r21          application/vnd.docker.distribution.manifest.v2+json sha256:e39e92c0611d8975131c69768e0af9601dad16a0c88c57e21887ee018e21ea34 256.4 MiB linux/amd64 -      
admin@ASEC-VM-00044:~$

[AkihiroSuda]

The original digest (sha256:6f57d7212b34ed17dc148f08269b1cf15ed278e1395a047e82c713868b0337a1) is lost on docker save, so ctr import can't reproduce it.

$ docker save bitnami/postgresql:13.9.0-debian-11-r21@sha256:6f57d7212b34ed17dc148f08269b1cf15ed278e1395a047e82c713868b0337a1 | tar xv
6f6acf2b77228cd91d2a79f4a770d3fbdf05aa45b0d51d3b4dd3b457c9f481ba/
6f6acf2b77228cd91d2a79f4a770d3fbdf05aa45b0d51d3b4dd3b457c9f481ba/VERSION
6f6acf2b77228cd91d2a79f4a770d3fbdf05aa45b0d51d3b4dd3b457c9f481ba/json
6f6acf2b77228cd91d2a79f4a770d3fbdf05aa45b0d51d3b4dd3b457c9f481ba/layer.tar
f82a58bcec5134210caf952155a1894efa25b87c864096431aa3c27522861dd2/
f82a58bcec5134210caf952155a1894efa25b87c864096431aa3c27522861dd2/VERSION
f82a58bcec5134210caf952155a1894efa25b87c864096431aa3c27522861dd2/json
f82a58bcec5134210caf952155a1894efa25b87c864096431aa3c27522861dd2/layer.tar
fe2ebce8a61abca3e36812e3dc451493844362c0082f031192ad9af7d69008cc.json
manifest.json

$ jq . < manifest.json 
[
  {
    "Config": "fe2ebce8a61abca3e36812e3dc451493844362c0082f031192ad9af7d69008cc.json",
    "RepoTags": null,
    "Layers": [
      "f82a58bcec5134210caf952155a1894efa25b87c864096431aa3c27522861dd2/layer.tar",
      "6f6acf2b77228cd91d2a79f4a770d3fbdf05aa45b0d51d3b4dd3b457c9f481ba/layer.tar"
    ]
  }
]

(However, nerdctl save and nerdctl load can reproduce the digest as nerdctl save preserves the original OCI blobs)

[rgaduput]

Wow, that's a surprise. thanks @AkihiroSuda
But how does docker load brings the digest back to the original one, does it recalculate or something during the loading ?
Any idea if there is possibility of preserving the digest with docker save ?

[xiaoyaozi5566]

@rgaduput Did you find a solution? We also have a usecase that requires the SHA remains the same after ctr import

[rgaduput]

@xiaoyaozi5566 no, we could not find the solution.