Containerd Not Redirecting to the set Mirror Specified in Config.toml

[GuillaumeDorschner]

Description

I am attempting to redirect all Docker Hub requests to my Nexus repository via the containerd configuration. Despite setting up the config.toml file correctly to point to my Nexus repository, containerd is not redirecting Docker Hub requests to the Nexus repository as expected.

Steps to reproduce the issue

1. Set the local mirror in the config.toml file:

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["192.X.X.50:8XXX"]

2. Restart containerd to apply the changes: sudo systemctl restart containerd
3. Attempt to pull an image from Docker Hub using ctr: ctr images pull docker.io/node:latest

Describe the results you received and expected

The pull operation should have been redirected to the local mirror at 192.X.X.50:8XXX as per the configuration. Instead, the operation attempts to pull directly from Docker Hub and fails with the following error:

INFO[0000] trying next host                              error="failed to authorize: failed to fetch anonymous token: unexpected status: 400 Bad Request" host=registry-1.docker.io
ctr: failed to resolve reference "docker.io/node:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 400 Bad Request

What version of containerd are you using?

containerd containerd.io 1.6.21 3dce8eb

Any other relevant information

[root@worker1 ~]# runc --version
runc version 1.1.7
commit: v1.1.7-0-g860f061
spec: 1.0.2-dev
go: go1.19.9
libseccomp: 2.5.2

[root@worker1 ~]# crictl info
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
E0619 15:01:45.103749 2445335 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory""
FATA[0000] getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"

[root@worker1 ~]# uname -a
Linux worker1 4.18.0-477.13.1.el8_8.x86_64 #1 SMP Tue May 30 14:53:41 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

[root@worker1 ~]# cat /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["192.X.X.50:8XXX"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."repo.labo.io:8083"]
endpoint = ["http://repo.labo.io:8083"]

[AkihiroSuda]

ctr images pull docker.io/node:latest

ctr is not a CRI application, so you can't use ctr for testing "io.containerd.grpc.v1.cri" configs.
You may want to use crictl (https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md), or just test it with Kubernetes directly.

[GuillaumeDorschner]

Thank you for your response, @AkihiroSuda.

I've also tried to directly deploy the image on my Kubernetes cluster, but I received the following errors:

│
│ Events:                                                                                                                                                                                                                                  │
│   Type     Reason     Age                    From               Message                                                                                                                                                                  │
│   ----     ------     ----                   ----               -------                                                                                                                                                                  │
│   Normal   Pulling    4m11s (x4 over 5m38s)  kubelet            Pulling image "docker.io/node:latest"                                                                                                                                    │
│   Warning  Failed     4m11s (x4 over 5m38s)  kubelet            Failed to pull image "docker.io/node:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/node:latest": failed to resolve refere │
│ nce "docker.io/library/node:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/node/manifests/latest": dial tcp: lookup registry-1.docker.io on 192.168.137.1:53: server misbehaving                           │
│   Warning  Failed     4m11s (x4 over 5m38s)  kubelet            Error: ErrImagePull                                                                                                                                                      │
│   Warning  Failed     4m (x6 over 5m37s)     kubelet            Error: ImagePullBackOff                                                                                                                                                  │
│   Normal   BackOff    3m48s (x7 over 5m37s)  kubelet            Back-off pulling image "docker.io/node:latest"                                                                                                                           │
│   Normal   Scheduled  2m38s                  default-scheduler  Successfully assigned default/example-node-pod to worker1                                                                                                                               

The pod has this state: ImagePullBackOff.

Despite the configuration in config.toml set to redirect requests from docker.io to my Nexus repository at 192.168.X.X:8082, it seems like Kubernetes is still trying to directly pull the image from Docker Hub (registry-1.docker.io). I expected that Kubernetes would pull the image from the Nexus repository instead. Do you have any suggestions on how to resolve this issue?

[makzzz1986]

@GuillaumeDorschner Check your containerd logs, probably, you will see
Registry:{ConfigPath:/etc/containerd/certs.d Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]}
Seems containerd doesn't read a new configuration, you can try the old one /etc/containerd/config.toml:

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."registry.mypersonal.com"]
    endpoint = ["https://registry.mirror.org", "https://registry.mypersonal.com"]

But it doesn't work as well :)