Crictl can pull images but ctr gives unauthorized, private registry with basic auth

[DrissiReda]

Running the following works

crictl pull mainframe:5000/image:tag

But not this:

ctr -n=k8s.io pull mainframe:5000/image:tag

which gives "unauthorized"

I am using this config file:

/etc/containerd/config.toml

root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "mainframe:5000/pause:3.2"
    stats_collect_period = 10
    systemd_cgroup = true
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    disable_proc_mount = false
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.configs."mainframe:5000".auth]
        username = "admin"
        password = "password"
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."mainframe:5000"]
          endpoint = ["https://mainframe:5000"]
      [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://mainframe:5000"]
        [plugins.cri.registry.mirrors."k8s.gcr.io"]
          endpoint = ["https://mainframe:5000"]

    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

1. Install containerd
2. Use the config above
3. Put an image in a private registry secured by username/password

Describe the results you received:

Pulling with ctr images pull yields Unauthorized, but pulling with crictl pull works well.

Describe the results you expected:

Both commands to behave the same

Containerd version

containerd github.com/containerd/containerd v1.3.9 ea765aba0d05254012b0b9e595e995c09186427f

** runc version **

runc version 1.0.0-rc10
commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
spec: 1.0.1-dev

crictl info

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": false,
        "reason": "NetworkPluginNotReady",
        "message": "Network plugin returns error: cni plugin not initialized"
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/opt/cni/bin"
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cni-loopback",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "loopback",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "default",
      "defaultRuntime": {
        "runtimeType": "io.containerd.runtime.v1.linux",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false
      },
      "runtimes": {
        "default": {
          "runtimeType": "io.containerd.runtime.v1.linux",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "runtimeRoot": "",
          "options": null,
          "privileged_without_host_devices": false
        },
        "runc": {
          "runtimeType": "io.containerd.runc.v1",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "runtimeRoot": "",
          "options": null,
          "privileged_without_host_devices": false
        }
      },
      "noPivot": false
    },
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://mainframe:5000"
          ]
        },
        "k8s.gcr.io": {
          "endpoint": [
            "https://mainframe:5000"
          ]
        },
        "mainframe:5000": {
          "endpoint": [
            "https://mainframe:5000"
          ]
        }
      },
      "configs": {
        "mainframe:5000": {
          "auth": {
            "username": "admin",
            "password": "password",
            "auth": "",
            "identitytoken": ""
          },
          "tls": null
        }
      },
      "auths": null
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "sandboxImage": "mainframe:5000/pause:3.2",
    "statsCollectPeriod": 10,
    "systemdCgroup": true,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "containerdRootDir": "/var/lib/containerd",
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.13.15"
}

uname -a

Linux master0 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[mikebrow]

Hello. ctr does not use CRI config.. Thus the auth you specified in the CRI section of the config.toml is not being read by the ctr client. To use auth for ctr you need to add -u user:password to the image pull request

[DrissiReda]

Should I specify this configuration elsewhere? I don't want to have to use -u user:password everytime I have to ctr pull

[mikebrow]

at this point in time we don't have another way to specify auth for the ctr tool..

[DrissiReda]

Thank you, is it something that you guys are willing to implement later on? Would you accept a PR for it?

[mikebrow]

fair to have a discussion about it.. have not made a decision yet, as a team, regarding what path to take for host auth config improvements.. Note: https://github.com/containerd/containerd/blob/master/docs/hosts.md

That ^ document covers the recent changes made to support host config for all but host auth..

We might want to implement it similarly.. or maybe store auth info somewhere else

[DrissiReda]

Thanks, if you open a topic here or somewhere I'd be interested in following it.

[AkihiroSuda]

nerdctl (another containerd CLI) supports .docker/config.json for authentication

https://github.com/containerd/nerdctl

[jainasanthosh2023]

We are facing similar issue, where i tried to push local image to private registry like artifactory with ctr push -n k8s.io -u <user_name>, from this moment containerd is using this user for all image pulls in the cluster on this node.

Does "ctr push -n k8s.io -u <user_name>" cache the credentials in containerd memory?

[jainasanthosh2023]

Solved, containerd 1.6.7 is using .docker/config.json credentials to pull image in k8s ,if we use credentials to do docker login if that password expires then k8s pod image pull face 401 , 403 errors .
In our case we use artifactory as registry.

Only thing puzzling me why containerd still using docker credentials from docker config.json.

Lesson learned we should not use docker login if still used , we should logout from k8s nodes.