The config_path configuration item under plugins."io.containerd.grpc.v1.cri".registry doesn't seem to do anything

[jroose]

Description

This issue is causing my kubernetes cluster to fail to pull images after upgrading my containerd.

The config_path configuration item under plugins."io.containerd.grpc.v1.cri".registry doesn't seem to work as expected. Based on the documentation, I had expected it to make the use of --hosts-dir on ctr images pull redundant. Instead, ctr still fails to pull images from my private registry unless I explicitly add --hosts-dir=/etc/containerd/certs.d to the command.

I've tried on version 1.5.9 (the latest ubuntu 22.04 release) and the 1.6.14 binary release from the containerd.io website. Both exhibit this behavior.

In case it's relevant, I'm running on a Raspberry Pi 4 (arm64 architecture, not x86).

Steps to reproduce the issue

1. Install latest version of containered on an arm64 device (e.g. raspberry pi)
2. Configure private registries in /etc/containered/certs.d
3. Edit the default config.toml to include config_path="/etc/containerd/certs.d"

Describe the results you received and expected

I expected setting config_path to eliminate the need for using --hosts-dir on ctr images pull commands. I also expected my kubernetes cluster to continue to be able to pull images from my private registry, but after upgrading containerd to v1.5.9, it can't.

root@redacted-1:~# systemctl restart containerd
root@redacted-1:~# ctr images pull docker.redacted.com/redacted/redacted-image-name:b-132
docker.redacted.com/redacted/redacted-image-name:b-132: resolving      |--------------------------------------| 
elapsed: 0.2 s                                                   total:   0.0 B (0.0 B/s)                                         
ctr: failed to resolve reference "docker.redacted.com/redacted/redacted-image-name:b-132": pulling from host docker.redacted.com failed with status code [manifests b-132]: 400 Bad Request
root@redacted-1:~# ctr images pull docker.redacted.com/redacted/redacted-image-name:b-132 --hosts-dir /etc/containerd/certs.d/
docker.redacted.com/redacted/redacted-image-name:b-132:                  resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:58b52943729e983192c23a75e184003222f75acf1b47f3d2afc838ee5fb003e1: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0958e781caddc3442261759ac9220546437bcb5487a922872bc1093d7666b2f3:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:702f48377ee9eb5a0964db54e7607f63b3f3b7c0bf788ee37c58521e5b8baf82:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:be0de17fe24f767ec21bec97d0e8ea8f0d907fe05238a0bf9cce0995f529f7ea:    downloading    |++++++++++++++++++++++++++++++--------| 21.0 MiB/25.9 MiB 
layer-sha256:aad458e511f2ee0ef2f4d18963838da45ff783e4868f676bf008bff7be6149c7:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:450a3b764d68c09d469474e22dca5a8dc89697074c3d92adcf99b891c6171764:    downloading    |++++++++++++++++++--------------------| 22.0 MiB/46.2 MiB 
layer-sha256:a7cfb45e587b59a1acab8166c56f651d1e58815f2bd7ef0f3956c56d4dd0b105:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:bf53c6b3d60283b44ec6672c3b59aa10436d8cbf4aeeae934b1cfffa3f6c8513:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:bd7863732ed30dae0ed5ab8f3921a9ea1bbf9f15089e5cdb7cb9b6d567f62a5f:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:f6c09d650fede3f3199aec4b1ac3d770ec2b92170317bcdfed58bd63af100b7b:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:9cf7d895e3720994909798d6f24928cf166178b5e7e2be3e4bb7dac0cf9183cc:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:234b5824a11e52341868fb4ed2e2ac889aa1b91764ce7bed61bfa81873f2198b:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0c30d74a496be34002de3151b45d213abd60050b78691a3674b69389827fc37a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:36e18d4cd0f80c5092b99bb8c3bbb22a16877e1ce8f25923777d9b353dae509f:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.5 s                                                                    total:  43.0 M (17.1 MiB/s)                                      
^C
root@redacted-1:~# containerd config default > config.toml
root@redacted-1:~# diff config.toml /etc/containerd/config.toml
145c145
<       config_path = ""
---
>       config_path = "/etc/containerd/certs.d"
root@redacted-1:~# containerd --version
containerd github.com/containerd/containerd v1.6.14 9ba4b250366a5ddde94bb7c9d1def331423aa323

What version of containerd are you using?

Originally v1.5.9, then v1.6.14

Any other relevant information

root@redacted-1:~# runc --version
runc version 1.1.0-0ubuntu1
spec: 1.0.2-dev
go: go1.17.3
libseccomp: 2.5.3
root@redacted-1:~# crictl info 
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/opt/cni/bin"
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cni-loopback",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "loopback",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "Name": "cbr0",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "flannel",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"delegate\":{\"hairpinMode\":true,\"isDefaultGateway\":true},\"type\":\"flannel\"}"
            },
            {
              "Network": {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"type\":\"portmap\"}"
            }
          ],
          "Source": "{\n  \"name\": \"cbr0\",\n  \"cniVersion\": \"0.3.1\",\n  \"plugins\": [\n    {\n      \"type\": \"flannel\",\n      \"delegate\": {\n        \"hairpinMode\": true,\n        \"isDefaultGateway\": true\n      }\n    },\n    {\n      \"type\": \"portmap\",\n      \"capabilities\": {\n        \"portMappings\": true\n      }\n    }\n  ]\n}\n"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "runtimes": {
        "runc": {
          "runtimeType": "io.containerd.runc.v2",
          "runtimePath": "",
          "runtimeEngine": "",
          "PodAnnotations": [],
          "ContainerAnnotations": [],
          "runtimeRoot": "",
          "options": {
            "BinaryName": "",
            "CriuImagePath": "",
            "CriuPath": "",
            "CriuWorkPath": "",
            "IoGid": 0,
            "IoUid": 0,
            "NoNewKeyring": false,
            "NoPivotRoot": false,
            "Root": "",
            "ShimCgroup": "",
            "SystemdCgroup": false
          },
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": "",
          "cniConfDir": "",
          "cniMaxConfNum": 0
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false,
      "ignoreRdtNotEnabledErrors": false
    },
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": "",
      "ipPref": ""
    },
    "registry": {
      "configPath": "/etc/containerd/certs.d",
      "mirrors": {},
      "configs": {},
      "auths": {},
      "headers": {}
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "registry.k8s.io/pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "device_ownership_from_security_context": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "enableUnprivilegedPorts": false,
    "enableUnprivilegedICMP": false,
    "containerdRootDir": "/var/lib/containerd",
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.18.9",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK"
}
root@redacted-1:~# uname -a
Linux redacted-1 5.15.0-1021-raspi #23-Ubuntu SMP PREEMPT Fri Nov 25 15:27:43 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Show configuration if it is related to CRI plugin.

root@redacted-1:~# cat /etc/containerd/config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.k8s.io/pause:3.6"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.internal.v1.tracing"]
    sampling_ratio = 1.0
    service_name = "containerd"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/arm64/v8"]
    sched_core = false

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.service.v1.tasks-service"]
    rdt_config_file = ""

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    discard_blocks = false
    fs_options = ""
    fs_type = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""
    upperdir_label = false

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

  [plugins."io.containerd.tracing.processor.v1.otlp"]
    endpoint = ""
    insecure = false
    protocol = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.bolt.open" = "0s"
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

root@redacted-1:~# tree /etc/containerd/
/etc/containerd/
├── certs.d
│   └── docker.redacted.com
│       ├── ca.crt
│       ├── client.cert -> /etc/ssl/certs/redacted-1.redacted.com.crt
│       ├── client.key -> /etc/ssl/private/redacted-1.redacted.com.pem
│       └── host.toml
└── config.toml

2 directories, 5 files

root@redacted-1:~# cat /etc/containerd/certs.d/docker.redacted.com/host.toml 
server = "https://docker.redacted.com"

[host."https://docker.redacted.com"]
  capabilities = ["pull", "resolve", "push"]
  ca = "ca.crt"
  client = [["client.cert", "client.key"]]

[AkihiroSuda]

ctr does not use the CRI API, so the "io.containerd.grpc.v1.cri" config does not apply to ctr.

[jroose]

Thanks for enlightening me @AkihiroSuda. Since I'm using docker style for my certs and keys, do you know of a way to set --hosts-dir for ctr? Is there a separate section that is applicable in /etc/containerd/config.toml?

[AkihiroSuda]

Is there a separate section that is applicable in /etc/containerd/config.toml?

No, so you may want to create a custom wrapper script or a shell alias.

[AkihiroSuda]

Not ctr, but nerdctl supports hosts_dir via /etc/nerdctl/nerdctl.toml
https://github.com/containerd/nerdctl/blob/v1.1.0/docs/config.md