CRI-Plugin crashes when defining other runtimes in imported config-file

[axkng]

Description

Hi,
I have recently worked on integrating WebAssembly runtimes in Kubernetes.
Part of that was to configure additional runtimes in containerd.

Lets assume the following config inside /etc/containerd/config.toml:

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
imports = ["/etc/containerd/config.d/*.toml"]

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin.options]
BinaryName = "/bin/containerd-spin-shim-v2"

The important part are the last four lines:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin.options]
BinaryName = "/bin/containerd-spin-shim-v2"

Initially I wanted to place those in a separate file like /etc/containerd/config.d/webassembly.toml
However, if I do that the CRI plugin simply will not start and I could not find out why.
At first I thought I am doing things wrong in setting this up, but one day I was too lazy to put my config in the separate file and that is when things suddenly worked.
I went back and move those lines to the imported config again and CRI was crashing again.
I tried it with other settings like discard_unpacked_layers = true, which are working from imported configs.
But those regarding the runtimes are not.
Seems like a bug to me.

Steps to reproduce the issue

1. Setup containerd with a config similar to the above and put the runtimes parameters for WebAssembly into a config that gets imported.
2. Restart containerd
3. Run ctr plugin ls to see that the CRI plugin has crashed

Describe the results you received and expected

I was expecting that it does not make any difference in which config file I put my settings and containerd just keeps on working.

What version of containerd are you using?

containerd github.com/containerd/containerd 1.7.11 64b8a81

Any other relevant information

runc version 1.1.11

Show configuration if it is related to CRI plugin.

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
imports = ["/etc/containerd/config.d/*.toml"]

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin.options]
BinaryName = "/bin/containerd-spin-shim-v2"

[lengrongfu]

You can use follow config to set spin runtime, and you can use containerd config default command to generate default config. hope can help you.

    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      default_runtime_name = 'runc'
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false

      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.spin]
          runtime_type = "io.containerd.spin.v2"
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.spin.options]
          BinaryName = "/bin/containerd-spin-shim-v2"

[axkng]

Hi @lengrongfu ,
thank you for your comment.
The problem is however not to get a working config. I have that.
I am just reporting that if you set certain parameters in a config file that gets imported, the CRI plugin crashes, which seems like a bug to me.
I do not have a problem currently, this just seems faulty.

[estesp]

The error is in the containerd stdout/stderr when starting with your included config:

WARN[2024-04-04T18:30:37.688903174Z] failed to load plugin io.containerd.grpc.v1.cri  error="invalid plugin config: no corresponding runtime configured in `containerd.runtimes` for `containerd` `default_runtime_name = \"runc\""

This error is due to the current merging strategy of imported config sections. It definitely had some drawbacks, and specifically, a contributor has been working on an improved strategy in a PR I'll link below. In the current implementation, your second file with definitions for a new runtime entry is effectively replacing, not adding to, the definitions for runtimes in the main file. If you move everything about CRI to the second file (the whole subtree of the config) then the CRI plugin works properly.

More details in this PR: #9982

[axkng]

Hi @estesp ,
thank you for looking into this.
I can confirm that if I build my imported config like this it works:

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
runtime_type = "io.containerd.wasmedge.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge.options]
BinaryName = "/bin/containerd-shim-wasmedge-v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin.options]
BinaryName = "/bin/containerd-shim-spin-v2"

Containerd will restart with no errors and the CRI will be okay.
I must have overlooked the warning in the stdout/stderr of containerd initially. When I looked for it today, I found it of course.
I am curious to see how the PR works out in the end.