how can I check if containerd is running with systemd cgroup for runc? #5413

rdxmb · 2021-04-23T09:34:59Z

rdxmb
Apr 23, 2021

Hello,

I am trying to set

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            systemdCgroup = true

within a separate file under /etc/containerd/conf.d and include this with imports = ["/etc/containerd/conf.d/*.toml"] within /etc/containerd/config.toml.

Is there a way to check if this configuration does work? I have not found any way to check the "real" configuration. Are there any logs or this there any output which does parse that?

Answered by AkihiroSuda

Nov 20, 2022

Run sudo /usr/sbin/execsnoop-bpfcc -n runc, create some containers, and observe whether the execsnoop result contains --systemd-cgroup

$ sudo /usr/sbin/execsnoop-bpfcc -n runc
PCOMM            PID    PPID   RET ARGS
runc             7984   7974     0 /usr/local/bin/runc --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4/log.json --log-format json --systemd-cgroup create --bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4 --pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/b6725c857dd7492a37f73038d…

View full answer

fuweid · 2021-04-23T10:28:20Z

fuweid
Apr 23, 2021
Maintainer

HI， you can try to use crictl info command which is designed for CRI.

0 replies

mikebrow · 2021-04-23T16:56:51Z

mikebrow
Apr 23, 2021
Maintainer

Make sure you test it out on the latest 1.5 release candidate. We resolved a couple issues where runc.options were not being shown, such as via containerd config (default/dump) or in crictl info

3 replies

rdxmb Apr 23, 2021
Author

So if I am not going to compile containerd, I have to wait for the debian package/ppa, right?

rdxmb Apr 27, 2021
Author

ok, I have downloaded the latest nightly build and got the binary of containerd. This seems to parse and output the configuration with containerd config dump.

However,

cat /etc/containerd/conf.d/systemdcgroup.toml 
version = 2
root = "/var/lib/containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
oom_score = 0

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true

combined with
imports = ["/etc/containerd/conf.d/*.toml"] within /etc/containerd/config.toml results in

panic: reflect: reflect.Value.Set using unaddressable value

goroutine 1 [running]:
reflect.flag.mustBeAssignableSlow(0x99)
        /opt/hostedtoolcache/go/1.16.3/x64/src/reflect/value.go:260 +0x13f
reflect.flag.mustBeAssignable(...)
        /opt/hostedtoolcache/go/1.16.3/x64/src/reflect/value.go:247
reflect.Value.Set(0x55d2aac2e4c0, 0xc0005ac6f0, 0x99, 0x55d2aac2e4c0, 0xc0005ac6c0, 0x99)
        /opt/hostedtoolcache/go/1.16.3/x64/src/reflect/value.go:1558 +0x3d
github.com/imdario/mergo.deepMerge(0x55d2aac2e4c0, 0xc0005ac6f0, 0x99, 0x55d2aac2e4c0, 0xc0005ac6c0, 0x99, 0xc00051e730, 0x2, 0xc000144f60, 0x0, ...)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/imdario/mergo/merge.go:99 +0x317f
github.com/imdario/mergo.deepMerge(0x55d2aab1aa60, 0xc00039f9a0, 0x195, 0x55d2aab1aa60, 0xc000118960, 0x195, 0xc00051e730, 0x1, 0xc000144f60, 0x0, ...)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/imdario/mergo/merge.go:147 +0xaa6
github.com/imdario/mergo.deepMerge(0x55d2aace26e0, 0xc00039f860, 0x199, 0x55d2aace26e0, 0xc000118820, 0x199, 0xc00051e730, 0x0, 0xc000144f60, 0x199, ...)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/imdario/mergo/merge.go:93 +0x3079
github.com/imdario/mergo.merge(0x55d2aab58e20, 0xc00039f860, 0x55d2aab58e20, 0xc000118820, 0xc00051e878, 0x2, 0x2, 0x55d2aace26e0, 0xc000118820)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/imdario/mergo/merge.go:361 +0x345
github.com/imdario/mergo.Merge(...)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/imdario/mergo/merge.go:291
github.com/containerd/containerd/services/server/config.mergeConfig(0xc00039f860, 0xc000118820, 0xc000118820, 0x0)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/services/server/config/config.go:325 +0xa5
github.com/containerd/containerd/services/server/config.LoadConfig(0xc000596210, 0x29, 0xc00039f860, 0xc00039dc20, 0x1b)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/services/server/config/config.go:245 +0x207
github.com/containerd/containerd/cmd/containerd/command.glob..func2(0xc0003ba840, 0xc00044f100, 0xc00044f170)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/cmd/containerd/command/config.go:111 +0x77
github.com/urfave/cli.HandleAction(0x55d2aaac59e0, 0x55d2aad14330, 0xc0003ba840, 0xc0003ba840, 0x0)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0x107
github.com/urfave/cli.Command.Run(0x55d2aa7cf0ce, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55d2aa8484cd, 0x48, 0x0, ...)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:174 +0x579
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000387a40, 0xc0003ba580, 0x0, 0x0)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:404 +0x8f4
github.com/urfave/cli.Command.startApp(0x55d2aa7d145e, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55d2aa80992e, 0x24, 0x0, ...)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:329 +0x7ff
github.com/urfave/cli.Command.Run(0x55d2aa7d145e, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55d2aa80992e, 0x24, 0x0, ...)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:102 +0xa14
github.com/urfave/cli.(*App).Run(0xc000387880, 0xc00003c090, 0x3, 0x3, 0x0, 0x0)
        /home/runner/work/containerd/containerd/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:276 +0x7e8
main.main()
        github.com/containerd/containerd/cmd/containerd/main.go:33 +0x51

Is this a config problem or a bug?

EDIT:


/opt/containerd/containerd --version
containerd github.com/containerd/containerd b0fb8a5 b0fb8a5a04eff810861d258053bdac951ccf7b78

rdxmb Nov 9, 2021
Author

Can anybody help me with this? I'd like to use a config in conf.d in geerlingguy/ansible-role-containerd#3

csaltos · 2022-11-16T19:35:49Z

csaltos
Nov 16, 2022

After reading several blogs and trying a lot of things I got to find a workaround on a Chinese blog post:

You may use this configuration for etc/containerd/contig.toml:

version = 2

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    systemd_cgroup = true
    [plugins."io.containerd.grpc.v1.cri".containerd]
      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/usr/lib/cni"
      conf_dir = "/etc/cni/net.d"
  [plugins."io.containerd.internal.v1.opt"]
    path = "/var/lib/containerd/opt"

The important values are systemd_cgroup as true and the default_runtime with runtime_type with the old version using io.containerd.runtime.v1.linux ... the other values may be different for your environment, please use it only as a reference.

Once the configuration file is updated run systemctl restart containerd and check the new systemdCgroup as true running the command crictl info

I hope this is useful to someone else.

Tried on a Debian GNU/Linux 11 Bullseye AMD64 with containerd version 1.4.13~ds1 and cri-tools 1.0.0-beta.1-00

1 reply

AkihiroSuda Nov 20, 2022
Maintainer

io.containerd.runtime.v1.linux is deprecated, has no support for cgroup v2, and will be removed in containerd 2.0.

AkihiroSuda · 2022-11-20T11:33:59Z

AkihiroSuda
Nov 20, 2022
Maintainer

Run sudo /usr/sbin/execsnoop-bpfcc -n runc, create some containers, and observe whether the execsnoop result contains --systemd-cgroup

$ sudo /usr/sbin/execsnoop-bpfcc -n runc
PCOMM            PID    PPID   RET ARGS
runc             7984   7974     0 /usr/local/bin/runc --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4/log.json --log-format json --systemd-cgroup create --bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4 --pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4/init.pid b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4
runc             7999   7974     0 /usr/local/bin/runc --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4/log.json --log-format json --systemd-cgroup start b6725c857dd7492a37f73038db80b7443ea1ed9c24b2d98a4b4e15ee8dad24c4
runc             8028   7974     0 /usr/local/bin/runc --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/7168c8744f0d2212fac0123f874e348972afae8d2f96a225daa4bae5f593725a/log.json --log-format json --systemd-cgroup create --bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/7168c8744f0d2212fac0123f874e348972afae8d2f96a225daa4bae5f593725a --pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/7168c8744f0d2212fac0123f874e348972afae8d2f96a225daa4bae5f593725a/init.pid --console-socket /tmp/pty2458939095/pty.sock 7168c8744f0d2212fac0123f874e348972afae8d2f96a225daa4bae5f593725a
runc             8047   7974     0 /usr/local/bin/runc --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/7168c8744f0d2212fac0123f874e348972afae8d2f96a225daa4bae5f593725a/log.json --log-format json --systemd-cgroup start 7168c8744f0d2212fac0123f874e348972afae8d2f96a225daa4bae5f593725a

The execsnoo-bpfcc command can be installed by running sudo apt-get install bpfcc-tools on Ubuntu.

1 reply

cwrau Nov 16, 2023

You're a lifesaver, I couldn't for the life of me figure out why cgroupv2 wasn't working on our kubernetes setup

This made me realize that, even though it's set in the config (see bug #5837), the setting SystemdCgroup = true "never got set" / "got overridden" / ...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how can I check if containerd is running with systemd cgroup for runc? #5413

{{title}}

Replies: 4 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

how can I check if containerd is running with systemd cgroup for runc? #5413

rdxmb Apr 23, 2021

Replies: 4 comments · 5 replies

fuweid Apr 23, 2021 Maintainer

mikebrow Apr 23, 2021 Maintainer

rdxmb Apr 23, 2021 Author

rdxmb Apr 27, 2021 Author

rdxmb Nov 9, 2021 Author

csaltos Nov 16, 2022

AkihiroSuda Nov 20, 2022 Maintainer

AkihiroSuda Nov 20, 2022 Maintainer

cwrau Nov 16, 2023

rdxmb
Apr 23, 2021

Replies: 4 comments 5 replies

fuweid
Apr 23, 2021
Maintainer

mikebrow
Apr 23, 2021
Maintainer

rdxmb Apr 23, 2021
Author

rdxmb Apr 27, 2021
Author

rdxmb Nov 9, 2021
Author

csaltos
Nov 16, 2022

AkihiroSuda Nov 20, 2022
Maintainer

AkihiroSuda
Nov 20, 2022
Maintainer