1.6.13 memory.memsw.limit_in_bytes: no such file or directory

[baskamic]

Description

Errors started to appears after upgrade to 1.6.13 on k8s worker nodes with disabled swap.

RunContainerError: "failed to create containerd task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: open /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d342df_98e0_47d9_ac43_65e42fd694ef.slice/cri-containerd-ecbc9db26facdd91d47f8b5b54e20f97c66e6da2a0dac434fea9ec565e6a4be3.scope/memory.memsw.limit_in_bytes: no such file or directory:

Something similar to: cri-o/cri-o#5527

Steps to reproduce the issue

Unknown

Describe the results you received and expected

Received multiple errors and problems with pods creation.
Expected work as 1.6.12 without errors

What version of containerd are you using?

1.6.13

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[AkihiroSuda]

Might be relevant to:

• [release/1.6] cri: make swapping disabled with memory limit #7815

[cpuguy83]

I met the same issue thinking it was a quirk on the vm I'd spun up (hadn't used the toolchain before).
Since it failed both on my tests and in a normal runc container I just worked around it.

I'm wondering if there was a kernel update that causes memsw to not even exist, or maybe it really just is a coincidence that I hit this at the same time (no containerd involvement) and hadn't seen it before.

[MaxRink]

We are also running into this, with an current Ubuntu 2004

 Kernel Version:                5.4.0-135-generic
 OS Image:                      Ubuntu 20.04.5 LTS
 Operating System:              linux
 Architecture:                  amd64
 Container Runtime Version:     containerd://1.6.13

Edit:

and it only affects containers with requests and or limits set

[hopefulwalker]

I'm also running into this with ubuntu20.04.
master230 Ready control-plane 62m v1.24.9 192.168.56.230 <none> Ubuntu 20.04.5 LTS 5.4.0-131-generic containerd://1.6.13

my coredns met this problem.
Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: open /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b92b885_e844_4154_9674_62a17aac05e7.slice/cri-containerd-coredns.scope/memory.memsw.limit_in_bytes: no such file or directory: unknown

[fabiomirmar]

Same problem with 20.04 + kernel 5.4.0-135 + containerd 1.6.13. Reverting containerd to 1.6.12-1 avoids the issue.

[hkerma]

Hello, also running on that error, Ubuntu 20.04, kernel 5.4.0-110, containerd v1.6.13. Definitely comes from the latest release as I did not have such issues with 1.6.12-1.

[fuweid]

Is there anyone can share the pod spec? thanks

[fuweid]

cc @AkihiroSuda @mikebrow

I think we should check the /proc/swap or CONFIG_SWAP before we set the memorySwap in oci-runtime-spec.
Some kernels weren't built with the flag. The memory.memsw.limit_in_bytes file is missing in this case.

[claudiomerli]

Hi, i'm having same issue on k8s with containerd. Removing resources request/limit specification on pods fix the issue. I don't know if you set them on your container

[claudiomerli]

Downgrading containerd to 1.6.12 it works properly

[fabiomirmar]

In my case, what was crashing was coredns. Here's the pod spec:

https://pastebin.ubuntu.com/p/8b96vKfk6r/

[claudiomerli]

In my case, what was crashing was coredns. Here's the pod spec:

https://pastebin.ubuntu.com/p/8b96vKfk6r/

Yeah, exactly the same for me

[AkihiroSuda]

PR:

• cri: fix memory.memsw.limit_in_bytes: no such file or directory #7836

Workaround: append swapaccount=1 to the kernel cmdline

[brendandburns]

This also affects Debian 10: Linux kube0 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux where the swap controller is also disabled by default.

This is a pretty nasty bug b/c it causes all containers to fail to start (at least on my home system)

[AkihiroSuda]

Keeping open until releasing v1.6.14

[dmcgowan]

https://github.com/containerd/containerd/releases/tag/v1.6.14

[fatmanurozdemir]

same error but reverting containerd version did not work for me.
I get "failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/kubepods/besteffort/poddc92ef05-2620-46d9-aa99-20c3e5e8027/8d01c30fb02300775d081e866901dbf9a34612ded56acda0213a021332033164/cpu.max: no such file or directory: unknown" error when I want to run a pod inside an edge node(NVIDIA AGX Orin).
Kernel Version: 5.15.122-tegra
OS Image: Ubuntu 22.04.3 LTS
Operating System: linux
Architecture: aarch64
Container Runtime Version: containerd://1.6.27

The default containerd version was 1.6.27. I installed 1.6.12 and 1.7.12 versions but it did not resolve the issue.

[AkihiroSuda]

cpu.max: no such file or directory

The error seems quite different from the OP.
Please open a new issue.

[fatmanurozdemir]

I did #9651