container is sometimes not created : failed to create shim task: failed to mount rootfs component

[soyeone]

containerd : 1.6.2
nerdctl : 0.18.0

Hi,
I'm trying to change docker to containerd when using k8s in my server.
And I ran into a problem that containers are sometimes not created.

error message:

level=fatal msg="failed to create shim task: failed to mount rootfs component &{aufs none [dio xino=/dev/shm/aufs.xino dirperm1 br:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/27956/fs=rw:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/521/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/520/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/519/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/518/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/517/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/25/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/24/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/23/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/22/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/21/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/20/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/19/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/18/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/17/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/16/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/15/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/14/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/13/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/12/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/11/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/10/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/9/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/8/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/7/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/6/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/5/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/4/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/3/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/2/fs=ro+wh:/var/lib/containerd/io.containerd.snapshotter.v1.aufs/snapshots/1/fs=ro+wh]}: invalid argument: unknown"

This error doesn't occur every time and I don't know the reason why the issue occurred.
And If I try to create a container again after the failed message, it's created well.

When I used docker there was no error like that.
Also when one node has only one container, there is no error.

Any solution about this?

using command:
nerdctl --cgroup-manager systemd --snapshotter aufs run --privileged --env {env} -v {dir} -d {container}

[andy108369]

Try newer kernel.
I've had same looking error but with older kernel 5.15 and with ZFS-on-root.

---

If using ZFS-on-root, make sure to use hwe linux kernel, e.g. above 6.1

Older Linux kernels such as 5.15 gonna yield these errors:

kubelet[95559]: E0219 18:07:34.883174   95559 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component: invalid argument: unknown"

And the only way to see what's going on is dmesg:

overlay is containerd's snapshotter

# dmesg -T
[Mon Feb 19 16:55:43 2024] overlayfs: upper fs does not support RENAME_WHITEOUT.
[Mon Feb 19 16:55:43 2024] overlayfs: upper fs missing required features.

Hope that helps. (Am running K8s v1.28.6 / containerd v1.7.11 on Ubuntu 22.04 LTS, linux 6.5.0-18-generic)

[kkhstut]

You are my hero :)