state dir too large

[sunnoy]

du -sh ./*
0 ./containerd.sock
0 ./containerd.sock.ttrpc
0 ./io.containerd.grpc.v1.cri
0 ./io.containerd.runtime.v1.linux
7.1G ./io.containerd.runtime.v2.task

why io.containerd.runtime.v2.task so large ？

[fuweid]

The rootfs mount point is at state dir so that it looks huge. But the data is actually at root dir.

[sunnoy]

From the documentation, we can see that this directory mainly stores temporary files. Does this directory have parameters that can adjust gc? pod in the nodes in the cluster were evicted because the directory was too large

[leiqi96]

In general, io.containerd.runtime.v2.task is not too large.
maybe ,you can step into the directroy, and Check which file is larger.

[fuweid]

@sunnoy @leiqi96

The state dir is used to store temporary files, for instance, the bundle dir of k8s pod's container in io.containerd.runtime.v2.task/k8s.io.

In containerd design, the container bundle dir includes the mount point of rootfs. For example, the container e731a27's rootfs is /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs.

36K     /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/share/base-files
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/share/info
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/share/misc
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/share/ca-certificates
5.5M    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/share
8.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/lib
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/include
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/bin
8.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/sbin
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/src
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/games
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/libexec/kubernetes/kubelet-plugins/volume/exec
8.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/libexec/kubernetes/kubelet-plugins/volume
12K     /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/libexec/kubernetes/kubelet-plugins
16K     /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/libexec/kubernetes
20K     /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/libexec
108M    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/local/bin
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/local/share/ca-certificates
8.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/local/share
108M    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr/local
114M    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/usr
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/sbin
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/run
4.0K    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs/dev
116M    /var/run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs

The mount point is just view of the container's rootfs filesystem, which doesn't means that it contains the real data in here. When you use du, the du will walk through the mount point. And the container's rootfs includes writable layer and readonly layer (unpacked from image) when you use overlayfs.

overlay on /run/containerd/io.containerd.runtime.v2.task/k8s.io/e731a27984c360bdb18246eeac6e58abead920ec90af9066c64edf2d11d6aa33/rootfs type overlay (rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/14/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/11/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/4407/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/4407/work)

If your container doesn't write any data but the image is huge, the result of du will be large. If your container image is small but the container writes a lot of data into rootfs, the result of du will also be large. Even if your container image size is not too large, the result of du` will be large if there are multiple containers using the same image.

So I want to say du -h /var/run/containerd is not correct. It will go through all the containers' rootfs including the duplicate data. And it just provides a view!

In containerd, the container storage is managed by snapshot and stored in /var/lib/containerd basically. If your pod has been evicted by kubelet with disk pressure, it must be that pod's container writes a lot of data into rootfs or image size is large.

[sunnoy]

so the end result is to expand the divergence where the state directory is，got it ！

[313hemant313]

@sunnoy @fuweid
How to solve this ? as root storage is low, i do have another attached volume.

In /etc/containerd/config.toml
Have already change root = "/mount/volume/containerd" but still below overlay are present

/run/containerd/io.containerd.runtime.v2.task/k8s.io/140bd30302afe355101318cdb2e52a6acfbdef5907ad6af3c31c35011a375437/rootfs

[fuweid]

It's not an issue. please checkout the comment above. thanks

[313hemant313]

I have seen the comment above, but not sure how to do that.

Will be great if you can help me with some command.

[fuweid]

it's not an issue so you don't need to solve it. the state dir is used to hold the view of container's filesystem. the data is stored in root dir.

[313hemant313]

Yes but i do not have sufficient space in root dir, so wanted to replace this root dir to another attached volume directory.

Now i have changed both root and state variables in /etc/containerd/config.toml

root = "/mount/volume/vdb1/containerd"
state = "/mount/volume/vdb1/run/containerd"

Able to see now rootfs mounts are now on specified directory.
But still df -h command is saying 91G is available (Should have been more as this is the new attached volume)

overlay          91G   75G   16G  83% /mount/volume/vdb1/run/containerd/io.containerd.runtime.v2.task/k8s.io/c27cd485906b08317d811914405853796f5053f748802dcc919b62a8a2f222ce/rootfs
overlay          91G   75G   16G  83% /mount/volume/vdb1/run/containerd/io.containerd.runtime.v2.task/k8s.io/44ee8743bc6eb814390ff4b1f9127d7cb790854e83ff93a8eda3a053bd1ffb6d/rootfs
overlay          91G   75G   16G  83% /mount/volume/vdb1/run/containerd/io.containerd.runtime.v2.task/k8s.io/526bb7cf6148626700f6e77045dc0a99a9fd48de84391c8794c90aad05c3ab09/rootfs

[fuweid]

Well. 91GiB is total size. 16 GiB is available.
It's about your setting. Please check it by yourself.