1.6.x inside a Docker container fails to run containers (cannot enter cgroupv2 "/sys/fs/cgroup/default" with domain controllers -- it is in an invalid state)

[tianon]

Description

On 1.5.x (with either of runc 1.0.3 or 1.1.0) I can successfully run a Docker container with --privileged and /var/lib/containerd on a volume, run containerd inside, and run containers on that containerd instance without issue.

On 1.6.x (again, with either recent runc version) I get the following error when I try to run a container:

ctr: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/default" with domain controllers -- it is in an invalid state: unknown

Steps to reproduce the issue

1. build an image with the version of containerd + runc desired
2. docker run --name test --privileged --volume /var/lib/containerd your-docker-image containerd
   (feel free to use the tianon/containerd:1.6.1 or tianon/containerd:1.5.10 images if you do not want to build your own)
3. docker exec -it test bash -c 'ctr image pull docker.io/library/bash:latest && ctr run -t --rm docker.io/library/bash:latest test'

Describe the results you received and expected

Expected:

$ docker exec -it test bash -c 'ctr image pull docker.io/library/bash:latest && ctr run -t --rm docker.io/library/bash:latest test
docker.io/library/bash:latest:                                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:20f58dc05f356808d12635065b568d37966bb68f472c53edaa98ba8e16430d26: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:832a0baf56be374fb9d9151f3ef6a82c5754f69117f2c58fd235c11c801725cb:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:53e520fda1825ccb2b42bb9b8d5d3d9842f11063ff519347da83ca928b18bedd:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:c3141377f232ef4d065034b3332237dc2ca236ac742f83b93aa2d5f703b5e3bd:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.2 s                                                                    total:  2.7 Mi (1.2 MiB/s)                                       
unpacking linux/amd64 sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9...
done: 215.141188ms	
bash-5.1# 

Recevied:

$ docker exec -it test bash -c 'ctr image pull docker.io/library/bash:latest && ctr run -t --rm docker.io/library/bash:latest test
docker.io/library/bash:latest:                                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:20f58dc05f356808d12635065b568d37966bb68f472c53edaa98ba8e16430d26: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:832a0baf56be374fb9d9151f3ef6a82c5754f69117f2c58fd235c11c801725cb:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:c3141377f232ef4d065034b3332237dc2ca236ac742f83b93aa2d5f703b5e3bd:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:53e520fda1825ccb2b42bb9b8d5d3d9842f11063ff519347da83ca928b18bedd:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.2 s                                                                    total:  4.5 Mi (2.0 MiB/s)                                       
unpacking linux/amd64 sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9...
done: 176.456151ms	
ctr: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/default" with domain controllers -- it is in an invalid state: unknown

What version of containerd are you using?

containerd github.com/containerd/containerd v1.6.1 10f428d

Any other relevant information

I originally thought this was somehow related to opencontainers/runc#3387, but I'm a lot less sure of that now.

Show configuration if it is related to CRI plugin.

No response

[tianon]

Sorry, forgot to mention that this is on cgroupsv2 (although I think that's probably clear from the error message).

Edit: uname output:

5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux

Edit 2x: Docker host details: (although I'm not convinced these are actually relevant, especially since 1.5.x works fine)

• Docker 20.10.12
• containerd 1.4.12
• runc 1.0.2

[AkihiroSuda]

Please provide Dockerfile, and also please make sure you have this https://github.com/moby/moby/blob/38805f20f9bcc5e87869d6c79d432b166e1c88b4/hack/dind#L28-L38

# cgroup v2: enable nesting
if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
	# move the processes from the root group to the /init group,
	# otherwise writing subtree_control fails with EBUSY.
	# An error during moving non-existent process (i.e., "cat") is ignored.
	mkdir -p /sys/fs/cgroup/init
	xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
	# enable controllers
	sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
		> /sys/fs/cgroup/cgroup.subtree_control
fi

[tianon]

Ahhhhhhh, now I feel silly -- that's exactly what I was missing. Thank you, @AkihiroSuda!

Hopefully this can stand as a helpful reminder to the next poor sucker who tries to do this and fails with a similar error message. 😄 😇 🙈

[employed-alabaster-daringly]

I have a similar issue, however I am trying to start rootless dockerd and a containter within a manually created network and cgroup namespace. I enter using sudo nsenter --net=/path/ --cgroup=/path/ /bin/bash I then su to my low privilged non-root account. dockerd seems to start fine.

I then get another shell the same way as above, to start a container. I get a long error like this: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit "docker-dbbff2d9e395286c761a063c8205c8944d4031bb5ce60b59b97f2dc9ea290aed.scope" (properties [{Name:Description Value:"libcontainer container dbbff2d9e395286c761a063c8205c8944d4031bb5ce60b59b97f2dc9ea290aed"} {Name:Slice Value:"user.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [2145]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Interactive authentication required.: unknown.

I tried the above script, running it inside my namespace as root however the init/cgroup.procs apparently does not exist when you try to write to it. I can see it with ls. But if I try to echo anything into it, or write to it with nano I get write error: No such file or directory