Others("Device or resource busy (os error 16)"): unknown

[jsturtevant]

After building and running the demo example, I got the following error:

sudo ctr run --rm --runtime=io.containerd.wasmtime.v1 docker.io/library/wasmtest:latest testwasm
ctr: Others("Device or resource busy (os error 16)"): unknown

investigation

The task is marked as CREATED:

sudo ctr task ls
TASK          PID    STATUS
testwasm13    0      CREATED

but get the following error when trying to delete it:

sudo ctr task rm testwasm13
ERRO[0000] unable to delete testwasm13                   error="task must be stopped before deletion: created: failed precondition"
ctr: task must be stopped before deletion: created: failed precondition

Also get a slightly different error when trying to "stop" it:

sudo ctr task kill -s SIGKILL testwasm13
ctr: cannot kill non-running container, current state: Exited(TaskState { s: PhantomData }): failed precondition

other info

There seems to be two issues:

• the shim is and containerd are out of sync on the state of the shim. this leads to not being able to clean up the task/container
• There is an unhandled exception in

  runwasi/src/instance.rs

  Line 190 in e266bbb

  let res = unsafe { exec::fork(Some(cg.as_ref())) }?;

  This causes the Device or resource busy (os error 16)

versions

containerd version: containerd containerd.io 1.6.7 0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
shim version: built from main (e266bbb)
linux version:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

It does works on my WSL instance:

containerd containerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1    

 lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

[cpuguy83]

Seems like the issue could be due to this https://man.archlinux.org/man/clone3.2.en#EBUSY

Need to look up what to do in this case (domain controller enabled).

[jsturtevant]

We did reproduce this and found it seems specific to cgroupv2. WSL is running cgroupv1. Some further reading: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html

[jsturtevant]

When I force the os error 16 error using the latest containerd release https://github.com/containerd/containerd/releases/tag/v1.6.13 I am not longer seeing the second issue where I was unable to clean up tasks/containers.

It looks like containerd/containerd#7816 may have resolved this issue.