crictl show container status is running even container is exited.

[ningmingxiao]

Description

[root@A800-paas-controller-0-0:/home/pict]$ ./crictl ps -a |grep 32d20
32d20bd3f3b4f       2e00eadf86026       10 hours ago        Running             init-op-psd-pconf                           0                   d6776a2669916

[root@A800-paas-controller-0-0:/home/pict]$ nerdctl  -n k8s.io ps -a |grep 32d20bd3f3b4
32d20bd3f3b4    swr-plat:2524/admin/image/op-psd-pconf:v7.24.10.06.17704944                                  "/home/pconf/init.sh"     6 hours ago     Exited   

nerdctl -n k8s.io inspect 32d20bd3f3b show it is exited at 2024-03-27T07:52:07

[root@A800-paas-controller-0-0:/paasdata/op-log/op-containers-containerd]$ nerdctl  -n k8s.io inspect 32d20bd3f3b4 --mode=native
WARN[0000] failed to inspect NetNS                       error="failed to Statfs \"/proc/913301/ns/net\": no such file or directory" id=32d20bd3f3b4f825508c5638465c7bc5f56b458ae94ad1141429
[
    {
        "ID": "32d20bd3f3b4f825508c5638465c7bc9848f52f490f5f56b458ae94ad1141429",
        "Labels": {
            "io.cri-containerd.kind": "container",
            "io.kubernetes.container.name": "init-op-psd-pconf",
            "io.kubernetes.pod.name": "op-psd-pconf-cim-5bdbfbbbb8-vw8dw",
            "io.kubernetes.pod.namespace": "admin",
            "io.kubernetes.pod.uid": "49b28d61-8984-4ed3-8c4b-0762628c55fe",
            "shim-pid": "913209"
        },
        
         "Process": {
            "Pid": 913301,
            "Status": {
                "Status": "stopped",
                "ExitStatus": 0,
                "ExitTime": "2024-03-26T23:52:07.016309279Z"
            }
        }

containerd log

[root@A800-paas-controller-0-0:/paasdata/op-log/op-containers-containerd]$ cat containerd.dlog |grep 32d20bd3f3b4
Mar 27 07:51:36 A800-paas-controller-0-0 containerd[853996]: time="2024-03-27T07:51:36.419557799+08:00" level=info msg="Container 32d20bd3f3b4f825508c5638465c7bc9848f558ae94ad1141429: CDI devices from CRI Config.CDIDevices: []"
Mar 27 07:51:36 A800-paas-controller-0-0 containerd[853996]: time="2024-03-27T07:51:36.445696277+08:00" level=info msg="CreateContainer within sandbox \"d6776a26699169458bc4601d7da4546b59bba8fc452bf185ee80\" for &ContainerMetadata{Name:init-op-psd-pconf,Attempt:0,} returns container id \"32d20bd3f3b4f825508c5638465c7bc9848f52f490f5f1141429\""
Mar 27 07:51:36 A800-paas-controller-0-0 containerd[853996]: time="2024-03-27T07:51:36.446149033+08:00" level=info msg="StartContainer for \"32d20bd3f3b4f825508c563846f490f5f56b458ae94ad1141429\""
Mar 27 07:51:36 A800-paas-controller-0-0 containerd[853996]: time="2024-03-27T07:51:36.708622716+08:00" level=info msg="StartContainer for \"32d20bd3f3b4f825508c563846f490f5f56b458ae94ad1141429\" returns successfully"

container 32d20 shim is running.

root      913209  0.0  0.0 725532 17168 ?        Sl   07:51   0:05 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id d6776a26699169ee10d3dbe6fe458bc4601d7da4546b5
65535     913256  0.0  0.0    972     4 ?        Ss   07:51   0:00  \_ /pause

containerd reboot at 2024-03-27T07:52:07.651817896

[root@A800-paas-controller-0-0:/paasdata/op-log/op-containers-containerd]$ cat containerd.dlog |grep booted
Mar 26 12:47:10 a800-paas-controller-0-0 containerd[39035]: time="2024-03-26T12:47:10.732029934+08:00" level=info msg="containerd successfully booted in 0.033621s"
Mar 26 12:59:42 A800-paas-controller-0-0 containerd[97431]: time="2024-03-26T12:59:42.659079183+08:00" level=info msg="containerd successfully booted in 0.034071s"
Mar 27 07:47:55 A800-paas-controller-0-0 containerd[853996]: time="2024-03-27T07:47:55.239736792+08:00" level=info msg="containerd successfully booted in 2.689147s"
Mar 27 07:52:07 A800-paas-controller-0-0 containerd[920519]: time="2024-03-27T07:52:07.651817896+08:00" level=info msg="containerd successfully booted in 2.259311s"
Mar 27 18:21:18 A800-paas-controller-0-0 containerd[371467]: time="2024-03-27T18:21:18.917885590+08:00" level=info msg="containerd successfully booted in 2.822029s"

Steps to reproduce the issue

It is difficult to reproduce. It is often hapen when containerd is restarted and run init container in k8s.

Describe the results you received and expected

crictl show same status with ctr or nerdctl

What version of containerd are you using?

1.7.6

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[ja-pavi]

Hi! I'm a UT Austin CS student working on a final project in my virtualization course where my project team has to contribute to an open source project. I was wondering I could get this issue assigned to me, to work on with my group?

[dims]

@ja-pavi easier for you to start with something smaller like this one #9886 given that the person who filed the bug says ...

It is difficult to reproduce. It is often hapen when containerd is restarted and run init container in k8s.

[ja-pavi]

Okay thanks for the heads up, will check that one out!

[ningmingxiao]

Hi! I'm a UT Austin CS student working on a final project in my virtualization course where my project team has to contribute to an open source project. I was wondering I could get this issue assigned to me, to work on with my group?

you can see #9979 and #10002 and try to understand why use shim-v2(containerd-shim-runc-v2) can't reap zombie process but shim-v1 (containerd-shim) will not generate zombie process even use higer kernel version.