Gadget can't find any pods in cluster

[clanktron]

Apologies if this is just me missing some kernel/cluster requisite for the tool to work, but going through the docs I didn't think I missed anything.

Description

I followed the quickstart and installed gadget with krew, and then ran a successful gadget deploy on my cluster. I then ran kubectl gadget snapshot process -A and got no output. Looking at the gadget logs I can see the following error for every pod in the cluster.

time="2024-05-20T23:21:17Z" level=warning msg="Skip pod tekton-pipelines/tekton-triggers-webhook-674ff6656d-r5vqq: cannot find container (ID: containerd://2fd9934abafbd4b63a1eed9dd91b5f4768cde6791775aec0a9d4a89be983f3ee): loading container with id \"2fd9934abafbd4b63a1eed9dd91b5f4768cde6791775aec0a9d4a89be983f3ee\": container \"2fd9934abafbd4b63a1eed9dd91b5f4768cde6791775aec0a9d4a89be983f3ee\" in namespace \"k8s.io\": not found"

Impact

Gadget can't retrieve any cluster information at all, no commands that I've tested yield any data.

example:

$ k gadget snapshot process -A
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     COMM      PID       UID       GID

Environment and steps to reproduce

Cluster is a basic bare-metal k3s cluster running on a single arch linux node. Gadget pod that got deployed is ghcr.io/inspektor-gadget/inspektor-gadget:v0.28.1

$ uname -a
Linux minibox 6.8.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 28 Apr 2024 15:59:47 +0000 x86_64 GNU/Linux

$ cat /os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

$ k get nodes -o wide
NAME      STATUS   ROLES                  AGE    VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE     KERNEL-VERSION   CONTAINER-RUNTIME
minibox   Ready    control-plane,master   180d   v1.27.7+k3s2   192.168.1.xxx   <none>        Arch Linux   6.8.8-arch1-1    containerd://1.7.7-k3s1.27

[mauriciovasquezbernal]

@clanktron could you please share the full logs from Inspektor Gadget? I suspect that the runcfanotify

inspektor-gadget/pkg/gadgettracermanager/gadgettracermanager.go

Line 287 in bf08362

opts = append(opts, containercollection.WithContainerFanotifyEbpf())

is failing and then the pod informer is being used

inspektor-gadget/pkg/gadgettracermanager/gadgettracermanager.go

Line 300 in bf08362

opts = append(opts, containercollection.WithPodInformer(g.nodeName))

and then it's not working fine.

[clanktron]

time="2024-05-30T23:41:26Z" level=warning msg="Skip pod gadget/gadget-rp8hd: cannot find container (ID: containerd://baa3f8652dbc3b1d723889b52158f5156324791bf8a62ad5d119a17b22907f75): loading container with id \"baa3f8652dbc3b1d723889b52158f5156324791bf8a62ad5d119a17b22907f75\": container \"baa3f8652dbc3b1d723889b52158f5156324791bf8a62ad5d119a17b22907f75\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=info msg="gadget tracermgr set in kubemanager"
time="2024-05-30T23:41:26Z" level=info msg="Serving on gRPC socket /run/gadgettracermanager.socket"
time="2024-05-30T23:41:26Z" level=info msg="Starting Pod controller"
time="2024-05-30T23:41:26Z" level=info msg="Starting trace controller manager"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod blog/blog-58c6c5fcd7-chz6w: cannot find container (ID: containerd://43e03a002825066205c582544fd3df32bd584f291d0ba51f48b60d4df987840b): loading container with id \"43e03a002825066205c582544fd3df32bd584f291d0ba51f48b60d4df987840b\": container \"43e03a002825066205c582544fd3df32bd584f291d0ba51f48b60d4df987840b\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod blog/blog-58c6c5fcd7-rgqhz: cannot find container (ID: containerd://1a825fcea3ceafc3bc0cb20e4862f0588e771ff91197b81786c94ba9c66ee6e9): loading container with id \"1a825fcea3ceafc3bc0cb20e4862f0588e771ff91197b81786c94ba9c66ee6e9\": container \"1a825fcea3ceafc3bc0cb20e4862f0588e771ff91197b81786c94ba9c66ee6e9\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod blog/blog-58c6c5fcd7-zrjqv: cannot find container (ID: containerd://4b5fd9b84629577f7f7aeabf272784a6ed1c552eb1d9bee18409c9b55553c454): loading container with id \"4b5fd9b84629577f7f7aeabf272784a6ed1c552eb1d9bee18409c9b55553c454\": container \"4b5fd9b84629577f7f7aeabf272784a6ed1c552eb1d9bee18409c9b55553c454\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod cert-manager/cert-manager-cert-manager-5fd596bd7f-w8t85: cannot find container (ID: containerd://5c3d1c8a7778c6d0e3e1a16f15526cf646bb2e834a964a26c9c2d54a5048c7df): loading container with id \"5c3d1c8a7778c6d0e3e1a16f15526cf646bb2e834a964a26c9c2d54a5048c7df\": container \"5c3d1c8a7778c6d0e3e1a16f15526cf646bb2e834a964a26c9c2d54a5048c7df\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod cert-manager/cert-manager-cert-manager-cainjector-6db977b688-hf2v4: cannot find container (ID: containerd://75c6e8c46206217c74fcc16a216e55a056d058a141d0d58b826faa0d7e70078e): loading container with id \"75c6e8c46206217c74fcc16a216e55a056d058a141d0d58b826faa0d7e70078e\": container \"75c6e8c46206217c74fcc16a216e55a056d058a141d0d58b826faa0d7e70078e\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod cert-manager/cert-manager-cert-manager-webhook-754ff76478-trjk2: cannot find container (ID: containerd://c6216ffa76be9bf2c861d89ea89ea8ed20f6e671c4d2d85a46c457364c8856c8): loading container with id \"c6216ffa76be9bf2c861d89ea89ea8ed20f6e671c4d2d85a46c457364c8856c8\": container \"c6216ffa76be9bf2c861d89ea89ea8ed20f6e671c4d2d85a46c457364c8856c8\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod cockroach-operator-system/cockroach-operator-manager-8bfb56bfb-l47j2: cannot find container (ID: containerd://3029d3d2cdf39ed50e6385002b318cb529940cc8b8a90c258910c613e80124ef): loading container with id \"3029d3d2cdf39ed50e6385002b318cb529940cc8b8a90c258910c613e80124ef\": container \"3029d3d2cdf39ed50e6385002b318cb529940cc8b8a90c258910c613e80124ef\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod coredns/coredns-d665f557d-zdlrp: cannot find container (ID: containerd://7e3257990d5ea5b9ff894593301a2dd43b01057abeb4a9043d92ef12f81f7776): loading container with id \"7e3257990d5ea5b9ff894593301a2dd43b01057abeb4a9043d92ef12f81f7776\": container \"7e3257990d5ea5b9ff894593301a2dd43b01057abeb4a9043d92ef12f81f7776\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod filebrowser/filebrowser-748769b5cc-bjvzk: cannot find container (ID: containerd://33023aab3d921d3ec15efd72f4bead045d6d38a7be395355d28882fb7e772eee): loading container with id \"33023aab3d921d3ec15efd72f4bead045d6d38a7be395355d28882fb7e772eee\": container \"33023aab3d921d3ec15efd72f4bead045d6d38a7be395355d28882fb7e772eee\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod flux-system/helm-controller-b957fcf89-8775s: cannot find container (ID: containerd://33f92b150cb8bbdba587f2e70cb9136b4c61c2f8d56479289832ea07346e72ad): loading container with id \"33f92b150cb8bbdba587f2e70cb9136b4c61c2f8d56479289832ea07346e72ad\": container \"33f92b150cb8bbdba587f2e70cb9136b4c61c2f8d56479289832ea07346e72ad\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod flux-system/kustomize-controller-644f79985c-w7bzc: cannot find container (ID: containerd://8a6ae0c1f4bfff10a61662edf49088726c5bdcf2219b3f2386aea98483a08f33): loading container with id \"8a6ae0c1f4bfff10a61662edf49088726c5bdcf2219b3f2386aea98483a08f33\": container \"8a6ae0c1f4bfff10a61662edf49088726c5bdcf2219b3f2386aea98483a08f33\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod flux-system/notification-controller-ddf44665d-ng8nw: cannot find container (ID: containerd://33b143b47fc46f804153c0f77bde3e705535128b052e7f2a72dae56d4893cced): loading container with id \"33b143b47fc46f804153c0f77bde3e705535128b052e7f2a72dae56d4893cced\": container \"33b143b47fc46f804153c0f77bde3e705535128b052e7f2a72dae56d4893cced\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod flux-system/source-controller-9dfbc5cd-vp8zq: cannot find container (ID: containerd://f4ff1ff76bc21b5a4450cb351b1869b87ceaa34713beb0afae3dbca8d54df91a): loading container with id \"f4ff1ff76bc21b5a4450cb351b1869b87ceaa34713beb0afae3dbca8d54df91a\": container \"f4ff1ff76bc21b5a4450cb351b1869b87ceaa34713beb0afae3dbca8d54df91a\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod gadget/gadget-rp8hd: cannot find container (ID: containerd://baa3f8652dbc3b1d723889b52158f5156324791bf8a62ad5d119a17b22907f75): loading container with id \"baa3f8652dbc3b1d723889b52158f5156324791bf8a62ad5d119a17b22907f75\": container \"baa3f8652dbc3b1d723889b52158f5156324791bf8a62ad5d119a17b22907f75\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod heimdall/heimdall-0: cannot find container (ID: containerd://2ff69ed044f353b25172a950ab8fe162ab6a44d1732380c403712d32531183a9): loading container with id \"2ff69ed044f353b25172a950ab8fe162ab6a44d1732380c403712d32531183a9\": container \"2ff69ed044f353b25172a950ab8fe162ab6a44d1732380c403712d32531183a9\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod heimdall/heimdall-postgresql-0: cannot find container (ID: containerd://341fa5ce671e84016b90d436711771eecbceb7ff5bcf2ee06c4ab4a1b7ae175a): loading container with id \"341fa5ce671e84016b90d436711771eecbceb7ff5bcf2ee06c4ab4a1b7ae175a\": container \"341fa5ce671e84016b90d436711771eecbceb7ff5bcf2ee06c4ab4a1b7ae175a\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod jellyfin/jellyfin-6d6dd8765c-dtq6d: cannot find container (ID: containerd://b15081a20ce0b7617aaa474ed2a32503b473d763012e74176ac3fa68695164a4): loading container with id \"b15081a20ce0b7617aaa474ed2a32503b473d763012e74176ac3fa68695164a4\": container \"b15081a20ce0b7617aaa474ed2a32503b473d763012e74176ac3fa68695164a4\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod jupyter-system/continuous-image-puller-x2bj7: cannot find container (ID: containerd://815e3bb5252a27f92c59b18e741c6fad69ad5acbf2b0a72be1ee0834467dc577): loading container with id \"815e3bb5252a27f92c59b18e741c6fad69ad5acbf2b0a72be1ee0834467dc577\": container \"815e3bb5252a27f92c59b18e741c6fad69ad5acbf2b0a72be1ee0834467dc577\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod jupyter-system/proxy-677859796b-6jhs6: cannot find container (ID: containerd://877666003dd8920f706a49ad31c817642e6bf408944ec116698448b74bb47b37): loading container with id \"877666003dd8920f706a49ad31c817642e6bf408944ec116698448b74bb47b37\": container \"877666003dd8920f706a49ad31c817642e6bf408944ec116698448b74bb47b37\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod jupyter-system/user-scheduler-8c47f5f75-4cxf7: cannot find container (ID: containerd://f3205dc9d65f4b99580b3983401d28e4b2f5f1869ef6ce73a9d692a9ae7424a3): loading container with id \"f3205dc9d65f4b99580b3983401d28e4b2f5f1869ef6ce73a9d692a9ae7424a3\": container \"f3205dc9d65f4b99580b3983401d28e4b2f5f1869ef6ce73a9d692a9ae7424a3\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod jupyter-system/user-scheduler-8c47f5f75-h56lb: cannot find container (ID: containerd://8327d5e9c362c1c42b2df69aca8814a3a4d8067e942e7d1af0c834e6fe138fe5): loading container with id \"8327d5e9c362c1c42b2df69aca8814a3a4d8067e942e7d1af0c834e6fe138fe5\": container \"8327d5e9c362c1c42b2df69aca8814a3a4d8067e942e7d1af0c834e6fe138fe5\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod kube-system/coredns-77ccd57875-hppwj: cannot find container (ID: containerd://e53028d0f0297071ea9291d4271a81b377c4f7092cd4273d9620fd4e55835e6f): loading container with id \"e53028d0f0297071ea9291d4271a81b377c4f7092cd4273d9620fd4e55835e6f\": container \"e53028d0f0297071ea9291d4271a81b377c4f7092cd4273d9620fd4e55835e6f\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod kube-system/local-path-provisioner-957fdf8bc-9tnlx: cannot find container (ID: containerd://82b9dc96d6283392ec32778159943c3c416385acb781c22716b88cf512550e8e): loading container with id \"82b9dc96d6283392ec32778159943c3c416385acb781c22716b88cf512550e8e\": container \"82b9dc96d6283392ec32778159943c3c416385acb781c22716b88cf512550e8e\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod kube-system/metrics-server-648b5df564-zfnhv: cannot find container (ID: containerd://0f815dcaded37542b472960d2827ebf583139706a8101be7e102db3aee2aac74): loading container with id \"0f815dcaded37542b472960d2827ebf583139706a8101be7e102db3aee2aac74\": container \"0f815dcaded37542b472960d2827ebf583139706a8101be7e102db3aee2aac74\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod littlelink/littlelink-server-5f5b9758d5-vgsgj: cannot find container (ID: containerd://f5bfcd1a7f6b1153256b642c1d4ad0a07de10f1692bbebe9ad89fcb4685a6c18): loading container with id \"f5bfcd1a7f6b1153256b642c1d4ad0a07de10f1692bbebe9ad89fcb4685a6c18\": container \"f5bfcd1a7f6b1153256b642c1d4ad0a07de10f1692bbebe9ad89fcb4685a6c18\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod littlelink/littlelink-server-5f5b9758d5-x4mkg: cannot find container (ID: containerd://3084ff0360f1e4c09f1df5b8501746fa33f4ed34ecb9ff4bab2a17543fe97dcb): loading container with id \"3084ff0360f1e4c09f1df5b8501746fa33f4ed34ecb9ff4bab2a17543fe97dcb\": container \"3084ff0360f1e4c09f1df5b8501746fa33f4ed34ecb9ff4bab2a17543fe97dcb\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-attacher-f98ff75fb-9nq6v: cannot find container (ID: containerd://797b364eab817079ae0b8a2a430bb5027797310dd9698e44756da7f7cc315aec): loading container with id \"797b364eab817079ae0b8a2a430bb5027797310dd9698e44756da7f7cc315aec\": container \"797b364eab817079ae0b8a2a430bb5027797310dd9698e44756da7f7cc315aec\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-attacher-f98ff75fb-nm9lf: cannot find container (ID: containerd://77c182df05a47dc70af46840d7ac27a230005c1fdd737ea4c5a2112db29faf71): loading container with id \"77c182df05a47dc70af46840d7ac27a230005c1fdd737ea4c5a2112db29faf71\": container \"77c182df05a47dc70af46840d7ac27a230005c1fdd737ea4c5a2112db29faf71\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-attacher-f98ff75fb-w89tw: cannot find container (ID: containerd://d25fb233fdc68411debbbd307b54eec8aa3b70deaedd3f1f8cfab75f9db083c8): loading container with id \"d25fb233fdc68411debbbd307b54eec8aa3b70deaedd3f1f8cfab75f9db083c8\": container \"d25fb233fdc68411debbbd307b54eec8aa3b70deaedd3f1f8cfab75f9db083c8\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-provisioner-65cb5cc4ff-h592p: cannot find container (ID: containerd://152d3c1e5191be8b2c745fdafc82c6ce3fb396f94270408d275969e1dd383177): loading container with id \"152d3c1e5191be8b2c745fdafc82c6ce3fb396f94270408d275969e1dd383177\": container \"152d3c1e5191be8b2c745fdafc82c6ce3fb396f94270408d275969e1dd383177\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-provisioner-65cb5cc4ff-kj5qp: cannot find container (ID: containerd://42705a2079fb8573d04371c84fdfaa6fec21109d7b787f16a6d2f9ed97102405): loading container with id \"42705a2079fb8573d04371c84fdfaa6fec21109d7b787f16a6d2f9ed97102405\": container \"42705a2079fb8573d04371c84fdfaa6fec21109d7b787f16a6d2f9ed97102405\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-provisioner-65cb5cc4ff-vf752: cannot find container (ID: containerd://bf98878f054117b508518da7c2b455c7528bdcc24671c0082a0fd7fef86c1432): loading container with id \"bf98878f054117b508518da7c2b455c7528bdcc24671c0082a0fd7fef86c1432\": container \"bf98878f054117b508518da7c2b455c7528bdcc24671c0082a0fd7fef86c1432\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-resizer-77cddbfc7c-2lmbz: cannot find container (ID: containerd://45abbdc9acab6e5b610211f655a68b186053436f70090f53282e1905827681d3): loading container with id \"45abbdc9acab6e5b610211f655a68b186053436f70090f53282e1905827681d3\": container \"45abbdc9acab6e5b610211f655a68b186053436f70090f53282e1905827681d3\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-resizer-77cddbfc7c-7qz6t: cannot find container (ID: containerd://bf7f89e376381451173f12c38b6e0fcbf03f9f1a289ff144234a5fd453d3daee): loading container with id \"bf7f89e376381451173f12c38b6e0fcbf03f9f1a289ff144234a5fd453d3daee\": container \"bf7f89e376381451173f12c38b6e0fcbf03f9f1a289ff144234a5fd453d3daee\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-resizer-77cddbfc7c-dbf97: cannot find container (ID: containerd://759eb29a93845b9b50bb0886929e09f00500e65d2f9112bd8bf69f5a80f3f9d9): loading container with id \"759eb29a93845b9b50bb0886929e09f00500e65d2f9112bd8bf69f5a80f3f9d9\": container \"759eb29a93845b9b50bb0886929e09f00500e65d2f9112bd8bf69f5a80f3f9d9\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-snapshotter-78cb65cc4f-4zh52: cannot find container (ID: containerd://317d8e73b219d3d7ec3ca98f64b2e3f667b52622474bb5dbd287f24e98043df1): loading container with id \"317d8e73b219d3d7ec3ca98f64b2e3f667b52622474bb5dbd287f24e98043df1\": container \"317d8e73b219d3d7ec3ca98f64b2e3f667b52622474bb5dbd287f24e98043df1\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-snapshotter-78cb65cc4f-cd4hz: cannot find container (ID: containerd://b117159fbb8447bdecd4b240f37cada7dce614fb90cc6a2a6cdc3c91f6f4feba): loading container with id \"b117159fbb8447bdecd4b240f37cada7dce614fb90cc6a2a6cdc3c91f6f4feba\": container \"b117159fbb8447bdecd4b240f37cada7dce614fb90cc6a2a6cdc3c91f6f4feba\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/csi-snapshotter-78cb65cc4f-mmzgs: cannot find container (ID: containerd://0918fdb1a50f3523e76b3c60b4871a0cda6b12ddd222541131bd57cf405afb02): loading container with id \"0918fdb1a50f3523e76b3c60b4871a0cda6b12ddd222541131bd57cf405afb02\": container \"0918fdb1a50f3523e76b3c60b4871a0cda6b12ddd222541131bd57cf405afb02\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/engine-image-ei-68f17757-g65hr: cannot find container (ID: containerd://3ad02ab5fb97fdefcfe3c2e6d5d12af5eaa8e9d2deaa131b3f90609cee5227e2): loading container with id \"3ad02ab5fb97fdefcfe3c2e6d5d12af5eaa8e9d2deaa131b3f90609cee5227e2\": container \"3ad02ab5fb97fdefcfe3c2e6d5d12af5eaa8e9d2deaa131b3f90609cee5227e2\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/instance-manager-d68bc6b2e4a8343ec64946ede4e95a11: cannot find container (ID: containerd://0bf867bc19888d91662d80a0554e47ce3b354271e0e96b62d703d40f8cbf5fb3): loading container with id \"0bf867bc19888d91662d80a0554e47ce3b354271e0e96b62d703d40f8cbf5fb3\": container \"0bf867bc19888d91662d80a0554e47ce3b354271e0e96b62d703d40f8cbf5fb3\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/longhorn-csi-plugin-txjmf: cannot find container (ID: containerd://71e005f67baf5275a9441d60572930a4f891a0cc023349b6371d560aeaa8d13f): loading container with id \"71e005f67baf5275a9441d60572930a4f891a0cc023349b6371d560aeaa8d13f\": container \"71e005f67baf5275a9441d60572930a4f891a0cc023349b6371d560aeaa8d13f\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/longhorn-csi-plugin-txjmf: cannot find container (ID: containerd://bd1a735e7e5f42136f3c929381e885efa600bf6c20d992abd7bccf5d15b2d9df): loading container with id \"bd1a735e7e5f42136f3c929381e885efa600bf6c20d992abd7bccf5d15b2d9df\": container \"bd1a735e7e5f42136f3c929381e885efa600bf6c20d992abd7bccf5d15b2d9df\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/longhorn-csi-plugin-txjmf: cannot find container (ID: containerd://92eaa3ba33f8952bc6fde665898b8a3e4722fe5ffb37057ccb0abcc12135ea4c): loading container with id \"92eaa3ba33f8952bc6fde665898b8a3e4722fe5ffb37057ccb0abcc12135ea4c\": container \"92eaa3ba33f8952bc6fde665898b8a3e4722fe5ffb37057ccb0abcc12135ea4c\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/longhorn-driver-deployer-5f8f97f67c-776nz: cannot find container (ID: containerd://6de88cf925e6f54c8b7b58f76ae159f63db44f3bd4244ee6fa93f7999a2ceceb): loading container with id \"6de88cf925e6f54c8b7b58f76ae159f63db44f3bd4244ee6fa93f7999a2ceceb\": container \"6de88cf925e6f54c8b7b58f76ae159f63db44f3bd4244ee6fa93f7999a2ceceb\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/longhorn-manager-p2q46: cannot find container (ID: containerd://43bf91b62477593a2a35d50a673fe7c9800c4320be181c12b1c270628c4ff2ee): loading container with id \"43bf91b62477593a2a35d50a673fe7c9800c4320be181c12b1c270628c4ff2ee\": container \"43bf91b62477593a2a35d50a673fe7c9800c4320be181c12b1c270628c4ff2ee\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/longhorn-ui-d6f97df49-lj2nh: cannot find container (ID: containerd://d7ef9b8538fcb078ccaf052d38537f8e4511ef65434dfdd4fa81751e0ab05264): loading container with id \"d7ef9b8538fcb078ccaf052d38537f8e4511ef65434dfdd4fa81751e0ab05264\": container \"d7ef9b8538fcb078ccaf052d38537f8e4511ef65434dfdd4fa81751e0ab05264\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod longhorn-system/longhorn-ui-d6f97df49-ptjpj: cannot find container (ID: containerd://0d8eb7563d3e60c221296d336f83f6a2ba12b98aac8ca408c123d625b83ca4b8): loading container with id \"0d8eb7563d3e60c221296d336f83f6a2ba12b98aac8ca408c123d625b83ca4b8\": container \"0d8eb7563d3e60c221296d336f83f6a2ba12b98aac8ca408c123d625b83ca4b8\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod metallb-system/controller-6b9fd67ff4-82bms: cannot find container (ID: containerd://1b5c01352e08be4b37dc963185aa24429bf8f5246b40b9d08db8e4f99857063b): loading container with id \"1b5c01352e08be4b37dc963185aa24429bf8f5246b40b9d08db8e4f99857063b\": container \"1b5c01352e08be4b37dc963185aa24429bf8f5246b40b9d08db8e4f99857063b\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod metallb-system/speaker-wz9vs: cannot find container (ID: containerd://c47b181a744afdb230a9f479d0ef2964c361ff8be7db3b6d78135b716d09c172): loading container with id \"c47b181a744afdb230a9f479d0ef2964c361ff8be7db3b6d78135b716d09c172\": container \"c47b181a744afdb230a9f479d0ef2964c361ff8be7db3b6d78135b716d09c172\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod mysql-operator/mysql-operator-79567964fc-tkc5j: cannot find container (ID: containerd://6d52d99ca43f2bd5f9050356e0e955bb99eb000e643c6bedb611f3c3db7db1f7): loading container with id \"6d52d99ca43f2bd5f9050356e0e955bb99eb000e643c6bedb611f3c3db7db1f7\": container \"6d52d99ca43f2bd5f9050356e0e955bb99eb000e643c6bedb611f3c3db7db1f7\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod tekton-pipelines-resolvers/tekton-pipelines-remote-resolvers-6d4c4fc854-w5rp9: cannot find container (ID: containerd://0a04c2ba8e3a4d61d5a0fbbda783dbaf047cb4d46a6572f784b65ee47a0abed5): loading container with id \"0a04c2ba8e3a4d61d5a0fbbda783dbaf047cb4d46a6572f784b65ee47a0abed5\": container \"0a04c2ba8e3a4d61d5a0fbbda783dbaf047cb4d46a6572f784b65ee47a0abed5\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod tekton-pipelines/tekton-events-controller-744496c748-skpdz: cannot find container (ID: containerd://b6f6e4a05d75759a21d9905021ce33a2d1e9a203b7ebf6b481dd8ff671ca10fb): loading container with id \"b6f6e4a05d75759a21d9905021ce33a2d1e9a203b7ebf6b481dd8ff671ca10fb\": container \"b6f6e4a05d75759a21d9905021ce33a2d1e9a203b7ebf6b481dd8ff671ca10fb\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod tekton-pipelines/tekton-pipelines-controller-67795468b5-czfnc: cannot find container (ID: containerd://9b2849fb2041b278a86da7f1872225874041fa76c5f9062bb2903f9fb0b1a028): loading container with id \"9b2849fb2041b278a86da7f1872225874041fa76c5f9062bb2903f9fb0b1a028\": container \"9b2849fb2041b278a86da7f1872225874041fa76c5f9062bb2903f9fb0b1a028\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod tekton-pipelines/tekton-pipelines-webhook-7cffbffd66-vrmdm: cannot find container (ID: containerd://784b99634f8d1bd1a22851eaedf8c8a90e6751f9af6f63cffb8477214b798fe7): loading container with id \"784b99634f8d1bd1a22851eaedf8c8a90e6751f9af6f63cffb8477214b798fe7\": container \"784b99634f8d1bd1a22851eaedf8c8a90e6751f9af6f63cffb8477214b798fe7\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod tekton-pipelines/tekton-triggers-controller-74b57b7fdc-4k5vw: cannot find container (ID: containerd://fb929d6892e0c9c82305447a02d32965ea3fdd0be5684b6e30a9188d4f806119): loading container with id \"fb929d6892e0c9c82305447a02d32965ea3fdd0be5684b6e30a9188d4f806119\": container \"fb929d6892e0c9c82305447a02d32965ea3fdd0be5684b6e30a9188d4f806119\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod tekton-pipelines/tekton-triggers-core-interceptors-cff9f87df-p6sfq: cannot find container (ID: containerd://a8b3a58e63b4718770caf10eb2ffd656826e3861b32947f095c7bab0acb16482): loading container with id \"a8b3a58e63b4718770caf10eb2ffd656826e3861b32947f095c7bab0acb16482\": container \"a8b3a58e63b4718770caf10eb2ffd656826e3861b32947f095c7bab0acb16482\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod tekton-pipelines/tekton-triggers-webhook-674ff6656d-r5vqq: cannot find container (ID: containerd://2fd9934abafbd4b63a1eed9dd91b5f4768cde6791775aec0a9d4a89be983f3ee): loading container with id \"2fd9934abafbd4b63a1eed9dd91b5f4768cde6791775aec0a9d4a89be983f3ee\": container \"2fd9934abafbd4b63a1eed9dd91b5f4768cde6791775aec0a9d4a89be983f3ee\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod traefik/traefik-traefik-external-6957cbddb-6rkm9: cannot find container (ID: containerd://86cfed42dbe0242fe6184a2a2cc39d127e1d318be1f99c2457846beed85eb4ca): loading container with id \"86cfed42dbe0242fe6184a2a2cc39d127e1d318be1f99c2457846beed85eb4ca\": container \"86cfed42dbe0242fe6184a2a2cc39d127e1d318be1f99c2457846beed85eb4ca\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod traefik/traefik-traefik-internal-954d5f84b-sdpxr: cannot find container (ID: containerd://88a53428f6d70978dc5b289caabbccd751a54f5ee1e1c7ada8230417d62d368f): loading container with id \"88a53428f6d70978dc5b289caabbccd751a54f5ee1e1c7ada8230417d62d368f\": container \"88a53428f6d70978dc5b289caabbccd751a54f5ee1e1c7ada8230417d62d368f\" in namespace \"k8s.io\": not found"
time="2024-05-30T23:41:26Z" level=warning msg="Skip pod vaultwarden/vaultwarden-6cb76c7684-9xsvk: cannot find container (ID: containerd://37925ea42931cc1aeeefc1f3980eeddcf84d635c42d910f385ae2925e38e4079): loading container with id \"37925ea42931cc1aeeefc1f3980eeddcf84d635c42d910f385ae2925e38e4079\": container \"37925ea42931cc1aeeefc1f3980eeddcf84d635c42d910f385ae2925e38e4079\" in namespace \"k8s.io\": not found"