Segmentation violation in podman/factory.go

[crrrazyden]

Context:

Ubuntu 22.04
Docker 24.0.5 (installed, containers present)
Podman 3.4.4
Cadvisor image v.0.48-test

Launching the advisor:

podman run \
  --pod=${POD_NAME} \
  --volume=/:/rootfs:ro \
  --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
  --volume=/var/run/podman:/var/run/podman:ro \
  --volume=/dev/disk:/dev/disk:ro \
  --volume=/dev/rbd:/dev/rbd:ro \
  --detach=true \
  --name=cadvisor \
  --privileged=true \
  --device=/dev/kmsg \
  --ulimit=host \
  --memory=512Mi \
  --cpus=0.5 \
  gcr.io/cadvisor/cadvisor-amd64:0.48-test \
  -listen_ip "127.0.0.1" \
  -port 8081

After several hours of work:

I0130 19:09:41.796163       1 manager.go:234] Version: {KernelVersion:5.15.0-78-generic ContainerOsVersion:Alpine Linux v3.16 DockerVersion: DockerAPIVersion: CadvisorVersion:0.48-test CadvisorRevision:d19df4f3}
I0130 19:09:41.798298       1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I0130 19:09:41.798460       1 factory.go:219] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
I0130 19:09:41.866494       1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I0130 19:09:41.867083       1 factory.go:219] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I0130 19:09:42.265240       1 plugin.go:89] Registering Podman factory
I0130 19:09:42.515637       1 factory.go:221] Registration of the podman container factory successfully
I0130 19:09:42.515691       1 factory.go:55] Registering systemd factory
I0130 19:09:42.515697       1 factory.go:221] Registration of the systemd container factory successfully
I0130 19:09:42.515738       1 factory.go:103] Registering Raw factory
I0130 19:09:42.515766       1 manager.go:1197] Started watching for new ooms in manager
I0130 19:09:42.516438       1 manager.go:320] Starting recovery of all containers
I0130 19:09:43.188875       1 manager.go:325] Recovery completed
I0130 19:09:43.465734       1 cadvisor.go:176] Starting cAdvisor version: 0.48-test-d19df4f3 on port 8081
W0130 19:56:57.358793       1 container.go:586] Failed to update stats for container "/machine.slice/libpod-conmon-71e4cc69b9026c33b61e75c899077da2b83eb8fc47bce561bf005c45b0deeee3.scope": unable to determine device info for dir: /rootfs/var/lib/containers/storage/overlay/a09587307b77588512828b2361f6500e1f1a330f3c4ce5d91bd61d510b0a7e66/diff: stat failed on /rootfs/var/lib/containers/storage/overlay/a09587307b77588512828b2361f6500e1f1a330f3c4ce5d91bd61d510b0a7e66/diff with error: no such file or directory, continuing to push stats
W0130 22:01:17.923524       1 container.go:586] Failed to update stats for container "/machine.slice/libpod-conmon-dd4f00f3dfc1a6f86bb919d55de542b0f3914de4ef45a8e007b0d865bf450194.scope": unable to determine device info for dir: /rootfs/var/lib/containers/storage/overlay/d0322b84dcf03a7e5425d53e89a2e7ab206a8d666a276930a3748359c0bd33d3/diff: stat failed on /rootfs/var/lib/containers/storage/overlay/d0322b84dcf03a7e5425d53e89a2e7ab206a8d666a276930a3748359c0bd33d3/diff with error: no such file or directory, continuing to push stats
W0131 01:07:46.665855       1 watcher.go:93] Error while processing event ("/sys/fs/cgroup/machine.slice/libpod-164b8a22da7973874468f0da39de5af2c682a1fc4a937b0970d25f0091f9febb.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/machine.slice/libpod-164b8a22da7973874468f0da39de5af2c682a1fc4a937b0970d25f0091f9febb.scope: no such file or directory
W0131 01:07:47.165967       1 manager.go:1170] Failed to process watch event {EventType:0 Name:/machine.slice/libpod-conmon-164b8a22da7973874468f0da39de5af2c682a1fc4a937b0970d25f0091f9febb.scope WatchSource:0}: item not found
W0131 01:38:53.396352       1 container.go:586] Failed to update stats for container "/machine.slice/libpod-conmon-98ce6e1fa911d55d48de1ac53d50b508639f1728bb8c8e24d54058c93aa124be.scope": unable to determine device info for dir: /rootfs/var/lib/containers/storage/overlay/3adbab70e57c4ef1398c3a76b50572e9ba8dd82d1f523409546285238acebd75/diff: stat failed on /rootfs/var/lib/containers/storage/overlay/3adbab70e57c4ef1398c3a76b50572e9ba8dd82d1f523409546285238acebd75/diff with error: no such file or directory, continuing to push stats
W0131 03:12:06.065878       1 watcher.go:93] Error while processing event ("/sys/fs/cgroup/machine.slice/libpod-025f0c05bb8825ce46916143592800bdb7f06e8d5f3fc9f1cd7857d2cdb34d00.scope": 0x40000100 == IN_CREATE|IN_ISDIR): readdirent /sys/fs/cgroup/machine.slice/libpod-025f0c05bb8825ce46916143592800bdb7f06e8d5f3fc9f1cd7857d2cdb34d00.scope: no such file or directory
W0131 03:12:06.466304       1 manager.go:1170] Failed to process watch event {EventType:0 Name:/machine.slice/libpod-conmon-025f0c05bb8825ce46916143592800bdb7f06e8d5f3fc9f1cd7857d2cdb34d00.scope WatchSource:0}: item not found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x9cd882]

goroutine 626 [running]:
github.com/google/cadvisor/container/podman.(*podmanFactory).CanHandleAndAccept(0x10?, {0xc00089b73e, 0xc003690420?})
        /go/src/github.com/google/cadvisor/container/podman/factory.go:94 +0x182
github.com/google/cadvisor/container.NewContainerHandler({0xc00089b73e, 0x5c}, 0x2335180?, {0xc0006c4a70, 0x1, 0x1}, 0x25?)
        /go/src/github.com/google/cadvisor/container/factory.go:266 +0x182
github.com/google/cadvisor/manager.(*manager).createContainerLocked(0xc00092c580, {0xc00089b73e, 0x5c}, 0xc000bc98e8?)
        /go/src/github.com/google/cadvisor/manager/manager.go:924 +0xcf
github.com/google/cadvisor/manager.(*manager).createContainer(0xc00092c580, {0xc00089b73e, 0x5c}, 0x0?)
        /go/src/github.com/google/cadvisor/manager/manager.go:911 +0xa5
github.com/google/cadvisor/manager.(*manager).watchForNewContainers.func1()
        /go/src/github.com/google/cadvisor/manager/manager.go:1164 +0x186
created by github.com/google/cadvisor/manager.(*manager).watchForNewContainers
        /go/src/github.com/google/cadvisor/manager/manager.go:1156 +0x27b

journalctl -u podman

Jan 31 06:18:36 tpa0004 systemd[1]: Starting Podman API Service...
Jan 31 06:18:36 tpa0004 systemd[1]: Started Podman API Service.
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="/usr/bin/podman filtering at log level info"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="[graphdriver] using prior storage driver: overlay"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="Found CNI network net-bsc-client-node-1-219 (type=macvlan) at /etc/cni/net.d/net-bsc-client-node-1-219.conflist"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="Found CNI network net-bsc-lb-3-219 (type=macvlan) at /etc/cni/net.d/net-bsc-lb-3-219.conflist"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="Setting parallel job count to 385"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="using systemd socket activation to determine API endpoint"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="using API endpoint: ''"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="API service listening on \"/run/podman/podman.sock\""
Jan 31 06:18:36 tpa0004 podman[4012897]: 2024-01-31 06:18:36.568814295 +0000 UTC m=+0.090005569 container died 8c740d06f8e40821e7d8596de630c54f09e16ef1dccaa221a82db3c5ada64bd8 (image=quay.io/ceph/ceph@sha256:1161e35e4>
Jan 31 06:18:36 tpa0004 podman[4012897]: @ - - [31/Jan/2024:06:18:36 +0000] "GET /v1.0.0/containers/8c740d06f8e40821e7d8596de630c54f09e16ef1dccaa221a82db3c5ada64bd8/json HTTP/1.1" 200 4902 "" "Go-http-client/1.1"
Jan 31 06:18:36 tpa0004 podman[4012897]: time="2024-01-31T06:18:36Z" level=info msg="Request Failed(Internal Server Error): container 8c740d06f8e40821e7d8596de630c54f09e16ef1dccaa221a82db3c5ada64bd8 does not exist in >
Jan 31 06:18:36 tpa0004 podman[4012897]: @ - - [31/Jan/2024:06:18:36 +0000] "GET /v1.0.0/containers/8c740d06f8e40821e7d8596de630c54f09e16ef1dccaa221a82db3c5ada64bd8/json HTTP/1.1" 500 178 "" "Go-http-client/1.1"

[Cub0n]

With cAdvisor 0.49.1 on ARMv7 I don not encounter such errors.