GELF loggind driver

[rusk0]

Greetings,

Can anyone give me some hints how can i send logs from containerd 1.6.19 to graylog using the GELF logging driver ?

I have set in my /etc/containerd/config.toml :

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
LogDriver = "gelf"
LogOpt = ["gelf-address=udp://1.2.3.4:12022"]
SystemdCgroup = true
Environment = ["ENVIRON=testing"]
(redacted values)

I can see the configuration is applied as per /var/log/messages :
Root: Options:map[Environment:[ENVIRON=testing] LogDriver:gelf LogOpt:[gelf-address=udp://1234:12022]

However i cannot see any traffic from the containerd server to the specified IP and/or port.

I can see i can connect to the graylog server/port :

ncat -u -zvw 2 172.16.6.91 30239
Ncat: Connected to 172.16.6.91:30239.

I have started multiple containers on the containerd server in multiple ways, manually created logs to make sure there's data to be sent, but i cannot detect nothing being sent to my graylog server...

sudo tcpdump -i any -n -X udp port 12022
(for example)

Any clues are greatly appreciated.

PS:

I was upgrading a kubernetes cluster which was using docker engine, where sending logs to graylog in a similar way worked fine. I am simply trying to rework it for containerd engine.

[AkihiroSuda]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
LogDriver = "gelf"
LogOpt = ["gelf-address=udp://1.2.3.4:12022"]
SystemdCgroup = true
Environment = ["ENVIRON=testing"]
(redacted values)

These configuration items except SystemdCgroup do not exist: https://github.com/containerd/containerd/blob/v1.6.24/runtime/v2/runc/options/oci.proto

gelf driver is currently not implemented in containerd.
A workaround would be to watch /var/log/containers with fluentbit or something else:
https://docs.fluentbit.io/manual/pipeline/outputs/gelf