kubeadm init panic when log_to_journald = true

[cuisongliu]

What happened?

using crio and config log_to_journald = true and log_level = "info" panic

DEBU[2023-03-16 14:43:36.785110154+08:00] Running conmon under slice system.slice and unitName crio-conmon-219c5cd9e8fb77ca729fb4f07772162b59248b43deaaf2e0ae1c65bb6725a391.scope  file="cgmgr/systemd.go:140"
[conmon:e] Include journald in compilation path to log to systemd journal
DEBU[2023-03-16 14:43:36.816926100+08:00] received signal                               file="crio/main.go:58" signal="broken pipe"
INFO[2023-03-16 14:43:36.816907473+08:00] createCtr: deleting container ID 219c5cd9e8fb77ca729fb4f07772162b59248b43deaaf2e0ae1c65bb6725a391 from idIndex  file="server/container_create.go:497" id=4cb37d87-2042-4b8a-9163-d8a012452b76 name=/runtime.v1.RuntimeService/CreateContainer
INFO[2023-03-16 14:43:36.816960772+08:00] createCtr: deleting container ID 219c5cd9e8fb77ca729fb4f07772162b59248b43deaaf2e0ae1c65bb6725a391 from idIndex  file="server/container_create.go:487" id=4cb37d87-2042-4b8a-9163-d8a012452b76 name=/runtime.v1.RuntimeService/CreateContainer
INFO[2023-03-16 14:43:36.816975149+08:00] createCtr: deleting container ID 219c5cd9e8fb77ca729fb4f07772162b59248b43deaaf2e0ae1c65bb6725a391 from idIndex  file="server/container_create.go:476" id=4cb37d87-2042-4b8a-9163-d8a012452b76 name=/runtime.v1.RuntimeService/CreateContainer
INFO[2023-03-16 14:43:36.820216303+08:00] createCtr: deleting container ID 219c5cd9e8fb77ca729fb4f07772162b59248b43deaaf2e0ae1c65bb6725a391 from idIndex  file="server/container_create.go:465" id=4cb37d87-2042-4b8a-9163-d8a012452b76 name=/runtime.v1.RuntimeService/CreateContainer
DEBU[2023-03-16 14:43:36.820289231+08:00] Response error: write child: broken pipe
failed to wait exit status 1 after failing with
github.com/cri-o/cri-o/internal/oci.(*runtimeOCI).CreateContainer.func1.1
	github.com/cri-o/cri-o/internal/oci/runtime_oci.go:186
github.com/cri-o/cri-o/internal/oci.(*runtimeOCI).CreateContainer.func1
	github.com/cri-o/cri-o/internal/oci/runtime_oci.go:198
github.com/cri-o/cri-o/internal/oci.(*runtimeOCI).CreateContainer
	github.com/cri-o/cri-o/internal/oci/runtime_oci.go:199
github.com/cri-o/cri-o/internal/oci.(*Runtime).CreateContainer
	github.com/cri-o/cri-o/internal/oci/oci.go:276
github.com/cri-o/cri-o/server.(*Server).createContainerPlatform
	github.com/cri-o/cri-o/server/container_create_linux.go:58
github.com/cri-o/cri-o/server.(*Server).CreateContainer
	github.com/cri-o/cri-o/server/container_create.go:515
github.com/cri-o/cri-o/server/cri/v1.(*service).CreateContainer
	github.com/cri-o/cri-o/server/cri/v1/rpc_create_container.go:12
k8s.io/cri-api/pkg/apis/runtime/v1._RuntimeService_CreateContainer_Handler.func1
	k8s.io/cri-api@v0.23.0-alpha.3/pkg/apis/runtime/v1/api.pb.go:8983
github.com/cri-o/cri-o/internal/log.UnaryInterceptor.func1
	github.com/cri-o/cri-o/internal/log/interceptors.go:56
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/cri-o/cri-o/server/metrics.UnaryInterceptor.func1
	github.com/cri-o/cri-o/server/metrics/interceptors.go:24
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34
k8s.io/cri-api/pkg/apis/runtime/v1._RuntimeService_CreateContainer_Handler
	k8s.io/cri-api@v0.23.0-alpha.3/pkg/apis/runtime/v1/api.pb.go:8985
google.golang.org/grpc.(*Server).processUnaryRPC
	google.golang.org/grpc@v1.42.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
	google.golang.org/grpc@v1.42.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
	google.golang.org/grpc@v1.42.0/server.go:921
runtime.goexit

What did you expect to happen?

kubeadm init success

How can we reproduce it (as minimally and precisely as possible)?

add log_to_journald = true to crio.conf

Anything else we need to know?

No response

CRI-O and Kubernetes version

$ crio --version
# paste output here
crio version 1.23.5
Version:          1.23.5
GitCommit:        d12c2842e9e4d051a2496e1740f846ffa22b3f1d
GitTreeState:     dirty
BuildDate:        1980-01-01T00:00:00Z
GoVersion:        go1.17.4
Compiler:         gc
Platform:         linux/arm64
Linkmode:         static
BuildTags:        apparmor, exclude_graphdriver_devicemapper, seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  true

$ kubectl version
# paste output here
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/arm64"}
The connection to the server apiserver.cluster.local:6443 was refused - did you specify the right host or port?

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
# paste output here
Linux default-node-0 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

VirtualBox

[cuisongliu]

DEBU[2023-03-16 14:43:35.639808316+08:00] running conmon: /usr/bin/conmon               args="[-b /run/crio/overlay-containers/31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88/userdata -c 31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88 --exit-dir /var/run/crio/exits -l /var/log/pods/kube-system_etcd-default-node-0_d904f746e688e181e743815485a72513/etcd/17.log --log-level debug -n k8s_etcd_etcd-default-node-0_kube-system_d904f746e688e181e743815485a72513_17 -P /run/crio/overlay-containers/31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88/userdata/conmon-pidfile -p /run/crio/overlay-containers/31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88/userdata/pidfile --persist-dir /var/lib/crio/overlay-containers/31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88/userdata -r /usr/bin/crun --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio -u 31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88 -s --log-path journald:]" file="oci/runtime_oci.go:139"
DEBU[2023-03-16 14:43:35.640032308+08:00] Running conmon under slice system.slice and unitName crio-conmon-31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88.scope  file="cgmgr/systemd.go:140"

[haircommander]

I'm guessing conmon wasn't compiled with journald support. to confirm, you could get journalctl -u crio-conmon-31877e4450ed20c0c1f0379a0b9e1a828fa78a98277c84fd9d003661a9934c88.scope which should print the output of conmon

Also, is that a segfault? like did the crio server crash after that? It looks like normal stacktrace printed with debug logs

[cuisongliu]

The crio service is normal, but the startup sandbox container reports an error.

broken pipe this error happen in create sandbox

[haircommander]

yeah so that error happens because conmon wasn't compiled with journald support. I recommend building conmon from scratch like in containers/conmon#348 (comment)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

Closing this issue since it had no activity in the past 90 days.