v1.23.6-rc1+k3s1 fails to fully come up when installing via RPM with selinux enabled

[rancher-max]

Same steps and failure as mentioned in rancher/rke2#2808.

The only difference is that with k3s the failure only presents itself when specifically including the selinux: true config flag.

Note that the same posted workaround works:

yum install container-selinux-2:2.167.0-1.module+el8.5.0+710+4c471e88 'dnf-command(versionlock)' -y
yum versionlock container-selinux

[brandond]

Summary

For some reason, the change to containerd to strip inheritable capabilities (containerd/containerd@6940524) is causing only symlinks to be created as either tmpfs_t or container_runtime_tmpfs_t instead of container_file_t. The type depends on the policy version. The newer policy version results in things being labeled as container_runtime_tmpfs_t, which processes running in the container ( as system_u:system_r:container_t:s0:c472,c498 ) are denied access to. The new context seems to be intentional as per containers/container-selinux@57d36ab but I don't understand why they would start labeling things with this context if things executing as container_t cannot access them.

WORKING
K3s 1.23.5+k3s1 / containerd 1.5.10 with container-selinux-2:2.167.0

/var/lib/kubelet/pods/621e6794-d0dc-4595-884a-814c1c043b89/volumes/kubernetes.io~projected/kube-api-access-md9wz/:
total 0
drwxrwxrwt. 3 root root system_u:object_r:container_file_t:s0:c709,c886 140 Apr 22 17:02 .
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0            35 Apr 22 17:02 ..
drwxr-xr-x. 2 root root system_u:object_r:container_file_t:s0:c709,c886 100 Apr 22 17:02 ..2022_04_22_21_02_58.1077793130
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c709,c886  13 Apr 22 17:02 ca.crt -> ..data/ca.crt
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c709,c886  32 Apr 22 17:02 ..data -> ..2022_04_22_21_02_58.1077793130
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c709,c886  16 Apr 22 17:02 namespace -> ..data/namespace
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c709,c886  12 Apr 22 17:02 token -> ..data/token

WORKING
K3s 1.23.5+k3s1 / containerd 1.5.10 with container-selinux-2:2.173.0

/var/lib/kubelet/pods/1a4b5b86-d588-4a26-b08d-38f62aa7c03b/volumes/kubernetes.io~projected/kube-api-access-r7xcl/:
total 0
drwxrwxrwt. 3 root root system_u:object_r:container_file_t:s0:c441,c778 140 Apr 22 16:44 .
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0            35 Apr 22 16:44 ..
drwxr-xr-x. 2 root root system_u:object_r:container_file_t:s0:c441,c778 100 Apr 22 16:44 ..2022_04_22_20_44_45.716921218
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c441,c778  13 Apr 22 16:44 ca.crt -> ..data/ca.crt
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c441,c778  31 Apr 22 16:44 ..data -> ..2022_04_22_20_44_45.716921218
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c441,c778  16 Apr 22 16:44 namespace -> ..data/namespace
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c441,c778  12 Apr 22 16:44 token -> ..data/token

WORKING
K3s 1.23.6-rc1+k3s1 / containerd 1.5.11 with container-selinux-2:2.167.0

/var/lib/kubelet/pods/7eee1ef2-f90f-464b-a5f0-6a818755dd64/volumes/kubernetes.io~projected/kube-api-access-rvrqb/:
total 0
drwxrwxrwt. 3 root root system_u:object_r:container_file_t:s0:c89,c620 140 Apr 22 17:01 .
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0           35 Apr 22 17:01 ..
drwxr-xr-x. 2 root root system_u:object_r:container_file_t:s0:c89,c620 100 Apr 22 17:01 ..2022_04_22_21_01_13.766655981
lrwxrwxrwx. 1 root root system_u:object_r:tmpfs_t:s0                    13 Apr 22 17:01 ca.crt -> ..data/ca.crt
lrwxrwxrwx. 1 root root system_u:object_r:tmpfs_t:s0                    31 Apr 22 17:01 ..data -> ..2022_04_22_21_01_13.766655981
lrwxrwxrwx. 1 root root system_u:object_r:tmpfs_t:s0                    16 Apr 22 17:01 namespace -> ..data/namespace
lrwxrwxrwx. 1 root root system_u:object_r:tmpfs_t:s0                    12 Apr 22 17:01 token -> ..data/token

BROKEN
K3s 1.23.6-rc1+k3s1 / containerd 1.5.11 with container-selinux-2:2.173.0

/var/lib/kubelet/pods/11d7d712-43ee-4166-923a-bd39a0361fea/volumes/kubernetes.io~projected/kube-api-access-6pxb6/:
total 0
drwxrwxrwt. 3 root root system_u:object_r:container_file_t:s0:c123,c259 140 Apr 22 16:55 .
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0            35 Apr 22 16:55 ..
drwxr-xr-x. 2 root root system_u:object_r:container_file_t:s0:c123,c259 100 Apr 22 16:55 ..2022_04_22_20_55_15.1865202030
lrwxrwxrwx. 1 root root system_u:object_r:container_runtime_tmpfs_t:s0   13 Apr 22 16:55 ca.crt -> ..data/ca.crt
lrwxrwxrwx. 1 root root system_u:object_r:container_runtime_tmpfs_t:s0   32 Apr 22 16:55 ..data -> ..2022_04_22_20_55_15.1865202030
lrwxrwxrwx. 1 root root system_u:object_r:container_runtime_tmpfs_t:s0   16 Apr 22 16:55 namespace -> ..data/namespace
lrwxrwxrwx. 1 root root system_u:object_r:container_runtime_tmpfs_t:s0   12 Apr 22 16:55 token -> ..data/token

Details
The audit messages look like this, and prevent all containerized processes from reading the Kubernetes service-account tokens from tmpfs:

type=AVC msg=audit(1650661022.159:2097): avc: denied { read } for pid=71428 comm="coredns" name="token" dev="tmpfs" ino=537071 scontext=system_u:system_r:container_t:s0:c498,c537 tcontext=system_u:object_r:container_runtime_tmpfs_t:s0 tclass=lnk_file permissive=0
type=SYSCALL msg=audit(1650661022.159:2097): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=c000576840 a2=80000 a3=0 items=0 ppid=69301 pid=71428 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="coredns" exe="/coredns" subj=system_u:system_r:container_t:s0:c498,c537 key=(null)ARCH=x86_64 SYSCALL=openat AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

type=AVC msg=audit(1650652307.883:440): avc: denied { read } for pid=12719 comm="helm_v3" name="token" dev="tmpfs" ino=83158 scontext=system_u:system_r:container_t:s0:c255,c655 tcontext=system_u:object_r:container_runtime_tmpfs_t:s0 tclass=lnk_file permissive=0
type=SYSCALL msg=audit(1650652307.883:440): arch=c000003e syscall=262 success=no exit=-13 a0=ffffffffffffff9c a1=c0004b03c0 a2=c0007677c8 a3=0 items=0 ppid=12718 pid=12719 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm="helm_v3" exe="/usr/bin/helm_v3" subj=system_u:system_r:container_t:s0:c255,c655 key=(null)ARCH=x86_64 SYSCALL=newfstatat AUID="unset" UID="sysadm" GID="sysadm" EUID="sysadm" SUID="sysadm" FSUID="sysadm" EGID="sysadm" SGID="sysadm" FSGID="sysadm"

[brandond]

For the sake of experimentation, I installed vanilla containerd:

[root@rock01 ~]# rpm -ivh https://download.docker.com/linux/centos/8/x86_64/stable/Packages/containerd.io-1.5.11-3.1.el8.x86_64.rpm

[root@rock01 ~]# rpm -qa | grep -E 'containerd|container-selinux'
containerd.io-1.5.11-3.1.el8.x86_64
container-selinux-2.173.0-1.module+el8.5.0+735+2f243138.noarch

[root@rock01 ~]# kubectl get node -o wide
NAME               STATUS   ROLES                  AGE   VERSION            INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION                 CONTAINER-RUNTIME
rock01.lan.khaus   Ready    control-plane,master   19m   v1.23.6-rc1+k3s1   10.0.1.156    <none>        Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   containerd://1.5.11

[root@rock01 ~]# kubectl get node -o yaml | grep node-arg
      k3s.io/node-args: '["server","--selinux","--container-runtime-endpoint","/var/run/containerd/containerd.sock"]'

[root@rock01 ~]# cat /etc/containerd/config.toml
#   Copyright 2018-2022 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

#disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.cri.cni]
  bin_dir = "/var/lib/rancher/k3s/data/current/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.cri.containerd.runtimes.runc.options]
	SystemdCgroup = false

Everything is labeled as it was with 1.5.10:

/var/lib/kubelet/pods/a9ec72f4-5ca5-4224-bdbe-c989528e745e/volumes/kubernetes.io~projected/kube-api-access-xsf4k/:
total 0
drwxrwxrwt. 3 root root system_u:object_r:container_file_t:s0:c61,c394 140 Apr 22 18:32 .
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0           35 Apr 22 18:32 ..
drwxr-xr-x. 2 root root system_u:object_r:container_file_t:s0:c61,c394 100 Apr 22 18:32 ..2022_04_22_22_32_03.1704908283
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c61,c394  13 Apr 22 18:32 ca.crt -> ..data/ca.crt
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c61,c394  32 Apr 22 18:32 ..data -> ..2022_04_22_22_32_03.1704908283
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c61,c394  16 Apr 22 18:32 namespace -> ..data/namespace
lrwxrwxrwx. 1 root root system_u:object_r:container_file_t:s0:c61,c394  12 Apr 22 18:32 token -> ..data/token

[root@rock01 ~]# ps -ZC coredns
LABEL                               PID TTY          TIME CMD
system_u:system_r:container_t:s0:c61,c394 88334 ? 00:00:00 coredns

[brandond]

I confirmed that both vanilla containerd and k3s containerd are running with the correct contexts and are setting the correct contexts on the files:

[root@rock01 ~]# ps -ZC containerd
LABEL                               PID TTY          TIME CMD
system_u:system_r:container_runtime_t:s0 5306 ?  00:00:01 containerd

[root@rock01 ~]# ls -la /proc/5306/exe
lrwxrwxrwx. 1 root root 0 Apr 22 19:09 /proc/5306/exe -> /var/lib/rancher/k3s/data/02651a664540b008a96d3eaf113024fda362eb58f9b7ea4f86bd533ea2905f30/bin/containerd

I see k3s containerd setting the correct contexts on the files:

[root@rock01 ~]# grep token containerd3.log
5308  setxattr("/var/lib/kubelet/pods/ffbee2c9-7bcd-49b0-8982-e84fb5344148/volumes/kubernetes.io~projected/kube-api-access-zp5qg/..2022_04_22_23_29_50.1626641759/token", "security.selinux", "system_u:object_r:container_file"..., 47, 0 <unfinished ...>
5308  setxattr("/var/lib/kubelet/pods/ffbee2c9-7bcd-49b0-8982-e84fb5344148/volumes/kubernetes.io~projected/kube-api-access-zp5qg/token", "security.selinux", "system_u:object_r:container_file"..., 47, 0) = 0

Here is the vanilla containerd behavior; note that it stats the files before setting their selinux contexts

[root@rock01 ~]# grep token containerd.log
6618  newfstatat(AT_FDCWD, "/var/lib/kubelet/pods/7ac1a375-8211-4a6e-9f01-4f374db833ee/volumes/kubernetes.io~projected/kube-api-access-zq52m/..2022_04_22_23_18_56.2485026855/token",  <unfinished ...>
6618  newfstatat(AT_FDCWD, "/var/lib/kubelet/pods/7ac1a375-8211-4a6e-9f01-4f374db833ee/volumes/kubernetes.io~projected/kube-api-access-zq52m/token",  <unfinished ...>
6618  lsetxattr("/var/lib/kubelet/pods/7ac1a375-8211-4a6e-9f01-4f374db833ee/volumes/kubernetes.io~projected/kube-api-access-zq52m/token", "security.selinux", "system_u:object_r:container_file"..., 47, 0 <unfinished ...>
6618  lsetxattr("/var/lib/kubelet/pods/7ac1a375-8211-4a6e-9f01-4f374db833ee/volumes/kubernetes.io~projected/kube-api-access-zq52m/..2022_04_22_23_18_56.2485026855/token", "security.selinux", "system_u:object_r:container_file"..., 47, 0 <unfinished ...>

However the end result is the same, both processes set the correct container_file context on the symlink, which for some reason is getting incorrectly transitioned to container_runtime_tmpfs_t type by the new policy.

[brandond]

Root caused to opencontainers/selinux#172 - the issue was indeed with setting contexts on the symlinks vs symlink targets (setxattr vs stat+lsetxattr).

[rancher-max]

Validated using v1.23.6-rc3+k3s1

• Can successfully install on an selinux enforcing system, with k3s fully hardened and with the selinux flag
• Confirmed containerd version being used on both server and agent nodes: containerd://1.5.11-k3s2