Pods with user namespaces don't start

[gnufied]

What happened?

I tried to create a simple pod with user namespace feature on Fedora 36 and running crio-1.25 from testing repos and got:

  Normal   Scheduled               3m1s              default-scheduler  Successfully assigned default/sandbox to 127.0.0.1
  Warning  FailedCreatePodSandBox  3m1s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(49c15575d72965948516678688fce478b8af3942011b55bf357cd7a9c752e838): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/3f113d34b8e0519a97a9e1cb457e0d1ef683f933086b0a07f42bf15b1c7af702/mapped/0": invalid argument
  Warning  FailedCreatePodSandBox  2m47s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(da907684b2de6e99cba4d26df0b33044f54fdb68f94336b10cee8847088aa424): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/72e45dd8e0a32debaba4cb66cd3ebf01b2a05333e4d8b1d91ece86fd9d36e051/mapped/0": invalid argument
  Warning  FailedCreatePodSandBox  2m34s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(d371e43a99f5ca111317d0e916fe0d7cb81201c6d5ac7eacfffe81dc45c9f2ee): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/2a68f19376117d4e00ae2da4acc6e7663207c40e9f8f4d20459099bd538f1509/mapped/0": invalid argument
  Warning  FailedCreatePodSandBox  2m19s             kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(82a104d7d59973d42dfab5303ce67122b86890bdcfe4e91001335a61b3bd8acf): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/f801685ab0822b8c754e27ef81ef50a06becf37d39f3e08e8932385c90ec77a4/mapped/0": invalid argument
  Warning  FailedCreatePodSandBox  2m6s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(e9b613dcddf91e754d98124ca58d2408dbc7edb2b2d69f3d57840fffde65fa56): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/fc256d588e615bd86067c942bd064fbb427636ff83d5407c9ff316d0f1739ff0/mapped/0": invalid argument
  Warning  FailedCreatePodSandBox  114s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(f49f7504a2eadd3631ad1447cf0d667e9559c476f8bc3c2ca83b96eb09692c1f): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/1d48d5fe4e3f3ab97d3ca83ae64ccffb44d94b21bec4a0823c332f5c07bf1724/mapped/0": invalid argument
  Warning  FailedCreatePodSandBox  100s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(5b36c032904e8114ef52e6c7c731bbb6bdf45cb19fd52886afb7f5195c9fb3ee): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/73bb536b2696345f7176bd4bb87dee76ca534841c23ccf6be3de217238ff9081/mapped/0": invalid argument
  Warning  FailedCreatePodSandBox  87s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(76555be81a380928e66d83f18da43cde1f88015b027746d62e202c2903baa501): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/bb88c047a9ef629419400a9044163aa7f4a69d6cee92910ad4c08c41c39416b8/mapped/0": invalid argument            
  Warning  FailedCreatePodSandBox  73s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(c6879f26203e245e508da5833a6aac8eb8349c903215a39a1900dd753863342a): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/ce2c8b78ca0c181eb56bb0417bbb9adcbdb6360ad2eab309473102a04c65e4b5/mapped/0": invalid argument            
  Warning  FailedCreatePodSandBox  6s (x5 over 59s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(3bf6850d19ab4c058d6d9628879b5cf503f9458da5bb2e834c97ed56e43cb087): create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/725c24043939a8565ae411d6c8e7a555e3fc3135a518ea095f3c5988b40a3359/mapped/0": invalid argument

I could confirm that dropping hostUsers: false enabled pod to start correctly...

What did you expect to happen?

Following Pod should have started:

apiVersion: "v1"
kind: "Pod"
metadata:
  name: "sandbox"
  labels:
    name: "sandbox"
spec:
  hostUsers: false
  containers:
    -
      name: "sandbox"
      image: gcr.io/google_containers/busybox
      command:
        - "/bin/sh"
        - "-c"
        - "while true; do date; echo `date`; sleep 5; done"
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
    runAsUser: 4000
    runAsGroup: 4000

How can we reproduce it (as minimally and precisely as possible)?

Create above pod in a k8s cluster

Anything else we need to know?

Log from kubelet:

I1118 20:27:50.522495   97385 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="default/sandbox"
I1118 20:27:50.522532   97385 kuberuntime_manager.go:681] "computePodActions got for pod" podActions={KillPod:true CreateSandbox:true SandboxID: Attempt:0 NextInitContainerToStart:nil ContainersToStart:[0] ContainersToKill:map[] EphemeralContainersToStart:[]} pod="default/sandbox"
I1118 20:27:50.696777   97385 projected.go:189] Setting up volume kube-api-access-pdff5 for pod 8fd48fa7-782c-41de-9efe-7e0b9e1137bd at /var/lib/kubelet/pods/8fd48fa7-782c-41de-9efe-7e0b9e1137bd/volumes/kubernetes.io~projected/kube-api-access-pdff5
E1118 20:27:51.224516   97385 remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(18c87684aeb7ac3b274e7d226772420b3b5a11feb78a7601966d57446ce77cbc): create mapped mount for \"/var/lib/containers/storage/overlay\" on \"/var/lib/containers/storage/overlay/e33121943e106743e7de8dc23f489ca5b2dbbb03fd3a6fd47ed39fbefcc0729f/mapped/0\": invalid argument"
E1118 20:27:51.224606   97385 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(18c87684aeb7ac3b274e7d226772420b3b5a11feb78a7601966d57446ce77cbc): create mapped mount for \"/var/lib/containers/storage/overlay\" on \"/var/lib/containers/storage/overlay/e33121943e106743e7de8dc23f489ca5b2dbbb03fd3a6fd47ed39fbefcc0729f/mapped/0\": invalid argument" pod="default/sandbox"
E1118 20:27:51.224660   97385 kuberuntime_manager.go:782] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(18c87684aeb7ac3b274e7d226772420b3b5a11feb78a7601966d57446ce77cbc): create mapped mount for \"/var/lib/containers/storage/overlay\" on \"/var/lib/containers/storage/overlay/e33121943e106743e7de8dc23f489ca5b2dbbb03fd3a6fd47ed39fbefcc0729f/mapped/0\": invalid argument" pod="default/sandbox"
E1118 20:27:51.224763   97385 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"sandbox_default(8fd48fa7-782c-41de-9efe-7e0b9e1137bd)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"sandbox_default(8fd48fa7-782c-41de-9efe-7e0b9e1137bd)\\\": rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(18c87684aeb7ac3b274e7d226772420b3b5a11feb78a7601966d57446ce77cbc): create mapped mount for \\\"/var/lib/containers/storage/overlay\\\" on \\\"/var/lib/containers/storage/overlay/e33121943e106743e7de8dc23f489ca5b2dbbb03fd3a6fd47ed39fbefcc0729f/mapped/0\\\": invalid argument\"" pod="default/sandbox" podUID=8fd48fa7-782c-41de-9efe-7e0b9e1137bd
I1118 20:27:51.224801   97385 event.go:294] "Event occurred" object="default/sandbox" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="FailedCreatePodSandBox" message="Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0 in pod sandbox k8s_sandbox_default_8fd48fa7-782c-41de-9efe-7e0b9e1137bd_0(18c87684aeb7ac3b274e7d226772420b3b5a11feb78a7601966d57446ce77cbc): create mapped mount for \"/var/lib/containers/storage/overlay\" on \"/var/lib/containers/storage/overlay/e33121943e106743e7de8dc23f489ca5b2dbbb03fd3a6fd47ed39fbefcc0729f/mapped/0\": invalid argument"

Kubernetes version

latest master

Cloud provider

None

OS version

Fedora 36

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@gnufied: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gnufied]

/sig node

[gnufied]

cc @rata @giuseppe @msau42

[rata]

To sync this issue with the slack thread: https://kubernetes.slack.com/archives/C0BP8PW9G/p1668803413275189, posting what we discussed there: this seems CRIO specific, so I'd let @giuseppe handle it. It seems the fix was merged but not released in CRIO yet.

[gnufied]

Is there a distro/environment where this feature works out of box after enabling the alpha feature gate? If not - how are we going to e2e this thing?

[rata]

@gnufied I guess CRIO will work out of the box, there is just a bug fix pending.

For containerd, we expect to merge the patches for the upcoming 1.7 release at the end of this year. For e2e tests, we have already wrote some of them (merged them in kubernetes too), but as with any change that requires changes in the container runtime and the kubelet, we will be careful and making sure we feel comfortable with what is tested on CI when migrating the feature.

[gnufied]

What is the CRIO bug you are referring to? Can we get a link.

I was evaluating this from call we had in sig-storage meeting. I will keep this issue open and keep an eye on it, so as we can test the various point of integration.

[rata]

Lets unify the conversation here or in slack (I say let's do it on slack), but having it duplicated creates these issues :)

@gnufied this issue @giuseppe mentioned you on slack: https://kubernetes.slack.com/archives/C0BP8PW9G/p1668803514789069?thread_ts=1668803413.275189&cid=C0BP8PW9G

[giuseppe]

this is the issue, that is already fixed in containers/storage, and only needs to be vendored in CRI-O: containers/storage#1308

[SergeyKanzhelev]

/close

following the conversation it looks like the issue is not in kubernetes. So closing this bug here, please file a bug with CRI-O.

[k8s-ci-robot]

@SergeyKanzhelev: Closing this issue.

In response to this:

/close

following the conversation it looks like the issue is not in kubernetes. So closing this bug here, please file a bug with CRI-O.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.