Windows Pod with RunAsUserName and a Projected Volume does not honor file permissions in the volume

[aravindhp]

What happened:

When a Windows Pod is created with a Projected Volume and RunAsUserName set, file permissions are not handled in the projected volume. In addition if a Windows Pod is created with a Projected Volume and RunAsUser set,

    securityContext:
      runAsUser: 1000640000
      windowsOptions:
        runAsUserName: ContainerAdministrator

the Pod will be stuck at ContainerCreating:

E0611 17:38:46.666139    2260 atomic_writer.go:404] pod win-webserver/win-webserver-6dd6cd7d9-ng62b volume kube-api-access-62z59: unable to change file c:\var\lib\kubelet\pods\a7bddda2-a355-48cd-8ee2-71776d6de78f\volumes\kubernetes.io~projected\kube-api-access-62z59\..2021_06_11_17_38_46.035861910\token with owner 1000640000: chown c:\var\lib\kubelet\pods\a7bddda2-a355-48cd-8ee2-71776d6de78f\volumes\kubernetes.io~projected\kube-api-access-62z59\..2021_06_11_17_38_46.035861910\token: not supported by windows
E0611 17:38:46.666139    2260 atomic_writer.go:175] pod win-webserver/win-webserver-6dd6cd7d9-ng62b volume kube-api-access-62z59: error writing payload to ts data directory c:\var\lib\kubelet\pods\a7bddda2-a355-48cd-8ee2-71776d6de78f\volumes\kubernetes.io~projected\kube-api-access-62z59\..2021_06_11_17_38_46.035861910: chown c:\var\lib\kubelet\pods\a7bddda2-a355-48cd-8ee2-71776d6de78f\volumes\kubernetes.io~projected\kube-api-access-62z59\..2021_06_11_17_38_46.035861910\token: not supported by windows

What you expected to happen:

The Pod should go to Running with the project volume files having the correct container user ownership.

How to reproduce it (as minimally and precisely as possible):

Create a Windows Pod with a Projected Volume and runAsUserName set (optionally with runAsUser to cause the stuck at ContainerCreating issue).

Anything else we need to know?:

The KEP, proposal for file permission handling in projected service account volume and its implementation did not take Windows in to account. An os.Chown() was introduced which is not supported on Windows. In addition RunAsUserName being set on Windows Pods was not taken into account.

Environment:

• Kubernetes version (use kubectl version): 1.21.1
• Cloud provider or hardware configuration: AWS / Azure
• OS: Windows Server 2019 1809
• Kernel:

[aravindhp]

/sig windows

[aravindhp]

/assign

[jsturtevant]

/triage accepted
/priority important-soon

[aravindhp]

/retitle Windows Pod with RunAsUserName and a Projected Volume does not honor file permissions in the volume

[jsturtevant]

/milestone v1.22

[aravindhp]

Please follow sig-windows slack thread for current state.

[aravindhp]

Currently investigating what is the current state of owner ship across users in different pods and containers. Created a simple ping container image with two users:

FROM mcr.microsoft.com/windows/servercore:ltsc2019
RUN net user TestUser1 /add
RUN net user TestUser2 /add
CMD ["ping", "-t", "localhost"]

Created the following pod:

apiVersion: v1
kind: Pod
metadata:
  name: win-ping
spec:
  tolerations:
  - key: "os"
    value: "Windows"
    Effect: "NoSchedule"
  containers:
  - name: ping1
    image: quay.io/aravindh/win-ping
    imagePullPolicy: Always
    securityContext:
      windowsOptions:
        runAsUserName: TestUser1
    volumeMounts:
    - name: for-ping1
      mountPath: /projected-volume
    - name: c-volume
      mountPath: /from-host
  - name: ping2
    image: quay.io/aravindh/win-ping
    imagePullPolicy: Always
    securityContext:
      windowsOptions:
        runAsUserName: TestUser2
    volumeMounts:
    - name: for-ping2
      mountPath: /projected-volume
    - name: c-volume
      mountPath: /from-host
  nodeSelector:
    kubernetes.io/os: windows
  volumes:
  - name: for-ping1
    projected:
      sources:
      - secret:
          name: user
  - name: for-ping2
    projected:
      sources:
      - secret:
          name: pass
  - name: c-volume
    hostPath:
      path: /

I then execed into container ping1:

 kubectl exec win-ping -it powershell.exe -c ping1

and ran the following:

PS C:\> whoami
win-ping\testuser1
PS C:\> cat C:\from-host\var\lib\kubelet\pods\d10783ef-919d-46bf-ab38-2b951f063632\volumes\kubernetes.io~projected\for-ping2\password.txt 
1f2d1e2e67df 

So testuser1 is able to read a file projected for testuser2

[aravindhp]

I repeated the experiment with two pods running on the same host:

apiVersion: v1
kind: Pod
metadata:
  name: win-ping1
spec:
  tolerations:
  - key: "os"
    value: "Windows"
    Effect: "NoSchedule"
  containers:
  - name: ping1
    image: quay.io/aravindh/win-ping
    imagePullPolicy: Always
    securityContext:
      windowsOptions:
        runAsUserName: TestUser1
    volumeMounts:
    - name: for-ping1
      mountPath: /projected-volume
    - name: c-volume
      mountPath: /from-host
  nodeSelector:
    kubernetes.io/os: windows
  volumes:
  - name: for-ping1
    projected:
      sources:
      - secret:
          name: user
  - name: c-volume
    hostPath:
      path: /
---
apiVersion: v1
kind: Pod
metadata:
  name: win-ping2
spec:
  tolerations:
  - key: "os"
    value: "Windows"
    Effect: "NoSchedule"
  containers:
  - name: ping2
    image: quay.io/aravindh/win-ping
    imagePullPolicy: Always
    securityContext:
      windowsOptions:
        runAsUserName: TestUser2
    volumeMounts:
    - name: for-ping2
      mountPath: /projected-volume
    - name: c-volume
      mountPath: /from-host
  nodeSelector:
    kubernetes.io/os: windows
  volumes:
  - name: for-ping2
    projected:
      sources:
      - secret:
          name: pass
  - name: c-volume
    hostPath:
      path: /

I then execed into win-ping1 pod:

 kubectl exec win-ping1 -it powershell.exe

and ran the following:

PS C:\> whoami
win-ping1\testuser1
PS C:\> cat C:\from-host\var\lib\kubelet\pods\b701defa-6138-4375-a6ec-f0c155973348\volumes\kubernetes.io~projected\for-ping2\password.txt 
1f2d1e2e67df

Again, testuser1 is able to read a file projected for testuser2

[voigt]

Hey there (@immuzz, @jsturtevant, @aravindhp), Bug-Triage here 👋 ,
Given the fact that the code freeze for 1.22 already started I'd suggest moving this issue to 1.23. wdyt?

[marosset]

@voigt - We'll move it to 1.23 thanks!

[marosset]

/milestone v1.23

[jyotimahapatra]

/milestone v1.25

[markjacksonfishing]

Hi @aravindhp . My name is Marky Jackson and I am one of the k8s 1.25 bug triage shadow assigned to track this body of work. It looks like all PR’s have been merged. Just checking in to see if this is still on track for k8s 1.25?

[jsturtevant]

We don't have a clear long-term solution to issue. There has been some discussions to fix this at the OS layer but there are no timelines.
/milestone clear

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[marosset]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[marosset]

/lifecycle frozen

This is a valid issue but requires OS level changes (which have been discussed in the past) to address.

[k8s-triage-robot]

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Deprioritize it with /priority important-longterm or /priority backlog
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[MageshSrinivasulu]

Any Updates? Is this issue fixed?

Facing a similar issue where the consul agent daemon set supposed to run on every node is failing on the Windows 2022 node since the security context is having runAsGroup runAsUser fsGroup

[jsturtevant]

Any Updates? Is this issue fixed?

No updates. It requires a significant change in the way the Windows Containers handles permissions and is under consideration but isn't something that we will be address in the short term.

Facing a similar issue where the consul agent daemon set supposed to run on every node is failing on the Windows 2022 node since the security context is having runAsGroup runAsUser fsGroup

You can checkout the PodOS field as this was designed to help with different fields that aren't relevant to the OS. Running the same DaemonSet configuration across nodes is something that sounds good as it reduces duplication but in practice we've found that it's best to have two separate DS specific for Windows or Linux since the configuration usually ends up being significantly different.