Podman 4.0.0+ fails to SELinux-relabel symlinks in volumes when using the ":Z" option, resulting in "permission denied" errors (Regression, bisected)

[joanbm]

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Description

When mounting a volume containing symlinks using Podman 4.0.0+ in a SELinux-enabled system (e.g. a clean Fedora 36 Workstation beta install), specifying the :Z flag in order to relabel the files in the volume, the symlink is not relabeled (but regular files are), resulting in "permission denied" errors.

Running the same commands in a similar system, but using Podman 3.4.4 (e.g. a clean Fedora 35 Workstation install), the problem does not reproduce.

Steps to reproduce the issue

1. Install Podman 4.0.0+ in a SELinux-enabled system, e.g. a clean Fedora 36 Workstation beta install.

2. Open a terminal and set up some test files:

$ mkdir stuff && cd stuff
$ echo hello > file
$ ln -s file link
$ ls -Z
unconfined_u:object_r:user_home_t:s0 file
unconfined_u:object_r:user_home_t:s0 link

3. Try to mount and relabel the files into a container using a volume:

$ podman run --rm -v .:/stuff:Z busybox ls -lah /stuff
ls: /stuff/link: Permission denied
total 4K     
drwxr-xr-x    1 root     root          16 Mar 30 20:38 .
dr-xr-xr-x    1 root     root          36 Mar 30 20:39 ..
-rw-r--r--    1 root     root           6 Mar 30 20:38 file

Describe the results you received:

I received a permission denied error when trying to access the "link" symlink but not for the "file" regular file.

Describe the results you expected:

I expected the "link" symlink to be accessible inside the container just like the "file" regular file.

Additional information you deem important (e.g. issue happens only occasionally):

The regular file is relabeled while the symlink is not relabeled:

$ ls -Z
system_u:object_r:container_file_t:s0:c177,c899 file
           unconfined_u:object_r:user_home_t:s0 link

Running with --security-opt label=disable resolves the issue, confirming it's related to SELinux:

$ podman run --rm -v .:/stuff:Z --security-opt label=disable busybox ls -lah /stuff
drwxr-xr-x    1 root     root          16 Mar 30 20:38 .
dr-xr-xr-x    1 root     root          36 Mar 30 20:39 ..
-rw-r--r--    1 root     root           6 Mar 30 20:38 file
lrwxrwxrwx    1 root     root           4 Mar 30 20:38 link -> file

strace shows that setxattr is being used over the symlink (note that setxattr does follow symlinks):

$ strace -f podman run --rm -v .:/stuff:Z busybox ls -lah /stuff 2>&1 | grep -w link | grep setxattr
[pid 111800] setxattr("/home/podmantest/stuff/link", "security.selinux", "system_u:object_r:container_file"..., 47, 0 <unfinished ...>

Compare the output above with Fedora 35 + Podman 3.4.4. The problem does not reproduce and lsetxattr is being used over the symlink:

$ mkdir stuff && cd stuff
$ echo hello > file
$ ln -s file link
$ ls -Z
unconfined_u:object_r:user_home_t:s0 file
unconfined_u:object_r:user_home_t:s0 link
$ podman run --rm -v .:/stuff:Z busybox ls -lah /stuff
drwxr-xr-x    1 root     root          16 Mar 30 20:40 .
dr-xr-xr-x    1 root     root          36 Mar 30 20:40 ..
-rw-r--r--    1 root     root           6 Mar 30 20:40 file
lrwxrwxrwx    1 root     root           4 Mar 30 20:40 link -> file
$ ls -Z
system_u:object_r:container_file_t:s0:c35,c116 file
system_u:object_r:container_file_t:s0:c35,c116 link
$ strace -f podman run --rm -v .:/stuff:Z busybox ls -lah /stuff 2>&1 | grep -w link | grep setxattr
[pid  2214] lsetxattr("/home/podmantest/stuff/link", "security.selinux", "system_u:object_r:container_file"..., 47, 0 <unfinished ...>

This change in behavior is caused by commit f564c9a (upstream opencontainers/selinux@1b18907), reverting it fixes the issue.

Note how the commit renames the original setFileLabel -> lSetFileLabel, and introduces a new setFileLabel function that does follows symlinks.

Output of podman version:

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.18beta2

Built:      Thu Mar  3 15:56:09 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: workstation
    version: "36"
  eventLogger: journald
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.17.1-300.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 7235469312
  memTotal: 8329519104
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8328835072
  swapTotal: 8328835072
  uptime: 25.05s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podmantest/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podmantest/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/podmantest/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319369
  BuiltTime: Thu Mar  3 15:56:09 2022
  GitCommit: ""
  GoVersion: go1.18beta2
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.0.2-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Reproduced on both local QEMU-KVM virtual machines and bare metal, running Fedora 35/36 Beta.

[Luap99]

@rhatdan PTAL

[rhatdan]

This is what the fix in opencontainers/selinux is all about, calling lsetfilecon versus setfilecon.

[rhatdan]

opencontainers/selinux#173

[joanbm]

Thanks, I didn't realize a fix was already available on the dependency. I tested applying #13689 and it does indeed fix the issue.

[joanbm]

4.0.2 + #13689 works as far as I tested, but 4.0.3 + #13689 fails to run RUN from a Dockerfile:

$ cat > Dockerfile
FROM busybox
RUN echo hello
$ podman build . -ttest
STEP 1/2: FROM busybox
STEP 2/2: RUN echo hello
error running container: setting file label on "/proc/self/fd/12": lsetxattr /proc/self/fd/12: operation not supported
ERRO[0005] did not get container create message from subprocess: read |0: i/o timeout 
Error: error building at STEP "RUN echo hello": error while running runtime: exit status 1

My guess is that containers/buildah#3630 started using label.Relabel on /proc/self/fd/123, internally label.Relabel calls selinux.Chcon which at the same merge started using setxattr instead of lsetxattr, however opencontainers/selinux#173 changed selinux.Chcon back to lsetxattr which breaks the first PR.

EDIT: Indeed I removed the call to runLabelStdioPipes from containers/buildah#3630 and podman build works again.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Should be fixed in podman 4.1

[ncdc]

@rhatdan I'm having problems with both :z and :Z with a Mac host:

$ podman run --rm -it -v $(pwd)/foo:/foo:z busybox
Error: error preparing container da0823e62fe20ff819ce7027cd3957a85fe721db8ea4098ced4c3ed64d11d4c4 for attach: lsetxattr /Users/angoldst/foo: operation not supported

$ podman run --rm -it -v $(pwd)/foo:/foo:Z busybox
Error: error preparing container 8719effbe6680f726e16e53a067c96236d48d9362709367822ee7566d1a5993a for attach: lsetxattr /Users/angoldst/foo: operation not supported

$ podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18.1
Built:        Thu May  5 16:07:47 2022
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18
Built:        Fri May  6 12:15:54 2022
OS/Arch:      linux/amd64

Not sure if this is the same issue/root cause, or a different one? Please let me know if you'd like me to file a new one. Thanks!

[joanbm]

@ncdc Looks like a different issue to me, the operation should not fail on 4.1.0 and for you it doesn't fail on /proc/fd/N but rather over the host folder you are trying to map.

Also, Mac doesn't support SELinux at all, does it? So you may be able to drop the z/Z flags altogether. Though if that's the case, maybe it'd be nice if Podman ignored them by itself so the same command could work on both Linux and Mac hosts.

[rhatdan]

Yes, the :Z and :z will not work on networked mounted volumes, which are used on a MAC. We are working on virtiofsd support for SELinux which could also work on a MAC, but this does not exists yet.

[alexanderadam]

I seem to have a similar issue on Podman 3.4.4 (which is the default on Ubuntu 22.04) but according to the issue description this shouldn't happen on 3.4.4.
Launching a GitLab container on Podman 3.4.4 fails however, because it tries to create a symlink at launch and it files while --security-opt label=disable doesn't help either.

PS: Or is there at least a possibility to properly install and maintain a current stable Podman version on Ubuntu 22.04?

[joanbm]

@alexanderadam Looks like a different issue, the one reported here was only on Podman 4.x.x and was related to SELinux, which Ubuntu doesn't use by default. Your issue looks more like a users/permissions issue.

You're probably not going to get much help in this GitHub tracker unless you can reproduce it on the latest version. No idea how it can be installed on Ubuntu 22.04 though.

FWIW I tried to run your image from the Ubuntu 22.04.1 Server LiveCD shell and it appears to get pretty far without the error you mention:

root@ubuntu-server:/# apt-get update -qq
root@ubuntu-server:/# apt-get install -y podman >/dev/null 2>/dev/null
root@ubuntu-server:/# useradd -m -s /bin/bash user
root@ubuntu-server:/# su user
user@ubuntu-server:/$ podman run --rm docker.io/gitlab/gitlab-ee:15.3.1-ee.0
Trying to pull docker.io/gitlab/gitlab-ee:15.3.1-ee.0...
Getting image source signatures
Copying blob d75fa255d79e done  
Copying blob a20088b9a96a done  
Copying blob 56693a6694ed done  
Copying blob 984b7bed62e0 done  
Copying blob 3b65ec22a9e9 done  
Copying blob c95a380c61e5 done  
Copying blob 66cd605c0ea3 done  
Copying blob 9771e1da108b done  
Copying config f11c5f0ccc done  
Writing manifest to image destination
Storing signatures
Thank you for using GitLab Docker Image!
Current version: gitlab-ee=15.3.1-ee.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab editor /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

Cleaning stale PIDs & sockets
cat: /var/opt/gitlab/gitlab-rails/VERSION: No such file or directory
Installing gitlab.rb config...
Generating ssh_host_rsa_key...
Generating public/private rsa key pair.
[...]
Preparing services...
Starting services...
Configuring GitLab...
/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 34: ulimit: max user processes: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
ffi-libarchive could not be loaded, libarchive is probably not installed on system, archive_file will not be available
[2022-08-30T19:19:07+00:00] INFO: Started Chef Infra Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
[...3800 lines omitted...]
        - create symlink at /opt/gitlab/service/postgresql to /opt/gitlab/sv/postgresql

So your issue may be related to how you're running podman or which permissions you have in your volume mounts if you have any, so try to include those in your reports.