conmon permission denied when running podman #1132

ljrk0 · 2022-03-18T12:57:05Z

Describe the bug
Very similar to #1089, running podman results in `"permission denied" by conmon.

Reproduction steps
Steps to reproduce the behavior:

If yo run an arbitrary container image you get:

$ podman run --rm docker.io/library/ubuntu
Error: fork/exec /usr/bin/conmon: permission denied

I also noticed that contrary to #1089 podman images does not produce an error.

System details

On QEMU-KVM
Fedora CoreOS version: 35.20220227.3.0

Additional information

Journal:

$ journalctl
Mar 18 09:54:15 earth systemd[6239]: Started podman-11855.scope.
Mar 18 09:54:15 earth podman[11855]: 
Mar 18 09:54:15 earth podman[11855]: 2022-03-18 09:54:15.867813985 +0000 UTC m=+0.108181471 container create 92a1a9871b7a7fb12f645ef5603ae823c8c0139002501fa3e4371a1c5ac7b74b (image=docker.io/library/ubuntu:latest, name=competent_dijkstra)
Mar 18 09:54:15 earth audit[11896]: AVC avc:  denied  { entrypoint } for  pid=11896 comm="podman" path="/usr/bin/conmon" dev="sda4" ino=2950748447 scontext=unconfined_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0 trawcon="system_u:object_r:conmon_exec_t:s0"
Mar 18 09:54:15 earth audit[11896]: SYSCALL arch=c000003e syscall=59 success=no exit=-13 a0=c0004d8a40 a1=c000604500 a2=c000391600 a3=8 items=0 ppid=11855 pid=11896 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts3 ses=11 comm="podman" exe="/usr/bin/podman" subj=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 key=(null)
Mar 18 09:54:15 earth kernel: audit: type=1400 audit(1647597255.878:2354): avc:  denied  { entrypoint } for  pid=11896 comm="podman" path="/usr/bin/conmon" dev="sda4" ino=2950748447 scontext=unconfined_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0 trawcon="system_u:object_r:conmon_exec_t:s0"
Mar 18 09:54:15 earth kernel: audit: type=1300 audit(1647597255.878:2354): arch=c000003e syscall=59 success=no exit=-13 a0=c0004d8a40 a1=c000604500 a2=c000391600 a3=8 items=0 ppid=11855 pid=11896 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts3 ses=11 comm="podman" exe="/usr/bin/podman" subj=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 key=(null)
Mar 18 09:54:15 earth podman[11855]: 2022-03-18 09:54:15.821572685 +0000 UTC m=+0.061940171 image pull  docker.io/library/ubuntu
Mar 18 09:54:15 earth audit: PROCTITLE proctitle=706F646D616E0072756E002D2D726D00646F636B65722E696F2F6C6962726172792F7562756E7475
Mar 18 09:54:15 earth kernel: audit: type=1327 audit(1647597255.878:2354): proctitle=706F646D616E0072756E002D2D726D00646F636B65722E696F2F6C6962726172792F7562756E7475
Mar 18 09:54:15 earth podman[11855]: 2022-03-18 09:54:15.919934528 +0000 UTC m=+0.160302024 container remove 92a1a9871b7a7fb12f645ef5603ae823c8c0139002501fa3e4371a1c5ac7b74b (image=docker.io/library/ubuntu:latest, name=competent_dijkstra)

Confirmed this can be "fixed" by rolling back to the previous release from the first of March (35.20220213.3.0):

$ rpm-ostree rollback -r

The text was updated successfully, but these errors were encountered:

dustymabe · 2022-03-18T14:18:30Z

Can you give us any more details about your system and how it was set up?

Can you also show the output of this command?

sudo ostree admin config-diff | grep selinux/targeted/policy

ljrk0 · 2022-03-18T15:06:59Z

Sure, the system is quite "vanilla" as it's pretty much just a container host with the following butane file:

variant: fcos
version: 1.3.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 ...
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: earth
    - path: /etc/NetworkManager/system-connections/Wired Connection 1.nmconnection
      mode: 0600
      contents:
        inline: |
          [connection]
          id=Wired connection 1
          type=ethernet
          interface-name=ens3

          [ipv4]
          method=auto

          [ipv6]
          addr-gen-mode=stable-privacy
          address1=...
          method=auto

It was installed using the following embedded butane/ignition config which chainloaded the above in the installation process:

variant: fcos
version: 1.3.0
systemd:
  units:
    - name: install.service
      enabled: true
      contents: |
        [Unit]
        Description=Run CoreOS Installer
        Requires=coreos-installer-pre.target
        After=coreos-installer-pre.target
        OnFailure=emergency.target
        OnFailureJobMode=replace-irreversibly

        [Service]
        Type=oneshot
        ExecStart=/usr/bin/coreos-installer install -i /var/setup.ign /dev/sda
        ExecStart=/usr/bin/systemctl --no-block reboot
        StandardOutput=kmsg+console
        StandardError=kmsg+console

        [Install]
        RequiredBy=default.target
storage:
  files:
    - path: /var/setup.ign
      mode: 0644
      contents:
        compression: null
        local: example.ign

The following however seems to indicate a modified selinux policy? This is however on the rollbacked (i.e., working version). There's a --os switch to specify the operating system, but I've yet to figure out what to put as OSNAME for it to work:

$ sudo ostree admin config-diff | grep selinux/targeted/policy
M    selinux/targeted/policy/policy.33

Nor did I manage to actually get some more specific diffs of the selinux policy. These are the current deployments:

$ sudo ostree admin status
* fedora-coreos abae17787e4534bc0fb2b87caad036ada07fae77d05df432534a6e21cd40a1e6.0
    Version: 35.20220213.3.0
    origin: <unknown origin type>
  fedora-coreos a095aba2b64d064a0c6a5ba5662f81d432852c2e2bfdf04139172b81b5e1ad86.0 (rollback)
    Version: 35.20220227.3.0
    origin: <unknown origin type>

These are packages I layered after the butane/ignition default install:

$ sudo rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Fri 2022-03-18 15:02:31 UTC)
Deployments:
* fedora:fedora/x86_64/coreos/stable
                   Version: 35.20220213.3.0 (2022-03-01T01:45:12Z)
                BaseCommit: 16afaa9af694537aa00cb3e259eeca549908696dbc8d7d548f1f49cf2d0f693e
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
           LayeredPackages: docker-compose

  fedora:fedora/x86_64/coreos/stable
                   Version: 35.20220227.3.0 (2022-03-14T19:06:29Z)
                BaseCommit: 60d938261803ef4f9d927e774db51a96c75073eff81744ee5f528ab33f985ecd
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
           LayeredPackages: docker-compose flatpak

I don't recall changing anything in /etc besides net.ipv4.ip_unprivileged_port_start=80.

dustymabe · 2022-03-18T16:08:00Z

Yes odd. If you didn't explicitly change the SELinux policy then maybe something else did? Either way you can get back to the defaults using the workaround at the bottom of #701 (comment) and then re-attempt your upgrade. - you'd want to verify before and after that the config-diff shows no modification to you policy.

ljrk0 · 2022-03-18T16:28:59Z

Hm, the Flatpak package definitely did but nothing too weird, these are diffs of the non-binary files:

# diff -a /usr/etc/selinux/targeted/active/commit_num /etc/selinux/targeted/active/commit_num
1c1
< 8
\ No newline at end of file
---
> 9
\ No newline at end of file
# diff -a /usr/etc/selinux/targeted/active/file_contexts /etc/selinux/targeted/active/file_contexts
6256a6257
> /usr/libexec/flatpak-system-helper	--	system_u:object_r:flatpak_helper_exec_t:s0

however this is odd in itself, since the currently active deployment doesn't layer flatpak...

I've now tried to search in vain -- how do I "unrollback", i.e., boot to the newer version in order to test the workaround?

dustymabe · 2022-03-18T17:19:50Z

hmm. If I install flatpak and docker-compose on 35.20220213.3.0 I don't see any "modified" selinux-policy.

The best option I can think of for now retrying the upgrade is:

clean up the existing 2nd deployment - then let zincati upgrade you
- sudo rpm-ostree cleanup --rollback
manually run the upgrade
- sudo systemctl stop zincati.service && sudo rpm-ostree upgrade

ljrk0 · 2022-03-18T17:56:27Z

Hm, I've now simply removed all layered packages & upgraded to the latest version, still the same issue (but I didn't revert the selinux policies yet). Interestingly, the file_contexts diff has now become larger instead of smaller, after removing the layered packages. And they now contain references to conmon...

# diff -a /usr/etc/selinux/targeted/active/file_contexts /etc/selinux/targeted/active/file_contexts
1385a1386
> /var/lib/kublet(/.*)?	system_u:object_r:container_var_lib_t:s0
1619d1619
< /var/lib/kubelet(/.*)?	system_u:object_r:container_var_lib_t:s0
3973d3972
< /usr/bin/conmon	--	system_u:object_r:conmon_exec_t:s0
5040d5038
< /usr/local/bin/conmon	--	system_u:object_r:conmon_exec_t:s0
6258a6257
> /usr/libexec/flatpak-system-helper	--	system_u:object_r:flatpak_helper_exec_t:s0

After that I reverted the SE Linux policies as advised by you, rebooted, and indeed now everything seems to work again!

Thank you very much for your help and pointers, they were invaluable! For me the issue is fixed but if you are interested in tracking down the root cause, I'll try to help as best I can. Thanks a bunch!

dustymabe · 2022-03-18T18:19:12Z

Thanks for letting us know! I'll close this one out and look out if more people report the same.

sedlund · 2022-03-30T02:22:48Z

@dustymabe do you think this poseidon/typhoon#1142 is related?

dustymabe · 2022-03-30T19:35:36Z

@dustymabe do you think this poseidon/typhoon#1142 is related?

Maybe more closer to #1138 ?

heyakyra · 2022-04-08T16:42:38Z

I'm on Silverblue 36 and can't enter toolbox containers due to:

audit[10207]: AVC avc:  denied  { entrypoint } for  pid=10207 comm="podman" path="/usr/bin/conmon" dev="dm-0" ino=1546509 scontext=unconfined_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0 trawcon="system_u:object_r:conmon_exec_t:s0"

Although on a separate laptop with the same version of everything, toolbox work fine, presumably because the working machine was set up from a fresh install and on the non-working one I get

$ sudo ostree admin config-diff | grep selinux/targeted/policy
M    selinux/targeted/policy/policy.33
A    selinux/targeted/policy/policy.32
A    selinux/targeted/policy/policy.31

while the working one only shows:

$ sudo ostree admin config-diff | grep selinux/targeted/policy
M    selinux/targeted/policy/policy.33

@dustymabe

you can get back to the defaults using the workaround at the bottom of #701 (comment)

Do you mean running sudo rsync -rclv /usr/etc/selinux/ /etc/selinux/? I'm confused how to clear out modifications to SELinux config

dustymabe · 2022-04-09T02:11:30Z

Do you mean running sudo rsync -rclv /usr/etc/selinux/ /etc/selinux/?

yes, and reboot

travier · 2022-04-13T14:45:44Z

Reference doc for SELinux policy update issues (applies to all rpm-ostree based OS): https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_selinux_problems

heyakyra · 2022-04-21T00:44:35Z

So, foolishly when the rsync command didn't help, I went ahead and just purged /etc/selinux which somehow did resolve the podman issue, but has put me in a bit of a quagmire as chronicled in the forums: https://discussion.fedoraproject.org/t/silverblue-rawhide-2019-02-05-selinux-labels-missing/1080/12?u=kxra

Also running the command from the docs leaves me unable to boot

$ sudo rm -rf /etc/selinux
$ sudo cp -aT /usr/etc/selinux /etc/selinux

ljrk0 added the kind/bug label Mar 18, 2022

dustymabe closed this as completed Mar 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

conmon permission denied when running podman #1132

conmon permission denied when running podman #1132

ljrk0 commented Mar 18, 2022

dustymabe commented Mar 18, 2022

ljrk0 commented Mar 18, 2022 •

edited

dustymabe commented Mar 18, 2022

ljrk0 commented Mar 18, 2022 •

edited

dustymabe commented Mar 18, 2022

ljrk0 commented Mar 18, 2022

dustymabe commented Mar 18, 2022

sedlund commented Mar 30, 2022

dustymabe commented Mar 30, 2022

heyakyra commented Apr 8, 2022

dustymabe commented Apr 9, 2022 •

edited

travier commented Apr 13, 2022

heyakyra commented Apr 21, 2022 •

edited

conmon permission denied when running podman #1132

conmon permission denied when running podman #1132

Comments

ljrk0 commented Mar 18, 2022

dustymabe commented Mar 18, 2022

ljrk0 commented Mar 18, 2022 • edited

dustymabe commented Mar 18, 2022

ljrk0 commented Mar 18, 2022 • edited

dustymabe commented Mar 18, 2022

ljrk0 commented Mar 18, 2022

dustymabe commented Mar 18, 2022

sedlund commented Mar 30, 2022

dustymabe commented Mar 30, 2022

heyakyra commented Apr 8, 2022

dustymabe commented Apr 9, 2022 • edited

travier commented Apr 13, 2022

heyakyra commented Apr 21, 2022 • edited

ljrk0 commented Mar 18, 2022 •

edited

ljrk0 commented Mar 18, 2022 •

edited

dustymabe commented Apr 9, 2022 •

edited

heyakyra commented Apr 21, 2022 •

edited