error running newuidmap: "invalid internal status"

[markstos]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Unable to run a container with podman 4.0.1 on Arch Linux

Steps to reproduce the issue:

1. podman run -ti --rm --network=host docker.io/mongo:4.4 mongo

Describe the results you received:

ERRO[0000] error running `/usr/bin/newuidmap 291662 0 1000 1 1 10000 65536`:  
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using "/usr/bin/newuidmap": exit status 1 

Running podman system migrate does not fix it. I got a different error before I ran podman system migrate, now lost.

Describe the results you expected:

Expected it to work.

Additional information you deem important (e.g. issue happens only occasionally):

Both /etc/subuid and /etc/subgid contain mark:10000:65536 and are symlinked to a file in my home directory.

Here's what happens if I try to run the newuid command directly:

❯ /usr/bin/newuidmap 291829 0 1000 1 1 10000 65536
newuidmap: Could not open proc directory for target 291829

Output of podman version:

podman --version reports 4.0.1, but podman version fails with the same "invalid internal status" error as above.

(paste your output here)

Output of podman info --debug:

The command can't run. It fails the same as above.

Package info (e.g. output of rpm -q podman or apt list podman):

❯ paru -Qi podman
Name            : podman
Version         : 4.0.1-2
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : conmon  containers-common  crun  iptables  libdevmapper.so=1.02-64  libgpgme.so=11-64
                  libseccomp.so=2-64  slirp4netns
Optional Deps   : apparmor: for AppArmor support
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  netavark: for a new container-network-stack implementation
                  podman-compose: for docker-compose compatibility
                  podman-docker: for Docker-compatible CLI
Required By     : ansible-bender
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 79.67 MiB
Packager        : David Runge <dvzrv@archlinux.org>
Build Date      : Sun 13 Mar 2022 07:44:23 AM EDT
Install Date    : Sun 13 Mar 2022 04:37:18 PM EDT
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[giuseppe]

could you try running manually these commands?

$ unshare -U sleep 100 &
$ /usr/bin/newuidmap $! 0 1000 1 1 10000 65536

Does the newuidmap call succeed?

[markstos]

It failed. So I'm guessing this is not a podman problem? If not, it would be helpful if podman's diagnostic message could provide a little more direction on how to possibly resolve the issue.

I'd really love to use rootless containers with Podman on Arch Linux.

$ unshare -U sleep 100 &
[1] 381035
$ /usr/bin/newuidmap $! 0 1000 1 1 10000 65536 || echo "failed: $?"
failed: 1

After I run unshare, I checked the uid_map file for the process. The file exists, but it is empty:

cat /proc/381035/uid_map 

[rhatdan]

I would figure either newuidmap is not setuid or does not have the filecap setuid applied to it.
Also is there entries in /etc/subuid for this user.

[markstos]

@rhatdan You were right that Arch Linux didn't have the setuid bit set on newuidmap. It seems it's been that been way for 5+ years so is probably not considered a "bug" ( https://github.com/lxc/lxd/issues/3397#issuecomment-307632741 ).

It would be helpful if Podman could check for the setuid bit and warn about that problem if it's not set.

But setting the setuid bit still didn't didn't fix it. As seen below, it looks like the subuid and subgid permissions are set correctly.

mark$ unshare -U sleep 100 &
[1] 392795

mark$ /usr/bin/newuidmap $! 0 1000 1 1 10000 65536 || echo "failed: $?"
failed: 1

mark$ ls -l /bin/newuidmap 
-rwsr-sr-x 1 root libvirt 37528 Jan 27 08:47 /bin/newuidmap

mark$ groups
libvirt docker video input audio wheel mark

mark$ cat /etc/subuid 
mark:10000:65536

mark$ cat /etc/subgid 
mark:10000:65536

mark$ grep mark /etc/passwd
mark:x:1000:1000::/home/mark:/bin/fish

mark$ ls -lthd /etc/subuid /etc/subgid
lrwxrwxrwx 1 root root 28 Apr  1 10:17 /etc/subgid -> /home/mark/.local/etc/subgid
lrwxrwxrwx 1 root root 28 Apr  1 10:17 /etc/subuid -> /home/mark/.local/etc/subuid

(The "group" of newuid was originally set to "root", but after a test failed using that, I tried setting the group to "libvirt" to see if that helped. It didn't).

[markstos]

@rhatdan I presume you closed this because the permissions of newuidmap are not a Podman bug. I get that. That's fair.

Consider this a "feature request" for Podman to detect this case and exit gracefully with an error message that newuidmap doesn't have the required permissions.

I've tried unsuccessfully to get going with Podman multiple times due to rough edges that I'm sure are due in part to running it outside of Fedora. Smoothing out these rough edges with better diagnostics can help grow the user base and in turn the pool of contributors.

[rhatdan]

Opened a PR containers/storage#1188 to check on failure of newuidmap and newgidmap.

[markstos]

Thank you @rhatdan !

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since this is in podman 4.1, closing.