APT fails on Debian-based rootful containers with --userns=auto

[jonasdemoor]

Environment

• Debian 11 "bullseye"

# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

• SubID mappings

# more /etc/sub{uid,gid}
::::::::::::::
/etc/subuid
::::::::::::::
containers:2147483648:2147483646
::::::::::::::
/etc/subgid
::::::::::::::
containers:2147483648:2147483646

UID ranges taken from: https://github.com/systemd/systemd/blob/main/docs/UIDS-GIDS.md#summary

• Podman v4.6.0 (compiled from source on Debian 11)

# podman info
host:
  arch: amd64
  buildahVersion: 1.31.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: ugent-conmon_2.1.7-60_amd64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.7, commit: unknown'
  cpuUtilization:
    idlePercent: 98.54
    systemPercent: 0.77
    userPercent: 0.69
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: bullseye
    distribution: debian
    version: "11"
  eventLogger: journald
  freeLocks: 2045
  hostname: <redacted>
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.0-23-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 6525829120
  memTotal: 8330928128
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: ugent-aardvark-dns_1.7.0-60_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: ugent-netavark_1.7.0-60_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: ugent-crun_1.8.6-60_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 536866816
  swapTotal: 536866816
  uptime: 1h 17m 58.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 3059744768
  graphRootUsed: 1135034368
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.6.0
  Built: 1690873736
  BuiltTime: Tue Aug  1 09:08:56 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.0

Description

When trying to make use of Podman's automatic user namespace mapping with --userns=auto, APT fails to run inside the container. I'm starting containers as the root user.

# podman run --userns auto -it docker.io/library/debian:bullseye-slim bash
root@397626021d9a:/# apt update
E: setgroups 65534 failed - setgroups (22: Invalid argument)
E: setegid 65534 failed - setegid (22: Invalid argument)
Reading package lists... Done
E: setgroups 65534 failed - setgroups (22: Invalid argument)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (22: Invalid argument)
E: Method http has died unexpectedly!
E: Sub-process http returned an error code (112)

Digging a bit further into this, I noticed that APT uses a dedicated _apt user with nogroup as its primary group.

root@397626021d9a:/# grep apt /etc/passwd
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin

At first glance, it seems like the auto-detection mechanism doesn't allocate enough ID's for the container.

# podman inspect 397626021d9a | jq '.[].HostConfig.IDMappings'
{
  "UidMap": [
    "0:2147549182:1024"
  ],
  "GidMap": [
    "0:2147549182:1024"
  ]
}

When manually specifying the size of 65535, APT works inside the container.
I also tried 65534, but that resulted in the same error as above.

root@nomadwodev01:~# podman run --userns auto:size=65535 -it docker.io/library/debian:bullseye-slim bash
root@854851aa6707:/# apt update
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [252 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [17.3 kB]
Fetched 8661 kB in 1s (7956 kB/s)                           
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.

I think it might be related to the following fix: #1473.
In our case, the above workaround is sufficient for now, since we have lots of ID's we can allocate.

Feel free to let me know if you'd need more information.