Rootless Podman with zfs Storage Driver -- "Error: cannot find root filesystem"

[craigsloggett]

Issue Description

Rootless Podman is unable to list ZFS filesystems.

Here is the rootless storage.conf file:

[storage]
driver = "zfs"

[storage.options.zfs]
fsname = "zroot/containers"
mountopt = "nodev"

This is the error I get when attempting to do anything with rootless Podman:

$ podman system info
Error: cannot find root filesystem zroot/containers: exit status 1: "/usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers" => cannot open 'zroot/containers': dataset does not exist

When I run the command manually using the same user, I am able to list the filesystem just fine:

$ /usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers
zroot/containers        -       290816  480957955968    /home/<user>/.local/share/containers  lz4     filesystem      -       0       290816  290816  137728  290816

~/.local/share/containers is owned by <user>:<user> and here are the ZFS permissions:

$ zfs allow zroot/containers
---- Permissions on zroot/containers ---------------------------------
Local+Descendent permissions:
        user <user> create,destroy,mount,snapshot

Steps to reproduce the issue

1. $ podman system reset
2. $ su -
3. # zfs create -o mountpoint=/home/<user>/.local/share/containers -o dedup=on zroot/containers
4. # zfs allow <user> create,destroy,mount,snapshot zroot/containers
5. # chown <user>:<user> /home/<user>/.local/share/containers
6. exit
7. $ podman system info

Describe the results you received

After running podman system info (or any Podman command as a regular user):

$ podman system info
Error: cannot find root filesystem zroot/containers: exit status 1: "/usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers" => cannot open 'zroot/containers': dataset does not exist

Describe the results you expected

The zfs storage driver is able to "see" the dataset and returns the system info without error.

podman info output

# podman --version
podman version 4.7.1

# uname -a
Linux saturn 6.5.8-arch1-1 containers/podman#1 SMP PREEMPT_DYNAMIC Thu, 19 Oct 2023 22:52:14 +0000 x86_64 GNU/Linux

# cat /etc/os-release 
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

Interestingly, using the zfs storage driver as root works fine on the same system:

# podman system info                                                                                                                                                                                                                                           [63/4503]
host:                   
  arch: amd64 
  buildahVersion: 1.32.0            
  cgroupControllers:                                             
  - cpuset     
  - cpu                        
  - io                                                
  - memory           
  - hugetlb                      
  - pids               
  - rdma     
  - misc      
  cgroupManager: systemd                         
  cgroupVersion: v2
  conmon:            
    package: /usr/bin/conmon is owned by conmon 1:2.1.8-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 99.98
    systemPercent: 0.01
    userPercent: 0
  cpus: 40 
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2048                         
  hostname: saturn
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.5.8-arch1-1
  linkmode: dynamic   
  logDriver: journald
  memFree: 132329963520                 
  memTotal: 135066349568          
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:                                    
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.8.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.8.0 
    package: /usr/lib/podman/netavark is owned by netavark 1.8.0-1
    path: /usr/lib/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.10-1
    path: /usr/bin/crun
    version: |-                                  
      crun version 1.10
      commit: c053c83c57551bca13ead8600237341818975974
      rundir: /run/crun
      spec: 1.0.0                    
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux          
  pasta:   
    executable: ""   
    package: "" 
    version: ""
  remoteSocket:  
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 1h 57m 42.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: zfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 480958218240
  graphRootUsed: 262144
  graphStatus:
    Compression: lz4
    Parent Dataset: zroot/var/lib/containers
    Parent Quota: "no"
    Space Available: "480957980544"
    Space Used By Parent: "241664"
    Zpool: zroot
    Zpool Health: ONLINE
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.1
  Built: 1696583554
  BuiltTime: Fri Oct  6 05:12:34 2023
  GitCommit: ef83eeb9c7482826672f3efa12db3d61c88df6c4-dirty
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.1

[rhatdan]

No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.

[f1d094]

No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.

@rhatdan: Please clarify. Do you mean there are no podman devs who support zfs storage at the moment? Using overlayfs where zfs is native is not desirable.

I came to report the same bug but with a minor variation in the error:

$ STORAGE_DRIVER=zfs podman system info
Error: cannot find root filesystem rpool/var/lib/containers/storage-<user>: exit status 1: "/usr/sbin/zfs fs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/var/lib/containers/storage-<user>" => cannot open 'rpool/var/lib/containers/storage-<user>': dataset does not exist

If the error code is derived from the actual command executed, then it should be noted that: /usr/sbin/zfs fs list -rHp... is not a valid command.

I don't know where to look but I assume someone familiar with the code could verify it isn't a basic fat-finger error with minimal effort.