Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: invalid argument, permission denied when running containers with --userns=auto in rootful on FCOS #17171

Closed
lukasmrtvy opened this issue Jan 20, 2023 · 10 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue

Comments

@lukasmrtvy
Copy link

lukasmrtvy commented Jan 20, 2023

Issue Description

When running some ( probably the same ) containers with --userns=auto via API ( rootful ), sometimes ( its not random pattern I believe ) I will get invalid argument and permission denied error, but just for only one container, the other will run correctly. ( My application is using http://podman.io/libpod/containers/{name}/attach endpoint for streaming logs ). I am trying with 4.4.0-dev on Fedora CoreOS. I do not have the reproducer yet but I will edit this ASAP once I figure out how to reproduce this problem.

Jan 18 22:58:14 ip-10-1-94-157 podman[1084]: time="2023-01-18T22:58:14Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/merged: invalid argument"
Jan 18 22:58:14 ip-10-1-94-157 podman[1084]: time="2023-01-18T22:58:14Z" level=info msg="Request Failed(Conflict): preparing container 34414fbb5c439e0db40e1a5c068acb48ab2b5c29e9050a5feaeb3db80146218a for attach: mounting storage for container 34414fbb5c439e0db40e1a5c068acb48ab2b5c29e9050a5feaeb3db80146218a: creating overlay mount to /var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/merged, mount_data=\"lowerdir=/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/WUXOKHJUOPWJWK3BCZUSDIFWN4:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/diff1:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/IVE3DK32Q3PTXFEZ5XCBLKDVLM:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/IOXXSGV6RGSQE3VSQRQ3VT5N6P:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/HIQQV3SHYSJMVVBWMBDGHZJTG7:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/R3SCWQDEKP5RZDRSZTLOG3524D:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/2NTMRE74EULTEMQRC77B4OT67N:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/VRXVHRZLZF5OEVV5Q34BN55H2H:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/RECTHXXK3O23B5Y2YRXGT6YU2D:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/GS5RHTJ6CDYWHITCH7X4GHBJPF:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/ZRZ23KK4HEDOGK3IB2IRHHRHUT:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/BEU646LC7T32NVAJXBHOFWGNY3:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/AOJG6PLJZS7INSWPTKHRSZPRRE:/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/mapped/0/l/DFPYDIA5VSW3YVJOJ3UY3R5SEZ,upperdir=/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/diff,workdir=/var/lib/containers/storage/overlay/62428e0039a780c6b0a816a5e1543033e15ba1b619a903eaa2e962845162e9f2/work,nodev,metacopy=on,context=\\\"system_u:object_r:container_file_t:s0:c564,c896\\\"\": permission denied"

Related:

Steps to reproduce the issue

Steps to reproduce the issue

  1. TODO

Describe the results you received

Some containers are not running with --users=auto in rootful on FCOS.

Describe the results you expected

Some containers are running with --users=auto in rootful on FCOS.

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-2.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: '
  cpuUtilization:
    idlePercent: 99.74
    systemPercent: 0.07
    userPercent: 0.19
  cpus: 16
  distribution:
    distribution: fedora
    variant: coreos
    version: "37"
  eventLogger: journald
  hostname: ip-10-1-36-183
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.0.12-300.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 63377928192
  memTotal: 66354380800
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-0.0-20230118104133.e6c1ca3.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 62fdf71b63c8026fc879bd1f6459bddb16599538
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +LIBKRUN +WASM:wasmedge +WASM:wasmtime +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 36m 16.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1098974756864
  graphRootUsed: 8778694656
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /var/home/core
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.0-dev
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.0-dev

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Cloud:

AWS EC2 instance

OS:

Fedora CoreOS 37.20221211.3.0

Version:

podman version
Client:       Podman Engine
Version:      4.4.0-dev
API Version:  4.4.0-dev
Go Version:   go1.19.5
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Podman package from copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next

podman-0.0.git.17745.8f0a0f3e-1.fc37.x86_64

Additional information

No response

@vrothberg
Copy link
Member

Thanks for reaching out!

@giuseppe @rhatdan WDYT?

@giuseppe
Copy link
Member

have you tried the workaround suggested here: coreos/fedora-coreos-tracker#1305 (comment) ?

@lukasmrtvy
Copy link
Author

lukasmrtvy commented Jan 26, 2023

@giuseppe yes ( does not work ), its a slightly different problem, in coreos/fedora-coreos-tracker#1305 no container can start, while I can start containers, but seems that attaching when running two same containers can cause these symptoms ( but again its failing only for one container, probably for the newer one )

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Mar 1, 2023

@giuseppe @lukasmrtvy Any update on this one?

@lukasmrtvy
Copy link
Author

lukasmrtvy commented Mar 3, 2023

@rhatdan same error in 4.4.1 ( from dmesg [ 4272.392529] overlayfs: failed to resolve '/var/lib/containers/storage/overlay/a00f5922145a995092256a9087461fe62ca68b8eec5a5f72145c53cbbb925e6f/mapped/0/l/736REOLUVK2A7NNRGUSI3D2AA2': -13 )

I also noticed there is another error ending with"system_u:object_r:container_file_t:s0:c97,c611": "no such file or directory"
[ 4251.770374] overlayfs: failed to resolve ( from dmesg '/var/lib/containers/storage/overlay/6b3a6a04a01dd9995322602805f7f425de7f576318de2d0d4b0e56bb2404dd40/mapped/0/l/diff1': -2)

both related to attaching the container. Will try to do more investigation.

@ykuksenko
Copy link

@lukasmrtvy have you tried either deleting the underlying container image and pulling again or exporting the image, deleting and importing back again? The thread that @giuseppe mentioned had those workarounds added recently, after original one he referred to. These seem to fix the immediate issue though not the underlying cause for me.

I have not found a pattern as to why it happens or how to reproduce this. This happened again to me but this time it was the container launched correctly a few times and then it did not (same podman version and os version between launches). The podman output is the same each time but dmesg output does seem to vary a bit for me. I have seen return code -2 most of the time but I have also seen -13 and -1. The -13 and -1 return code outputs match your -13 output just the code is different. -2 for me matches your output.

@lukasmrtvy
Copy link
Author

lukasmrtvy commented Mar 8, 2023

@ykuksenko thanks

using ( podman.service is triggered by podman.socket, so this should work out of the box ):

[Unit]
Description=Disable idmapped overlayfs mounts of Podman containers (bugfix)
Before=podman.service
RequiresMountsFor=%t/containers

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'if [ -f /run/containers/storage/overlay/idmapped-lower-dir-true ]; then mv /run/containers/storage/overlay/idmapped-lower-dir-true /run/containers/storage/overlay/idmapped-lower-dir-false; else if [ ! -d /run/containers/storage/overlay ]; then mkdir -p -m 700 /run/containers/storage/overlay; fi; touch /run/containers/storage/overlay/idmapped-lower-dir-false; fi'
RemainAfterExit=true

[Install]
RequiredBy=podman.service

snippet from coreos/fedora-coreos-tracker#1305 (comment) and its working, at least I dont see any errors mentioned in this issue anymore.

@github-actions
Copy link

github-actions bot commented Apr 8, 2023

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Apr 12, 2023

I think this has resolved itself, so closing.

@rhatdan rhatdan closed this as completed Apr 12, 2023
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Aug 27, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue
Projects
None yet
Development

No branches or pull requests

5 participants