Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[master] disable seccomp to prevent tar EPERM (Ubuntu 24.04 armhf on 20.04 host) #1007

Merged
merged 1 commit into from Mar 19, 2024

Conversation

thaJeztah
Copy link
Member

tar is failing to apply permissions, but only on armhf if seccomp is enabled

docker run --rm -e PLATFORM -e EPOCH='5' -e DEB_VERSION=0.0.0~20240319100031.2356372 -e VERSION=0.0.0-20240319100031-2356372 -e CLI_GITCOMMIT=2356372 -e ENGINE_GITCOMMIT=23e1af4 -e BUILDX_VERSION=v0.13.1 -e BUILDX_DEB_VERSION=0.13.1 -e BUILDX_GITCOMMIT=7884339 -e COMPOSE_VERSION=v2.25.0 -e COMPOSE_DEB_VERSION=2.25.0 -v /home/ubuntu/workspace/docker-ce-packaging_PR-1006/deb/debbuild/ubuntu-noble:/build  debbuild-ubuntu-noble/aarch64
+ set -e
+ mkdir -p /root/build-deb/engine
+ tar -C /root/build-deb -xzf /sources/engine.tgz
tar: engine/daemon/config: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: engine/daemon/stats: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: engine/daemon/images: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: engine/daemon/containerd: Cannot change mode to rwxrwxr-x: Operation not permitted
...
tar: engine/integration-cli: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: Exiting with failure status due to previous errors
make[1]: *** [Makefile:80: ubuntu-noble] Error 2
make[1]: Leaving directory '/home/ubuntu/workspace/docker-ce-packaging_PR-1006/deb'
make: *** [Makefile:83: ubuntu-noble] Error 2
script returned exit code 2

This is failing when the ubuntu 24.04 armhf container runs on a ubuntu 20.04 host;

Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:22:23 2024
 OS/Arch:           linux/arm
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:22:23 2024
  OS/Arch:          linux/arm
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Client: Docker Engine - Community
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-1053-aws
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: armv8l
 CPUs: 2
 Total Memory: 7.559GiB
 Name: ip-10-100-72-215
 ID: e901fd37-4acc-4712-8a91-b41a94767d78
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: dockerbuildbot
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

- What I did

- How I did it

- How to verify it

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

@thaJeztah thaJeztah marked this pull request as ready for review March 19, 2024 12:47
@thaJeztah thaJeztah self-assigned this Mar 19, 2024
@neersighted neersighted changed the title [master] disable seccomp to prevent tar ENOTSUP (Ubuntu 24.04 armhf on 20.04 host) [master] disable seccomp to prevent tar EPERM (Ubuntu 24.04 armhf on 20.04 host) Mar 19, 2024
tar is failing to apply permissions, but only on `armhf` if seccomp is enabled

    docker run --rm -e PLATFORM -e EPOCH='5' -e DEB_VERSION=0.0.0~20240319100031.2356372 -e VERSION=0.0.0-20240319100031-2356372 -e CLI_GITCOMMIT=2356372 -e ENGINE_GITCOMMIT=23e1af4 -e BUILDX_VERSION=v0.13.1 -e BUILDX_DEB_VERSION=0.13.1 -e BUILDX_GITCOMMIT=7884339 -e COMPOSE_VERSION=v2.25.0 -e COMPOSE_DEB_VERSION=2.25.0 -v /home/ubuntu/workspace/docker-ce-packaging_PR-1006/deb/debbuild/ubuntu-noble:/build  debbuild-ubuntu-noble/aarch64
    + set -e
    + mkdir -p /root/build-deb/engine
    + tar -C /root/build-deb -xzf /sources/engine.tgz
    tar: engine/daemon/config: Cannot change mode to rwxrwxr-x: Operation not permitted
    tar: engine/daemon/stats: Cannot change mode to rwxrwxr-x: Operation not permitted
    tar: engine/daemon/images: Cannot change mode to rwxrwxr-x: Operation not permitted
    tar: engine/daemon/containerd: Cannot change mode to rwxrwxr-x: Operation not permitted
    ...
    tar: engine/integration-cli: Cannot change mode to rwxrwxr-x: Operation not permitted
    tar: Exiting with failure status due to previous errors
    make[1]: *** [Makefile:80: ubuntu-noble] Error 2
    make[1]: Leaving directory '/home/ubuntu/workspace/docker-ce-packaging_PR-1006/deb'
    make: *** [Makefile:83: ubuntu-noble] Error 2
    script returned exit code 2

seccomp is blocking the fchmodat2 syscall because the host doesn't support it;

   fchmodat2(AT_FDCWD, "docker", 0775, AT_SYMLINK_NOFOLLOW) = -1 ENOSYS (Function not implemented)

This is failing when the ubuntu 24.04 armhf container runs on a ubuntu 20.04
host;

    Client: Docker Engine - Community
     Version:           25.0.3
     API version:       1.44
     Go version:        go1.21.6
     Git commit:        4debf41
     Built:             Tue Feb  6 21:22:23 2024
     OS/Arch:           linux/arm
     Context:           default

    Server: Docker Engine - Community
     Engine:
      Version:          25.0.3
      API version:      1.44 (minimum version 1.24)
      Go version:       go1.21.6
      Git commit:       f417435
      Built:            Tue Feb  6 21:22:23 2024
      OS/Arch:          linux/arm
      Experimental:     true
     containerd:
      Version:          1.6.28
      GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
     runc:
      Version:          1.1.12
      GitCommit:        v1.1.12-0-g51d5e94
     docker-init:
      Version:          0.19.0
      GitCommit:        de40ad0

    Client: Docker Engine - Community
     Version:    25.0.3
     Context:    default
     Debug Mode: false
     Plugins:
      buildx: Docker Buildx (Docker Inc.)
        Version:  v0.12.1
        Path:     /usr/libexec/docker/cli-plugins/docker-buildx
      compose: Docker Compose (Docker Inc.)
        Version:  v2.24.5
        Path:     /usr/libexec/docker/cli-plugins/docker-compose

    Server:
     Containers: 0
      Running: 0
      Paused: 0
      Stopped: 0
     Images: 5
     Server Version: 25.0.3
     Storage Driver: overlay2
      Backing Filesystem: extfs
      Supports d_type: true
      Using metacopy: false
      Native Overlay Diff: true
      userxattr: false
     Logging Driver: json-file
     Cgroup Driver: cgroupfs
     Cgroup Version: 1
     Plugins:
      Volume: local
      Network: bridge host ipvlan macvlan null overlay
      Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
     Swarm: inactive
     Runtimes: runc io.containerd.runc.v2
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
     runc version: v1.1.12-0-g51d5e94
     init version: de40ad0
     Security Options:
      apparmor
      seccomp
       Profile: builtin
     Kernel Version: 5.15.0-1053-aws
     Operating System: Ubuntu 20.04.6 LTS
     OSType: linux
     Architecture: armv8l
     CPUs: 2
     Total Memory: 7.559GiB
     Name: ip-10-100-72-215
     ID: e901fd37-4acc-4712-8a91-b41a94767d78
     Docker Root Dir: /var/lib/docker
     Debug Mode: false
     Username: dockerbuildbot
     Experimental: true
     Insecure Registries:
      127.0.0.0/8
     Live Restore Enabled: true

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Copy link
Member Author

Updated the commit message to match the title of the PR

Copy link
Member

@neersighted neersighted left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've confirmed this is related to fchmodat2(2); from a good run, we have:

fchmodat2(AT_FDCWD, "docker", 0775, AT_SYMLINK_NOFOLLOW) = -1 ENOSYS (Function not implemented)

And in a bad run:

fchmodat2(AT_FDCWD, "docker", 0775, AT_SYMLINK_NOFOLLOW) = -1 EPERM (Operation not permitted)
tar: docker: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: Exiting with failure status due to previous errors

This is ultimately because of moby/moby#42871; new syscalls must be added to the profiles (and supported by libseccomp, which is the issue here) in order to work. If they are unknown to either the profile or libseccomp, they will fail with EPERM, which triggers these... less than graceful degradations.

@neersighted neersighted merged commit f54c408 into docker:master Mar 19, 2024
6 checks passed
@thaJeztah thaJeztah deleted the disable_seccomp branch March 19, 2024 14:30
@clnperez
Copy link

since we're seeing this also on ppc64le -- can we just update the default profile to include fchmodat2 ? It looks like this one just recently got picked up: seccomp/libseccomp#406 (comment)

@thaJeztah
Copy link
Member Author

It's already in the profile (for engine v25.0.3 and up); moby/moby#47344

And looks like CI ran on 25.0.3;

Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:22:23 2024
 OS/Arch:           linux/arm
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:22:23 2024
  OS/Arch:          linux/arm
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I think the problem in this case is that the underlying host is running on an older kernel, and because of that, seccomp doesn't support the syscall, and causes it to be blocked;

 Kernel Version: 5.15.0-1053-aws
 Operating System: Ubuntu 20.04.6 LTS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants