docker run ignores --platform

[pufferbatterie]

Description

docker run ignores --platform option and leads to a format "exec format error"

Reproduce

1. Build an docker image with different platform than the host
   echo -ne "FROM --platform=linux/i386 alpine \n RUN apk add htop" | docker buildx build --load - -t alpine_platform --progress=plain
   Output:
   1 [internal] load build definition from Dockerfile
   4 [1/2] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
   5 [2/2] RUN apk add htop
   5 0.105 fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86/APKINDEX.tar.gz
   5 0.544 fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86/APKINDEX.tar.gz
   5 1.106 (1/3) Installing ncurses-terminfo-base (6.3_p20220521-r0)
   5 1.117 (2/3) Installing ncurses-libs (6.3_p20220521-r0)
   5 1.145 (3/3) Installing htop (3.2.0-r1)
   5 1.169 Executing busybox-1.35.0-r17.trigger
   5 1.173 OK: 7 MiB in 17 packages
   5 DONE 1.2s
   [Output clipped]

• From step 5 you can see the emulation is working inside the build
• For demonstarion i chose i386 to be able to execute a program on x86_64

2. Verify image arch
   root@debian:~# docker inspect alpine_platform | grep Arch
   "Architecture": "386",

3. Run the image with host arch
   root@debian:~# docker run --rm -it alpine_platform uname -m
   WARNING: The requested image's platform (linux/386) does not match the detected host platform (linux/amd64) and no specific platform was requested
   x86_64

4. Run the image with target arch
   root@debian:~# docker run --platform linux/i386 --rm -it alpine_platform uname -m
   Output:
   x86_64

5. If you try it with arm it will result in a "exec /bin/uname: exec format error"

Expected behavior

docker run should use the --platform argument or return a qualifier error message

docker version

Client: Docker Engine - Community
 Cloud integration: v1.0.29
 Version:           20.10.18
 API version:       1.41
 Go version:        go1.18.6
 Git commit:        b40c2f6
 Built:             Thu Sep  8 23:12:08 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.18
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.6
  Git commit:       e42327a
  Built:            Thu Sep  8 23:09:59 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.10.2)
  extension: Manages Docker extensions (Docker Inc., v0.2.9)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.19.0)

Server:
 Containers: 6
  Running: 1
  Paused: 0
  Stopped: 5
 Images: 4
 Server Version: 20.10.18
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.0-18-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.77GiB
 Name: debian
 ID: 7EL4:PQV6:O4VG:ZENZ:YCX6:6VOT:J3DJ:4KKE:V42G:I2ZN:WVO6:AGAS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

root@debian:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

virtualized on Win10 in hyper-v

[AkihiroSuda]

4. Run the image with target arch
   root@debian:~# docker run --platform linux/i386 --rm -it alpine_platform uname -m
   Output:
   x86_64

Expected behavior, because uname(2) returns the actual host processor.

5. If you try it with arm it will result in a "exec /bin/uname: exec format error"

You need docker run --privileged --rm tonistiigi/binfmt --install all
https://github.com/tonistiigi/binfmt

[AkihiroSuda]

You can confirm the image architecture with file

$ docker run -it --rm --platform linux/i386 alpine
/ # apk add file
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86/APKINDEX.tar.gz
(1/2) Installing libmagic (5.41-r0)
(2/2) Installing file (5.41-r0)
Executing busybox-1.35.0-r17.trigger
OK: 13 MiB in 16 packages
/ # file /bin/busybox 
/bin/busybox: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-i386.so.1, stripped

[marcpawl-arista]

Still broken on 24.0.1

$ docker --version
Docker version 24.0.1, build 6802122

bsn@jenkins-w1:~$ docker run --platform linux/amd64 ubuntu:focal uname -a
Unable to find image 'ubuntu:focal' locally
focal: Pulling from library/ubuntu
Digest: **sha256:c9820a44b950956a790c354700c1166a7ec648bc0d215fa438d3a339812f1d01**
Status: Downloaded newer image for ubuntu:focal
Linux d262adab4901 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ docker run --platform linux/arm64/v8 ubuntu:focal uname -a
Unable to find image 'ubuntu:focal' locally
focal: Pulling from library/ubuntu
Digest: **sha256:c9820a44b950956a790c354700c1166a7ec648bc0d215fa438d3a339812f1d01**
Status: Downloaded newer image for ubuntu:focal
exec /usr/bin/uname: exec format error

The same digest is being used.

[thaJeztah]

That's expected, and the correct behavior. When using a manifest-list (AKA "multi-arch image"), the image tag refers to the manifest index. Multi-arch images are a collection of images, with one variant per platform. The manifest "index" groups those images in a single list, and the digest of that manifest-index is what's shown (as it's the "root" of the image).

You can use the docker buildx imagetools inspect utility (or docker manifest inspect, but its output isn't very readable) to inspect that digest;

docker buildx imagetools inspect ubuntu@sha256:c9820a44b950956a790c354700c1166a7ec648bc0d215fa438d3a339812f1d01

Name:      docker.io/library/ubuntu@sha256:c9820a44b950956a790c354700c1166a7ec648bc0d215fa438d3a339812f1d01
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:c9820a44b950956a790c354700c1166a7ec648bc0d215fa438d3a339812f1d01

Manifests:
  Name:      docker.io/library/ubuntu@sha256:8c38f4ea0b178a98e4f9f831b29b7966d6654414c1dc008591c6ec77de3bf2c9
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/amd64

  Name:      docker.io/library/ubuntu@sha256:9b996ae92d0d9f629f0c952ee9cc66187d331373b40fdf38dbd7b6f26510a145
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/arm/v7

  Name:      docker.io/library/ubuntu@sha256:7bdccf116db125b3e6e39eb67ca9e2ae890386acf95a13a4e8b69466b6eba5e2
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/arm64/v8

  Name:      docker.io/library/ubuntu@sha256:7f8e05f0fe5051ae5758e32e10fb8f4d03dadbeaed11ad1b4568d55c7b807212
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/ppc64le

  Name:      docker.io/library/ubuntu@sha256:7fa3e1a7f357f2ee21918f3ad5eeab7a0d02a7452acdaecc47f585ed6e52d76a
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/s390x

When running or pulling the image, the docker engine picks the best matching platform from the list (e.g. linux/amd64), which can be overridden with the --platform option.

That looks to work in your example, as trying to run a linux/arm64/v8 container (binary) on your machine (a linux x86 machine) fails (unless something like QEMU userland emulation is installed);

exec /usr/bin/uname: exec format error

All of the above said, we are in the process of integrating the containerd image store, which includes improved support for multi-arch images. As part of that, we're working on improving the UX (and making multi-arch images more visible). See these issues (among others) for more details;

• UX: improve / design UX for multi-arch images #44582
• UX: improve messages / warnings /errors to include platform (os/arch/xx) #41838

[thaJeztah]

Based on the above, and the earlier comment from @AkihiroSuda #44291 (comment), this looks to be working correctly (but some bits can be confusing). I'll close this ticket because of the above, but feel free to continue the conversation.

[marcpawl-arista]

The root problem was not the manifest id, but the fact that we are getting the exec format error. Please reopen.

[laurazard]

The root problem was not the manifest id, but the fact that we are getting the exec format error. Please reopen.

It seems to me that there is some confusion re: what --platform does vs. what you expect it to do.

docker run --platform does not enable cross-platform emulation, what it does is instruct docker re: what platform image to pull/run. If you try to run an image built for arm while on an x86 host, you will likely run into issues, and that's not unexpected.

[marcpawl-arista]

My understanding of @laurazard comment is slightly different than the documentation quoted below. On a server, not desktop, docker run --platform will select the image being used but not do the emulation on servers to match the platform argument, which seems pointless as it is always going to fail.

It would be friendlier:

• to have the docker run command produce a clean error if the platform is for something that would need be be emulated, but emulation is not supported.
• if the docker run documentation for --platform explicitly said that emulation is for docker desktop only.

There is a difference in behavior on Mac docker for desktop vs Linux. On Mac you can run AMD64 and ARM64 image with the exact same command line that failed on Linux. Nice for Mac, but a bit of surprise when it does not work on Linux, and there is no diagnostic issued by docker run.

The key piece in the documentation that I missed earlier is that cross platform run is limited to Docker desktop.

https://docs.docker.com/build/building/multi-platform/
Docker Desktop provides binfmt_misc multi-architecture support, which means you can run containers for different Linux architectures such as arm, mips, ppc64le, and even s390x.

From https://docs.docker.com/engine/reference/commandline/run/
--platform |   | Set platform if server is multi-platform capable

P.S.
It looks like there is an advanced usage that will allow running arm64 containers on amd64 on Linux, I have not given it a try. https://www.stereolabs.com/docs/docker/building-arm-container-on-x86/

[AkihiroSuda]

emulation is for docker desktop only

Untrue.
docker run --privileged --rm tonistiigi/binfmt --install all should work on non-desktop too.
https://github.com/tonistiigi/binfmt

[laurazard]

@marcpawl-arista

There is a difference in behavior on Mac docker for desktop vs Linux. On Mac you can run AMD64 and ARM64 image with the exact same command line that failed on Linux. Nice for Mac, but a bit of surprise when it does not work on Linux, and there is no diagnostic issued by docker run.

This is inaccurate: Docker Desktop for Mac and Docker Desktop for Linux both provide binfmt emulation out of the box, afaict. Running Docker baremetal, outside of DD, doesn't provide emulation anywhere (although you can install binfmt manually, as explained by @AkihiroSuda).

---

The key thing to understand is that --platform doesn't and isn't meant to do emulation. --platform allows specifying a specific platform/architecture image to use, but the execution of that image/binary/etc. is ultimately up to the host and it's capabilities – which can be enhanced with utilities like binfmt, but these exist and work separately from Docker.

[marcpawl-arista]

@laurazard I stand corrected, I did not realize that there was a Linux Desktop product and was sloppy using the term Linux instead of "Docker bare metal".

We were running

docker run --rm --privileged docker/binfmt:66f9012c56a8316f9244ffd7622d7c21c1f6f28d

Will try the tonistiigi/binfmt image.

I really wish the docker/binfmt was the official image to use, since docker implies a level of confidence on the safety of the image, especially needed when running privileged. Where Tõnis Tiigi is some random dude living in San Francisco.

PS: Reference to tonistiigi/binfmt is in https://docs.docker.com/build/building/multi-platform/