Add platforms build #9729

glours · 2022-08-08T15:10:06Z

What I did
Add support of the platforms attribut, recently introduced in the Compose specification.

This feature will allow to build multi-arch images from the build command and push those imagines to a registry.

Huge thanks to @crazy-max for his help 🙏

Should fix #8531

(not mandatory) A picture of a cute animal, if possible in relation with what you did

nicksieger

LGTM! I have a couple questions.

pkg/compose/build.go

pkg/compose/build_buildkit.go

crazy-max

Some minor changes.

Also looking at some logic copied over from buildx to this repo I'm afraid we are going to drift in the future.

Just my 2 cents but Compose should not push Docker images at all. Making it platform-neutral to build and run is a valid use-case but it should end here. If users want to build and push Docker images from their compose files they should use Bake.

As this is already part of the specification, I guess we have to move forward. So in a follow-up we should instead forward build requests to docker buildx bake. Or maybe by using bake.ReadTargets and convert them to build opts like https://github.com/docker/buildx/blob/30d650862d26932ffa860465bcd61aa455595f60/commands/bake.go#L106-L120 but then you need extra logic to handle drivers and builders. Maybe there is room on our side to have better support for buildx as a library. cc @tonistiigi @jedevc

pkg/compose/build_buildkit.go

milas

Few questions from playing around with it but code LGTM at a high level

From a behavior standpoint, I think the most awkward thing is the necessity to create/activate a buildx builder out-of-band:

multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

Personally, I'd expect Compose to create & use a project-scoped buildx context, similar to how it creates a network automatically.

milas · 2022-08-31T14:40:45Z

pkg/compose/build.go

+		if err != nil {
+			return nil, err
+		}
+		plats = append(plats, p)


I think we need to make sure we don't add a duplicate if a platform is already present via DOCKER_DEFAULT_PLATFORM above:

services: hello: build: context: . platforms: - linux/amd64

$ DOCKER_DEFAULT_PLATFORM="linux/amd64" docker compose build hello ... => ERROR exporting to image 0.0s ------ > exporting to image: ------ failed to solve: number of platforms does not match references 2 1

(Given how cryptic the BuildKit error is, it might also be beneficial [perhaps in compose-go loader/validate?] to error if duplicates exist in service.build.platforms, but I'm less concerned about that.)

Also, is it weird that you can "extend" the supported platforms via DOCKER_DEFAULT_PLATFORM?

e.g. in the same example above, DOCKER_DEFAULT_PLATFORM=linux/arm64 docker compose build hello would build both AMD64 + ARM64 images even though only linux/amd64 is actually in compose.yaml.

Does it make sense for you if we manage the DOCKER_DEFAULT_PLATFORM like the service.platform, which means refuse to build if the DOCKER_DEFAULT_PLATFORM isn't defined in the service.build.platforms section?

pkg/compose/build.go

milas · 2022-08-31T14:53:06Z

pkg/compose/build.go

+			buildOptions.Exports = []bclient.ExportEntry{{
+				Type: "image",
+				Attrs: map[string]string{
+					"push": "true",
+				},
+			}}


Shouldn't this be type: "docker" + "load": "true" like the other one?

I believe this is legitimately trying to push to registry:

$ docker compose build hello ... => ERROR exporting to image 0.7s => => exporting layers 0.1s => => exporting manifest sha256:90bf7923cd97b6dec8f53161c7df1e6d2587ec52ccc060273cdc6b6ae8bd6c9c 0.0s => => exporting config sha256:ec0a66e1fe3ca40d1042b63f3f82a0ff10189f86c6b68c1dd2b6f3213ac7057a 0.0s => => exporting manifest sha256:8a1a9bea74c44f39ab83dd4a831d74845c1f4eda2c3f2a5c230e0988fa554236 0.0s => => exporting config sha256:296e0f515945635ccc81e78c428bdf2b341a725f8ad033afbff0db2d0615aec1 0.0s => => exporting manifest list sha256:87e79b4bda828912aba9e22edd32010ab65b56da406c307bf60b61e79616b8cb 0.0s => => pushing layers 0.5s ------ > exporting to image: ------ failed to solve: server message: insufficient_scope: authorization failed

I just realized it's doing this because you can't export multi-platform images with the docker export type

I wonder if we need a --output flag on build with a default of type=docker that behaves the same as the buildx flag, so it can error out if you try to do docker compose build --output=docker on a multi-platform image.

And TIL that docker compose push already exists 🤔

But of course, it does not build:

$ docker compose push An image does not exist locally with the tag: milas/hello

Shouldn't this be type: "docker" + "load": "true" like the other one?

I believe this is legitimately trying to push to registry:

$ docker compose build hello ... => ERROR exporting to image 0.7s => => exporting layers 0.1s => => exporting manifest sha256:90bf7923cd97b6dec8f53161c7df1e6d2587ec52ccc060273cdc6b6ae8bd6c9c 0.0s => => exporting config sha256:ec0a66e1fe3ca40d1042b63f3f82a0ff10189f86c6b68c1dd2b6f3213ac7057a 0.0s => => exporting manifest sha256:8a1a9bea74c44f39ab83dd4a831d74845c1f4eda2c3f2a5c230e0988fa554236 0.0s => => exporting config sha256:296e0f515945635ccc81e78c428bdf2b341a725f8ad033afbff0db2d0615aec1 0.0s => => exporting manifest list sha256:87e79b4bda828912aba9e22edd32010ab65b56da406c307bf60b61e79616b8cb 0.0s => => pushing layers 0.5s ------ > exporting to image: ------ failed to solve: server message: insufficient_scope: authorization failed

Nope because docker exporter doesn't support multi-arch builds, I just need to remove the push option by default

crazy-max · 2022-08-31T15:08:23Z

Few questions from playing around with it but code LGTM at a high level

From a behavior standpoint, I think the most awkward thing is the necessity to create/activate a buildx builder out-of-band:
multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
Personally, I'd expect Compose to create & use a project-scoped buildx context, similar to how it creates a network automatically.

If compose can define its own buildx store that should be possible I think (using BUILDX_CONFIG=$HOME/.docker/compose/buildx env var maybe?). Then created builders would be scoped to compose.

glours · 2022-08-31T15:30:51Z

Few questions from playing around with it but code LGTM at a high level

From a behavior standpoint, I think the most awkward thing is the necessity to create/activate a buildx builder out-of-band:
multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
Personally, I'd expect Compose to create & use a project-scoped buildx context, similar to how it creates a network automatically.

@milas IHMO buildx is Compose defaut builder, so if users decided to use the legacy builder, they don't expect to some automagic behaviour during the build process, especially with an explicit discarded builder, right?

milas · 2022-09-01T14:51:37Z

Noticed today while using this build that I can't launch this project: https://github.com/dockersamples/compose-dev-env

❯ ~/dev/compose/bin/build/docker-compose up
[+] Building 1.6s (19/19) FINISHED
 => [compose-dev-env-backend internal] load build definition from Dockerfile                                                                           0.0s
 => => transferring dockerfile: 368B                                                                                                                   0.0s
 => [compose-dev-env-backend internal] load .dockerignore                                                                                              0.0s
 => => transferring context: 2B                                                                                                                        0.0s
 => [compose-dev-env-backend internal] load metadata for docker.io/library/alpine:3.12                                                                 1.0s
 => [compose-dev-env-backend internal] load metadata for docker.io/library/golang:1.16                                                                 0.3s
 => [compose-dev-env-proxy internal] load build definition from Dockerfile                                                                             0.0s
 => => transferring dockerfile: 147B                                                                                                                   0.0s
 => [compose-dev-env-proxy internal] load .dockerignore                                                                                                0.0s
 => => transferring context: 2B                                                                                                                        0.0s
 => [compose-dev-env-proxy internal] load metadata for docker.io/library/nginx:1.21                                                                    0.3s
 => [compose-dev-env-proxy 1/3] FROM docker.io/library/nginx:1.21@sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514              0.0s
 => => resolve docker.io/library/nginx:1.21@sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514                                    0.0s
 => [compose-dev-env-proxy internal] load build context                                                                                                0.0s
 => => transferring context: 157B                                                                                                                      0.0s
 => CACHED [compose-dev-env-proxy 2/3] RUN apt-get update     && apt-get install -y git                                                                0.0s
 => CACHED [compose-dev-env-proxy 3/3] COPY conf /etc/nginx/conf.d/default.conf                                                                        0.0s
 => [compose-dev-env-backend] exporting to image                                                                                                       0.2s
 => => exporting layers                                                                                                                                0.2s
 => => exporting manifest sha256:db821b139c4bd8c170ca26319bc2a57f68e7efe5763663828e8e77c7afa70366                                                      0.0s
 => => exporting config sha256:e5afd9082c8c57de530edec9b3dd2e890c1f1690652f84314aaa786178f26fa7                                                        0.0s
 => => exporting manifest sha256:c946c74d196d05d35421d351df959d6c8e5ca1bbdea83e8b0ae1d16d1714f875                                                      0.0s
 => => exporting config sha256:3fe4a410f3de7c181689ef16cc08595d3cff1f789b3fcde2dea49395514e9950                                                        0.0s
 => [compose-dev-env-backend build 1/4] FROM docker.io/library/golang:1.16@sha256:5f6a4662de3efc6d6bb812d02e9de3d8698eea16b8eb7281f03e6f3e8383018e     0.0s
 => => resolve docker.io/library/golang:1.16@sha256:5f6a4662de3efc6d6bb812d02e9de3d8698eea16b8eb7281f03e6f3e8383018e                                   0.0s
 => [compose-dev-env-backend stage-2 1/2] FROM docker.io/library/alpine:3.12@sha256:c75ac27b49326926b803b9ed43bf088bc220d22556de1bc5f72d742c91398f69   0.3s
 => => resolve docker.io/library/alpine:3.12@sha256:c75ac27b49326926b803b9ed43bf088bc220d22556de1bc5f72d742c91398f69                                   0.0s
 => => sha256:0eeab5c200691bd777e227c6eea27f7ca3c8232b67118a76edac2dcde3186aa1 2.72MB / 2.72MB                                                         0.2s
 => => extracting sha256:0eeab5c200691bd777e227c6eea27f7ca3c8232b67118a76edac2dcde3186aa1                                                              0.1s
 => [compose-dev-env-backend internal] load build context                                                                                              0.0s
 => => transferring context: 3.08kB                                                                                                                    0.0s
 => CACHED [compose-dev-env-backend build 2/4] WORKDIR /go/src/github.com/org/repo                                                                     0.0s
 => CACHED [compose-dev-env-backend build 3/4] COPY . .                                                                                                0.0s
 => CACHED [compose-dev-env-backend build 4/4] RUN go build -o server .                                                                                0.0s
 => [compose-dev-env-backend stage-2 2/2] COPY --from=build /go/src/github.com/org/repo/server /server                                                 0.0s
[+] Running 3/3
 ⠿ Network compose-dev-env_default      Created                                                                                                        0.0s
 ⠿ Volume "compose-dev-env_db-data"     Created                                                                                                        0.0s
 ⠿ Container compose-dev-env-db-1       Created                                                                                                        0.0s
 ⠋ Container compose-dev-env-backend-1  Creating                                                                                                       0.0s
Error: No such image: compose-dev-env-backend

(Not actually running via dev envs feature, using the docker-compose.yaml in the root of that repo)

Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>

Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

…ompose file Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

…sts) support DOCKER_DEFAULT_PLATFORM when 'compose up --build' add tests to check behaviour when DOCKER_DEFAULT_PLATFORM is defined Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

…mands Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

milas

!!

milas · 2022-09-02T15:08:33Z

pkg/utils/slices.go

+
+// Contains helps to detect if a non-comparable struct is part of an array
+// only use this method if you can't rely on existing golang Contains function of slices (https://pkg.go.dev/golang.org/x/exp/slices#Contains)
+func Contains[T any](origin []T, element T) bool {


milas · 2022-09-02T15:34:46Z

@milas IHMO buildx is Compose defaut builder, so if users decided to use the legacy builder, they don't expect to some automagic behaviour during the build process, especially with an explicit discarded builder, right?

I think the awkwardness is for users who are using BuildKit but with the default Docker driver, so they will likely be confused if they don't know about buildx + builder contexts.

That said, I think the current approach is fine, as the error message does give you the command to create a buildx context, at which point things will work.

In the future, maybe a new flag, e.g. docker compose build --current-platform would be useful, so users who have multiple platforms defined can still build locally independently from up without needing to set up buildx + do extra builds for other platforms they don't need during development

lara-ec · 2022-09-14T13:14:23Z

Hi,
We have a docker-compose.yml that specifies platform: linux/amd64 for certain containers, since they don't properly run on arm64, so that the env also runs on M1 macs.
This change breaks our current config with error service.platform should be part of the service.build.platforms: "linux/amd64", and I need to add a platforms entry to all the build sections to fix that. Is this intentional?
Thanks :)

P.S. love the panda picture!

nicksieger · 2022-09-14T14:52:47Z

@lara-ec do your compose services that need to be constrained to amd64 have a build section at all? Would you mind including a sample snippet of your compose file? Thanks.

glours · 2022-09-14T20:09:02Z

@lara-ec can you confirm me that it's broken for all developers (M1 and Intel)?
The error happens for docker compose build command? What is the result of a docker compose up --build?

glours · 2022-09-14T21:45:04Z

@lara-ec can you test this PR and let me know if it fixes your issue for both arm64 and amd64 computers?

lara-ec · 2022-09-15T08:49:41Z

@nicksieger Yes, we need a build section, the issue is that the underlying image doesn't work on arm64.

service:
  platform: linux/amd64
  build:
    context: ./service/
    dockerfile: Dockerfile.dev

@glours I've only tested it on Intel, I can test it on M1 tomorrow. But at least on Intel, the PR fixes it :)
And it happens both with build and up --build.

glours · 2022-09-15T08:54:12Z

@lara-ec, @StefanScherer and I reproduced your issue, we confirm on our side that the PR resolve the problem.
We'll do a v2.11.1 release soon

glours force-pushed the add-platforms-build branch 2 times, most recently from 006502d to 460ae51 Compare August 8, 2022 15:25

glours marked this pull request as ready for review August 12, 2022 20:37

glours marked this pull request as draft August 12, 2022 20:37

glours force-pushed the add-platforms-build branch from 460ae51 to debd390 Compare August 16, 2022 17:46

glours force-pushed the add-platforms-build branch 3 times, most recently from 2706f74 to 8a46422 Compare August 31, 2022 10:18

glours marked this pull request as ready for review August 31, 2022 13:18

glours requested review from a team and crazy-max August 31, 2022 13:18

nicksieger reviewed Aug 31, 2022

View reviewed changes

pkg/compose/build.go Outdated Show resolved Hide resolved

pkg/compose/build_buildkit.go Outdated Show resolved Hide resolved

crazy-max reviewed Aug 31, 2022

View reviewed changes

pkg/compose/build_buildkit.go Outdated Show resolved Hide resolved

pkg/compose/build_buildkit.go Outdated Show resolved Hide resolved

pkg/compose/build_buildkit.go Outdated Show resolved Hide resolved

milas reviewed Aug 31, 2022

View reviewed changes

glours force-pushed the add-platforms-build branch from 430efa5 to 7a321d7 Compare August 31, 2022 15:01

glours force-pushed the add-platforms-build branch 2 times, most recently from 0ac4fa2 to f900a64 Compare August 31, 2022 19:04

glours requested review from nicksieger, crazy-max and milas August 31, 2022 19:06

glours force-pushed the add-platforms-build branch from f900a64 to 23c9aa6 Compare September 1, 2022 12:54

glours self-assigned this Sep 2, 2022

glours and others added 5 commits September 2, 2022 15:44

add support of platforms in build section

8b1b708

Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>

fix panic when using 'compose up --build'

537f023

Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

add a test with multiple service builds using platforms in the same c…

8ed2d8a

…ompose file Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

don't push images at the end of multi-arch build (and simplify e2e te…

e016faa

…sts) support DOCKER_DEFAULT_PLATFORM when 'compose up --build' add tests to check behaviour when DOCKER_DEFAULT_PLATFORM is defined Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

always use 'docker' export entry when building with 'up' or 'run' com…

44c55e8

…mands Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>

glours force-pushed the add-platforms-build branch from 43169d5 to 44c55e8 Compare September 2, 2022 13:45

milas approved these changes Sep 2, 2022

View reviewed changes

ndeloof approved these changes Sep 8, 2022

View reviewed changes

glours merged commit 0ac0e29 into docker:v2 Sep 8, 2022

glours mentioned this pull request Sep 14, 2022

keep the platform defined at service level during build if no build patforms provided #9847

Merged

johanneswuerbach mentioned this pull request Sep 15, 2022

feat: use current buildx builder #9466

Closed

laurazard mentioned this pull request Sep 15, 2022

Cannot use the buildx with the docker compose v2 (cache_to feature not supported) #9832

Closed

gachowy mentioned this pull request Sep 21, 2022

docker-compose cannot build image for different architecture #9864

Closed

strophy mentioned this pull request Sep 23, 2022

build: remove buildx bake workaround dashpay/platform#541

Merged

5 tasks

glours deleted the add-platforms-build branch January 11, 2023 14:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add platforms build #9729

Add platforms build #9729

glours commented Aug 8, 2022 •

edited

nicksieger left a comment

crazy-max left a comment

milas left a comment

milas Aug 31, 2022

milas Aug 31, 2022

glours Aug 31, 2022

milas Aug 31, 2022

milas Aug 31, 2022

milas Aug 31, 2022

glours Aug 31, 2022 •

edited

crazy-max commented Aug 31, 2022

glours commented Aug 31, 2022 •

edited

milas commented Sep 1, 2022

milas left a comment

milas Sep 2, 2022

milas commented Sep 2, 2022

lara-ec commented Sep 14, 2022 •

edited

nicksieger commented Sep 14, 2022

glours commented Sep 14, 2022 •

edited

glours commented Sep 14, 2022

lara-ec commented Sep 15, 2022 •

edited

glours commented Sep 15, 2022

Add platforms build #9729

Add platforms build #9729

Conversation

glours commented Aug 8, 2022 • edited

nicksieger left a comment

Choose a reason for hiding this comment

crazy-max left a comment

Choose a reason for hiding this comment

milas left a comment

Choose a reason for hiding this comment

milas Aug 31, 2022

Choose a reason for hiding this comment

milas Aug 31, 2022

Choose a reason for hiding this comment

glours Aug 31, 2022

Choose a reason for hiding this comment

milas Aug 31, 2022

Choose a reason for hiding this comment

milas Aug 31, 2022

Choose a reason for hiding this comment

milas Aug 31, 2022

Choose a reason for hiding this comment

glours Aug 31, 2022 • edited

Choose a reason for hiding this comment

crazy-max commented Aug 31, 2022

glours commented Aug 31, 2022 • edited

milas commented Sep 1, 2022

milas left a comment

Choose a reason for hiding this comment

milas Sep 2, 2022

Choose a reason for hiding this comment

milas commented Sep 2, 2022

lara-ec commented Sep 14, 2022 • edited

nicksieger commented Sep 14, 2022

glours commented Sep 14, 2022 • edited

glours commented Sep 14, 2022

lara-ec commented Sep 15, 2022 • edited

glours commented Sep 15, 2022

glours commented Aug 8, 2022 •

edited

glours Aug 31, 2022 •

edited

glours commented Aug 31, 2022 •

edited

lara-ec commented Sep 14, 2022 •

edited

glours commented Sep 14, 2022 •

edited

lara-ec commented Sep 15, 2022 •

edited