Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add platforms build #9729

Merged
merged 5 commits into from Sep 8, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
35 changes: 35 additions & 0 deletions cmd/compose/tracing.go
@@ -0,0 +1,35 @@
/*
Copyright 2020 Docker Compose CLI authors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package compose

import (
"github.com/moby/buildkit/util/tracing/detect"
"go.opentelemetry.io/otel"

_ "github.com/moby/buildkit/util/tracing/detect/delegated" //nolint:blank-imports
_ "github.com/moby/buildkit/util/tracing/env" //nolint:blank-imports
)

func init() {
detect.ServiceName = "compose"
// do not log tracing errors to stdio
otel.SetErrorHandler(skipErrors{})
}

type skipErrors struct{}

func (skipErrors) Handle(err error) {}
12 changes: 10 additions & 2 deletions go.mod
Expand Up @@ -101,7 +101,7 @@ require (
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.29.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.29.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.29.0 // indirect
go.opentelemetry.io/otel v1.4.1 // indirect
go.opentelemetry.io/otel v1.4.1
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.4.1 // indirect
go.opentelemetry.io/otel/internal/metric v0.27.0 // indirect
go.opentelemetry.io/otel/metric v0.27.0 // indirect
Expand All @@ -122,17 +122,25 @@ require (
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/apimachinery v0.24.1 // indirect; see replace for the actual version used
k8s.io/client-go v0.24.1 // indirect; see replace for the actual version used
k8s.io/client-go v0.24.1 // see replace for the actual version used
k8s.io/klog/v2 v2.60.1 // indirect
k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect
sigs.k8s.io/yaml v1.2.0 // indirect
)

require (
github.com/cenkalti/backoff/v4 v4.1.2 // indirect
github.com/cyberphone/json-canonicalization v0.0.0-20210303052042-6bc126869bf4 // indirect
github.com/googleapis/gnostic v0.5.5 // indirect
github.com/moby/spdystream v0.2.0 // indirect
github.com/serialx/hashring v0.0.0-20190422032157-8b2912629002 // indirect
github.com/zmap/zcrypto v0.0.0-20220605182715-4dfcec6e9a8c // indirect
github.com/zmap/zlint v1.1.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.4.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.4.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.4.1 // indirect
k8s.io/api v0.24.1 // indirect
)

replace (
Expand Down
9 changes: 9 additions & 0 deletions go.sum
Expand Up @@ -246,6 +246,7 @@ github.com/campoy/unique v0.0.0-20180121183637-88950e537e7e/go.mod h1:9IOqJGCPMS
github.com/cavaliercoder/go-cpio v0.0.0-20180626203310-925f9528c45e/go.mod h1:oDpT4efm8tSYHXV5tHSdRvBet/b/QzxZ+XyyPehvm3A=
github.com/cenkalti/backoff v2.1.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
github.com/cenkalti/backoff/v4 v4.1.1/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw=
github.com/cenkalti/backoff/v4 v4.1.2 h1:6Yo7N8UP2K6LWZnW94DLVSSrbobcWdVzAYOisuDPIFo=
github.com/cenkalti/backoff/v4 v4.1.2/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw=
github.com/census-instrumentation/opencensus-proto v0.2.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
Expand Down Expand Up @@ -498,6 +499,7 @@ github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5m
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
github.com/elazarl/goproxy v0.0.0-20191011121108-aa519ddbe484 h1:pEtiCjIXx3RvGjlUJuCNxNOw0MNblyR9Wi+vJGBFh+8=
github.com/elazarl/goproxy v0.0.0-20191011121108-aa519ddbe484/go.mod h1:Ro8st/ElPeALwNFlcTpWmkr6IoMFfkjXAvTHpevnDsM=
github.com/elazarl/goproxy/ext v0.0.0-20190711103511-473e67f1d7d2/go.mod h1:gNh8nYJoAm43RfaxurUnxr+N1PwuFV3ZMl/efxlIlY8=
github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
Expand Down Expand Up @@ -761,6 +763,7 @@ github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsC
github.com/googleapis/gnostic v0.2.2/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg=
github.com/googleapis/gnostic v0.5.1/go.mod h1:6U4PtQXGIEt/Z3h5MAT7FNofLnw9vXk2cUuW7uA/OeU=
github.com/googleapis/gnostic v0.5.5 h1:9fHAtK0uDfpveeqqo1hkEZJcFvYXAiCN3UutL8F9xHw=
github.com/googleapis/gnostic v0.5.5/go.mod h1:7+EbHbldMins07ALC74bsA81Ovc97DwqyJO1AENw9kA=
github.com/gookit/color v1.2.4/go.mod h1:AhIE+pS6D4Ql0SQWbBeXPHw7gY0/sjHoA4s/n1KB7xg=
github.com/gophercloud/gophercloud v0.1.0/go.mod h1:vxM41WHh5uqHVBMZHzuwNOHh8XEoIEcSTewFxm1c5g8=
Expand Down Expand Up @@ -1022,6 +1025,7 @@ github.com/moby/buildkit v0.10.4 h1:FvC+buO8isGpUFZ1abdSLdGHZVqg9sqI4BbFL8tlzP4=
github.com/moby/buildkit v0.10.4/go.mod h1:Yajz9vt1Zw5q9Pp4pdb3TCSUXJBIroIQGQ3TTs/sLug=
github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8=
github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c=
github.com/moby/sys/mount v0.1.0/go.mod h1:FVQFLDRWwyBjDTBNQXDlWnSFREqOo3OKX9aqhmeoo74=
github.com/moby/sys/mount v0.1.1/go.mod h1:FVQFLDRWwyBjDTBNQXDlWnSFREqOo3OKX9aqhmeoo74=
Expand Down Expand Up @@ -1235,6 +1239,7 @@ github.com/securego/gosec v0.0.0-20200401082031-e946c8c39989/go.mod h1:i9l/TNj+y
github.com/securego/gosec/v2 v2.3.0/go.mod h1:UzeVyUXbxukhLeHKV3VVqo7HdoQR9MrRfFmZYotn8ME=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
github.com/serialx/hashring v0.0.0-20190422032157-8b2912629002 h1:ka9QPuQg2u4LGipiZGsgkg3rJCo4iIUCy75FddM0GRQ=
github.com/serialx/hashring v0.0.0-20190422032157-8b2912629002/go.mod h1:/yeG0My1xr/u+HZrFQ1tOQQQQrOawfyMUH13ai5brBc=
github.com/shirou/gopsutil v0.0.0-20190901111213-e4ec7b275ada/go.mod h1:WWnYX4lzhCH5h/3YBfyVA3VbLYjlMZZAQcW9ojMexNc=
github.com/shirou/w32 v0.0.0-20160930032740-bb4de0191aa4/go.mod h1:qsXQc7+bwAM3Q1u/4XEfrquwF8Lw7D7y5cD8CuHnfIc=
Expand Down Expand Up @@ -1465,13 +1470,16 @@ go.opentelemetry.io/otel v1.4.1/go.mod h1:StM6F/0fSwpd8dKWDCdRr7uRvEPYdW0hBSlbdT
go.opentelemetry.io/otel/exporters/jaeger v1.4.1/go.mod h1:ZW7vkOu9nC1CxsD8bHNHCia5JUbwP39vxgd1q4Z5rCI=
go.opentelemetry.io/otel/exporters/otlp v0.20.0/go.mod h1:YIieizyaN77rtLJra0buKiNBOm9XQfkPEKBeuhoMwAM=
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.3.0/go.mod h1:VpP4/RMn8bv8gNo9uK7/IMY4mtWLELsS+JIP0inH0h4=
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.4.1 h1:imIM3vRDMyZK1ypQlQlO+brE22I9lRhJsBDXpDWjlz8=
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.4.1/go.mod h1:VpP4/RMn8bv8gNo9uK7/IMY4mtWLELsS+JIP0inH0h4=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.3.0/go.mod h1:hO1KLR7jcKaDDKDkvI9dP/FIhpmna5lkqPUQdEjFAM8=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.4.1 h1:WPpPsAAs8I2rA47v5u0558meKmmwm1Dj99ZbqCV8sZ8=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.4.1/go.mod h1:o5RW5o2pKpJLD5dNTCmjF1DorYwMeFJmb/rKr5sLaa8=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.3.0/go.mod h1:keUU7UfnwWTWpJ+FWnyqmogPa82nuU5VUANFq49hlMY=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.4.1 h1:AxqDiGk8CorEXStMDZF5Hz9vo9Z7ZZ+I5m8JRl/ko40=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.4.1/go.mod h1:c6E4V3/U+miqjs/8l950wggHGL1qzlp0Ypj9xoGrPqo=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.3.0/go.mod h1:QNX1aly8ehqqX1LEa6YniTU7VY9I6R3X/oPxhGdTceE=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.4.1 h1:8qOago/OqoFclMUUj/184tZyRdDZFpcejSjbk5Jrl6Y=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.4.1/go.mod h1:VwYo0Hak6Efuy0TXsZs8o1hnV3dHDPNtDbycG0hI8+M=
go.opentelemetry.io/otel/internal/metric v0.27.0 h1:9dAVGAfFiiEq5NVB9FUJ5et+btbDQAUIJehJ+ikyryk=
go.opentelemetry.io/otel/internal/metric v0.27.0/go.mod h1:n1CVxRqKqYZtqyTh9U/onvKapPGv7y/rpyOTI+LFNzw=
Expand All @@ -1498,6 +1506,7 @@ go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
go.uber.org/goleak v1.1.12 h1:gZAh5/EyT/HQwlpkCy6wTpqfH9H8Lz8zbm3dZh+OyzA=
go.uber.org/goleak v1.1.12/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
Expand Down
71 changes: 56 additions & 15 deletions pkg/compose/build.go
Expand Up @@ -81,6 +81,12 @@ func (s *composeService) build(ctx context.Context, project *types.Project, opti
Attrs: map[string]string{"ref": image},
})
}
if len(buildOptions.Platforms) > 1 {
buildOptions.Exports = []bclient.ExportEntry{{
Type: "image",
Attrs: map[string]string{},
}}
}
opts[imageName] = buildOptions
}

Expand Down Expand Up @@ -162,6 +168,15 @@ func (s *composeService) getBuildOptions(project *types.Project, images map[stri
if err != nil {
return nil, err
}
opt.Exports = []bclient.ExportEntry{{
Type: "docker",
Attrs: map[string]string{
"load": "true",
},
}}
if opt.Platforms, err = useDockerDefaultPlatform(project, service.Build.Platforms); err != nil {
opt.Platforms = []specs.Platform{}
}
opts[imageName] = opt
continue
}
Expand Down Expand Up @@ -206,7 +221,7 @@ func (s *composeService) doBuild(ctx context.Context, project *types.Project, op
if buildkitEnabled, err := s.dockerCli.BuildKitEnabled(); err != nil || !buildkitEnabled {
return s.doBuildClassic(ctx, project, opts)
}
return s.doBuildBuildkit(ctx, project, opts, mode)
return s.doBuildBuildkit(ctx, opts, mode)
}

func (s *composeService) toBuildOptions(project *types.Project, service types.ServiceConfig, imageTag string, sshKeys []types.SSHKey) (build.Options, error) {
Expand All @@ -215,20 +230,9 @@ func (s *composeService) toBuildOptions(project *types.Project, service types.Se

buildArgs := flatten(service.Build.Args.Resolve(envResolver(project.Environment)))

var plats []specs.Platform
if platform, ok := project.Environment["DOCKER_DEFAULT_PLATFORM"]; ok {
p, err := platforms.Parse(platform)
if err != nil {
return build.Options{}, err
}
plats = append(plats, p)
}
if service.Platform != "" {
p, err := platforms.Parse(service.Platform)
if err != nil {
return build.Options{}, err
}
plats = append(plats, p)
plats, err := addPlatforms(project, service)
if err != nil {
return build.Options{}, err
}

cacheFrom, err := buildflags.ParseCacheEntry(service.Build.CacheFrom)
Expand Down Expand Up @@ -352,3 +356,40 @@ func addSecretsConfig(project *types.Project, service types.ServiceConfig) (sess
}
return secretsprovider.NewSecretProvider(store), nil
}

func addPlatforms(project *types.Project, service types.ServiceConfig) ([]specs.Platform, error) {
plats, err := useDockerDefaultPlatform(project, service.Build.Platforms)
if err != nil {
return nil, err
}

if service.Platform != "" && !utils.StringContains(service.Build.Platforms, service.Platform) {
return nil, fmt.Errorf("service.platform should be part of the service.build.platforms: %q", service.Platform)
}

for _, buildPlatform := range service.Build.Platforms {
p, err := platforms.Parse(buildPlatform)
if err != nil {
return nil, err
}
if !utils.Contains(plats, p) {
plats = append(plats, p)
}
}
return plats, nil
}

func useDockerDefaultPlatform(project *types.Project, platformList types.StringList) ([]specs.Platform, error) {
var plats []specs.Platform
if platform, ok := project.Environment["DOCKER_DEFAULT_PLATFORM"]; ok {
if !utils.StringContains(platformList, platform) {
return nil, fmt.Errorf("the DOCKER_DEFAULT_PLATFORM value should be part of the service.build.platforms: %q", platform)
}
p, err := platforms.Parse(platform)
if err != nil {
return nil, err
}
plats = append(plats, p)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to make sure we don't add a duplicate if a platform is already present via DOCKER_DEFAULT_PLATFORM above:

services:
  hello:
    build:
      context: .
      platforms:
        - linux/amd64
$ DOCKER_DEFAULT_PLATFORM="linux/amd64" docker compose build hello
...
 => ERROR exporting to image                                                                                                                           0.0s
------
 > exporting to image:
------
failed to solve: number of platforms does not match references 2 1

(Given how cryptic the BuildKit error is, it might also be beneficial [perhaps in compose-go loader/validate?] to error if duplicates exist in service.build.platforms, but I'm less concerned about that.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, is it weird that you can "extend" the supported platforms via DOCKER_DEFAULT_PLATFORM?

e.g. in the same example above, DOCKER_DEFAULT_PLATFORM=linux/arm64 docker compose build hello would build both AMD64 + ARM64 images even though only linux/amd64 is actually in compose.yaml.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense for you if we manage the DOCKER_DEFAULT_PLATFORM like the service.platform, which means refuse to build if the DOCKER_DEFAULT_PLATFORM isn't defined in the service.build.platforms section?

}
return plats, nil
}