[master] fix and refactor static packages #665

crazy-max · 2022-03-29T09:51:51Z

fixes an issue introduced by #654 while adding buildx static package. also review the bits that generates the static packages:

with dockerd we need to detect if cross compilation is necessary. this is something we should fix upstream so we don't need this check and can full rely on --platform with buildx.
makefile has been simplified so we don't need to duplicate goals and rely only on TARGETPLATFORM.
Jenkinsfile is lighter and additionally generates static packages for additional platforms that were missing (linux/arm/v6, linux/arm/v7, linux/arm64)
adds a job matrix on GHA so we can also test its behavior on GitHub Runners.
static packages were missing for buildx against windows and darwin platforms.

Signed-off-by: CrazyMax crazy-max@users.noreply.github.com

crazy-max · 2022-03-29T13:21:35Z

moby engine cross compilation fails for linux/arm/v6. Should be fixed upstream: https://github.com/docker/docker-ce-packaging/runs/5737440583?check_suite_focus=true#step:4:2372

#43 0.132 
#43 0.138 ---> Making bundle: cross (in /build/bundles/cross)
#43 0.139 Cross building: /build/bundles/cross/linux/arm/v6
#43 0.182 Building: /build/bundles/cross/linux/arm/v6-daemon/dockerd-0.0.0-20220329083112-174e51c
#43 0.182 GOOS="linux" GOARCH="arm" GOARM="6"
#43 76.80 # github.com/docker/docker/cmd/dockerd
#43 76.80 loadinternal: cannot find runtime/cgo
#43 76.80 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
#43 76.80 gcc: error: unrecognized command-line option '-marm'; did you mean '-mabm'?

thaJeztah

Just a first glance over the changes, thought I'd post some initial "thinking out loud" comments

thaJeztah · 2022-03-30T12:03:08Z

scripts/target-platform

+
+# current arch/variant
+CUROS="linux"
+case "$(uname -m)" in


didn't xx also have a utility for something like this? (wondering if we can somehow centralise this effort)

yes there is the xx-info one and we should use it in a follow-up when we will be able to be sandboxed inside a Dockerfile. See #665 (comment)

Wouldn't it be better (and a lot simpler) to use go env GOARCH / go env GOOS here? 😇

If we don't get to assume access to go, maybe something like docker version --format '{{ .Server.Os }}/{{ .Server.Arch }}'?
(not sure how we reliably get "current variant" but that's pretty complicated/a guess regardless, so letting the auto-detection default to v7 seems pretty sane IMO 🙈)

thaJeztah · 2022-03-30T12:05:57Z

Jenkinsfile

    if (arch == 'armhf') {
-        // Running armhf builds on EC2 requires --platform parameter
-        // Otherwise it accidentally pulls armel images which then breaks the verify step
-        platform = "--platform=linux/${arch}"


this no longer needed? was slightly wondering if we must do the reverse and always explicitly specify platform 🤔

We can't with moby atm unfortunately. only cli, buildx supports it without effort. hence #665 (comment). if it's fixed on moby we can remove the docker_engine_cross logic in build-static script.

Makefile

static/Makefile

static/build-static

static/README.md

thaJeztah

LGTM

tianon

@thaJeztah asked me to take a look -- had more comments than I expected, but nothing super major 😅

tianon · 2022-06-29T18:06:26Z

scripts/target-platform

+
+# current arch/variant
+CUROS="linux"
+case "$(uname -m)" in


Wouldn't it be better (and a lot simpler) to use go env GOARCH / go env GOOS here? 😇

If we don't get to assume access to go, maybe something like docker version --format '{{ .Server.Os }}/{{ .Server.Arch }}'?
(not sure how we reliably get "current variant" but that's pretty complicated/a guess regardless, so letting the auto-detection default to v7 seems pretty sane IMO 🙈)

scripts/target-platform

tianon · 2022-06-29T18:45:00Z

static/build-static

+	[ -d "${ENGINE_DIR:?}/bundles" ] && rm -r "${ENGINE_DIR:?}/bundles"
+	(
+		cd "${ENGINE_DIR}"
+		mkdir -p autogen # FIXME: remove when https://github.com/moby/moby/pull/43431 merged


That PR is merged -- does that mean this line can go away now? 👀 (or does it need to point somewhere else for the FIXME now? 😅)

Yes for master or 22.06+ but not with 20.10 😣. Can update the comment though.

We don't need to take 20.10 into account (we use the 20.10 branch for that)

static/build-static

tianon · 2022-06-29T18:54:39Z

static/build-static

+scanBuildDir="${buildDir}/docker-scan"
+
+# create docker-container builder
+docker buildx inspect | grep -q 'Driver: docker-container' || docker buildx create --use


😬

This line has several assumptions baked into it and changes the user's default buildx builder -- at the very least, I think we should probably be explicit about --driver docker-container on the create command, right? (so that it explicitly matches what we checked for with inspect)

Yes indeed good point

static/build-static

crazy-max · 2022-06-29T19:47:00Z

Thanks a bunch for the review @tianon!

Wouldn't it be better (and a lot simpler) to use go env GOARCH / go env GOOS here? 😇

If we don't get to assume access to go, maybe something like docker version --format '{{ .Server.Os }}/{{ .Server.Arch }}'?
(not sure how we reliably get "current variant" but that's pretty complicated/a guess regardless, so letting the auto-detection default to v7 seems pretty sane IMO 🙈)

A bunch of the os, arch and variant detection logic is taken from xx which has bats tests to cover these cases. This project is used as cross comp helper in some of our repos. See https://github.com/tonistiigi/xx/blob/master/base/xx-info to have a better understanding of the logic. Maybe it would be better to fetch the scripts instead of just copying part of its logic. Will improve that and see if we can get rid of scripts/target-platform.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah · 2022-07-02T10:10:16Z

So, looks like something's still broken; the cross-compiled versions are missing the containerd binaries. It looks like --target=cross in the moby repository does not include them in the bundles directory (need to check why)

docker buildx build \
    --build-arg CGO_ENABLED= \
    --build-arg CONTAINERD_VERSION \
    --build-arg CROSS=true \
    --build-arg DEFAULT_PRODUCT_LICENSE \
    --build-arg DOCKER_CROSSPLATFORMS=linux/arm64 \
    --build-arg PACKAGER_NAME \
    --build-arg PLATFORM \
    --build-arg PRODUCT \
    --build-arg RUNC_VERSION \
    --build-arg VERSION=22.06.0-beta.1 \
    --output ./bundles \
    --target cross .

tree bundles
bundles
└── cross
    └── linux
        └── arm64-daemon
            ├── docker-proxy -> docker-proxy-22.06.0-beta.1
            ├── docker-proxy-22.06.0-beta.1
            ├── docker-proxy-22.06.0-beta.1.md5
            ├── docker-proxy-22.06.0-beta.1.sha256
            ├── dockerd -> dockerd-22.06.0-beta.1
            ├── dockerd-22.06.0-beta.1
            ├── dockerd-22.06.0-beta.1.md5
            └── dockerd-22.06.0-beta.1.sha256

3 directories, 8 files

thaJeztah · 2022-07-03T13:29:52Z

I dug a bit further, and currently;

the cross target on moby/moby only cross-compiles the docker daemon. The confusing bit was that containerd and runc are also built as part of the cross target, but not cross-compiled
I could make containerd cross-compile by passing the right env-vars, but runc looks to need a bit more work

So, I'll have to dig further to see how feasible that is (given that we hope to build both as part of containerd packaging soon, so it would be temporary).

For platforms that we have machines for in Jenkins this won't be an issue (linux/amd64 and linux/arm64), but for other platforms that may be an issue (arm32 variants including)

crazy-max · 2022-07-03T16:23:25Z

@thaJeztah moby/moby#43529 would fix this

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkins may set parameters to an empty value, in which case the build-args may be overriding the default value with an empty value. This patch explicitly unsets variables if they're empty (or not set). Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

This splits the CLI, Engine, and containerd packages to allow downloading the cli separate from the daemon, as well as (in future) allowing us to do a containerd release without also requiring an engine release. With this patch: make REF=v22.06.0-beta.0 VERSION=v22.06.0-beta.0 TARGETPLATFORM=linux/amd64 static static/build ├── bundles-ce-static-linux-x86_64.tar.gz └── linux └── amd64 ├── containerd-1.6.4.tgz ├── docker-buildx-plugin-0.8.2.tgz ├── docker-cli-22.06.0-beta.0.tgz ├── docker-engine-22.06.0-beta.0.tgz ├── docker-compose-plugin-2.6.1.tgz ├── docker-rootless-extras-22.06.0-beta.0.tgz └── docker-scan-plugin-0.17.0.tgz 2 directories, 8 files ls -lh static/build/linux/amd64/ total 215208 -rw-r--r-- 1 sebastiaan staff 31M Jun 29 00:21 containerd-1.6.4.tgz -rw-r--r-- 1 sebastiaan staff 14M Jun 29 00:21 docker-buildx-plugin-0.8.2.tgz -rw-r--r-- 1 sebastiaan staff 8.2M Jun 29 00:21 docker-cli-22.06.0-beta.0.tgz -rw-r--r-- 1 sebastiaan staff 19M Jun 29 00:21 docker-engine-22.06.0-beta.0.tgz -rw-r--r-- 1 sebastiaan staff 8.8M Jun 29 00:21 docker-compose-plugin-2.6.1.tgz -rw-r--r-- 1 sebastiaan staff 19M Jun 29 00:21 docker-rootless-extras-22.06.0-beta.0.tgz -rw-r--r-- 1 sebastiaan staff 4.4M Jun 29 00:21 docker-scan-plugin-0.17.0.tgz Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

GHA currently only has x86 machines, and the "cross" target in moby does not include containerd and runc. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

crazy-max force-pushed the fix-static-pkgs branch 9 times, most recently from 826baa6 to a93d12b Compare March 29, 2022 13:18

crazy-max force-pushed the fix-static-pkgs branch 6 times, most recently from fad8dbf to c34fe89 Compare March 29, 2022 21:42

crazy-max marked this pull request as ready for review March 29, 2022 22:41

crazy-max requested a review from thaJeztah March 29, 2022 22:41

crazy-max force-pushed the fix-static-pkgs branch 6 times, most recently from 10552c8 to 950306c Compare March 30, 2022 10:47

thaJeztah reviewed Mar 30, 2022

View reviewed changes

crazy-max force-pushed the fix-static-pkgs branch 3 times, most recently from 7b13168 to 4be12d7 Compare March 30, 2022 20:07

crazy-max mentioned this pull request Apr 9, 2022

fix cross compilation for arm platforms moby/moby#43474

Merged

crazy-max force-pushed the fix-static-pkgs branch from 4be12d7 to 843a6ee Compare April 14, 2022 19:20

thaJeztah previously approved these changes Jun 29, 2022

View reviewed changes

thaJeztah changed the title ~~fix and refactor static packages~~ [master] fix and refactor static packages Jun 29, 2022

tianon reviewed Jun 29, 2022

View reviewed changes

crazy-max dismissed thaJeztah’s stale review via b742291 June 29, 2022 21:19

crazy-max requested a review from thaJeztah June 29, 2022 22:07

thaJeztah mentioned this pull request Jun 30, 2022

Dockerfile: use consistent format for CONTAINERD_VERSION moby/moby#43757

Merged

script: target platform

00cfedc

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah mentioned this pull request Jun 30, 2022

[master] static: remove target_platform and switch to xx #713

Draft

thaJeztah force-pushed the fix-static-pkgs branch 2 times, most recently from af71d24 to 59a1875 Compare June 30, 2022 17:03

crazy-max and others added 10 commits July 4, 2022 11:45

static: fix and refactor

b8b411c

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

static: create bundles

42271e9

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

scripts: remove unsupported arches

d0219dc

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

makefile: utilities

d18c75e

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

static: add compose and scan cli plugins

91e2538

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

static: shfmt

cc73008

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

gha: disable cross-compile for linux

7665657

GHA currently only has x86 machines, and the "cross" target in moby does not include containerd and runc. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: upload static artifacts

9515153

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the fix-static-pkgs branch from 59a1875 to 9515153 Compare July 4, 2022 10:57

thaJeztah mentioned this pull request Jul 4, 2022

[master] fix and refactor static packages (carry) #715

Draft

crazy-max mentioned this pull request Jul 6, 2022

[master] fix and refactor static packages with fixed cross-comp on moby #718

Draft

tianon mentioned this pull request Jul 8, 2022

Add explicit "cli" aliases and move "latest" to "dind" in 22.06+ docker-library/docker#369

Merged

crazy-max mentioned this pull request Jan 25, 2023

[master] refactor static packages #835

Open

crazy-max marked this pull request as draft January 25, 2023 13:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[master] fix and refactor static packages #665

[master] fix and refactor static packages #665

crazy-max commented Mar 29, 2022 •

edited

crazy-max commented Mar 29, 2022

thaJeztah left a comment

thaJeztah Mar 30, 2022

crazy-max Mar 30, 2022 •

edited

tianon Jun 29, 2022

thaJeztah Mar 30, 2022

crazy-max Mar 30, 2022 •

edited

thaJeztah left a comment

tianon left a comment

tianon Jun 29, 2022

tianon Jun 29, 2022

crazy-max Jun 29, 2022

thaJeztah Jun 29, 2022

tianon Jun 29, 2022

crazy-max Jun 29, 2022

crazy-max commented Jun 29, 2022 •

edited

thaJeztah commented Jul 2, 2022

thaJeztah commented Jul 3, 2022

crazy-max commented Jul 3, 2022

[master] fix and refactor static packages #665

Are you sure you want to change the base?

[master] fix and refactor static packages #665

Conversation

crazy-max commented Mar 29, 2022 • edited

crazy-max commented Mar 29, 2022

thaJeztah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max Mar 30, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max Mar 30, 2022 • edited

Choose a reason for hiding this comment

thaJeztah left a comment

Choose a reason for hiding this comment

tianon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max commented Jun 29, 2022 • edited

thaJeztah commented Jul 2, 2022

thaJeztah commented Jul 3, 2022

crazy-max commented Jul 3, 2022

crazy-max commented Mar 29, 2022 •

edited

crazy-max Mar 30, 2022 •

edited

crazy-max Mar 30, 2022 •

edited

crazy-max commented Jun 29, 2022 •

edited