[epic] make BuildKit the default builder on Linux

[thaJeztah]

BuildKit is planned to replace the classic builder and has been available as an opt-in feature since Docker 18.06. BuildKit provides many enhancements over the classic builder, fixes bugs that cannot be fixed (or cannot easily be fixed) in the classic builder, and is overall recommended to be used if supported on the user's platform.

So far, BuildKit was still opt-in, which makes it harder to use (not all users are even aware of the new builder), but also inconsistent with our recommendation.

I'm opening this issue to discuss the option to make BuildKit the default builder in the next (docker) release.

Proposal: make BuildKit the default builder on Linux

• Toggle the default: if no environment variable or daemon configuration is set, BuildKit will be used by default
• Document the classic builder to be "in maintenance":
  • Bugs can be fixes, but within limits (no extensive modifications)
  • Feature-parity / enhancements can be made, but should (where possible) be limited to fixes that affect Windows (which does not yet have BuildKit support)
• Users are still able to disable BuildKit (DOCKER_BUILDKIT=0 or the equivalent daemon.json configuration)
• Update the documentation to document new features (where needed)

Why BuildKit is not yet the default?

We made BuildKit an opt-in feature get a wider audience for testing the new builder in many scenarios, allowing us to catch important regressions without breaking existing users. While stability and regressions have been addressed, and we're confident that BuildKit is a good replacement for most scenarios, there were still some things left to be done:

• No support on Windows. For BuildKit to be used on Windows, additional changes are still needed (e.g., changes to the runtime, using containerd). Work on this is still in progress, but there is no definitive ETA.
• Feature parity. The classic builder received some advanced options over the years; not all of those are implemented yet in BuildKit, and for some of those, no decision has been made yet if they will be implemented (or implemented in different ways). See the "Feature parity: (advanced) options supported by BuildKit" section below with more details.

Even though both of the above still hold true, I think it makes sense to switch BuildKit to be the default:

• Linux and Windows will never have full feature parity. They're different platforms, and not all things possible on Linux will be possible on Windows (and vice-versa). There are already differences between both platforms, and having a different builder will not substantially change that situation. (It should be properly documented though)
• Most of the options that are not (yet) supported are for "advanced" use-cases, and won't be used by most users. While switching the default to BuildKit would affect users of those advanced options, not switching is probably worse: users don't get the best experience "out of the box".

Prerequisites

• Review the missing features and changes in behavior between BuildKit and the classic builder
• Decide if any of those options should be considered a "blocking" option.
• Blocking issues: estimate if they can be fixed/addressed before the next release
• Non-blocking issues: update the documentation (and prepare release notes) to describe differences
• Create tracking issues for missing options that should be implemented in future (roadmap)

Feature parity: (advanced) options supported by BuildKit

Note: I also added a area/feature-parity label in the BuildKit repository to make it easier to find issues and pull requests related to feature-parity with the classic builder;
https://github.com/moby/buildkit/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%2Ffeature-parity

• Debugging is more complicated with BuildKit, because no "intermediate containers", nor "intermediate images" are created, therefore it's not (easily) possible to debug failures in build by starting a container from the intermediate step.
  • document steps / workarounds / recommendations on how to debug build failures when using BuildKit
  • create a plan/roadmap on how this can be improved in future (created: see Running containers from BuildKit results without images #40887)
• Build progress is printed on stderr instead of stdout (see [RFC] Add some output to logs for backward compat buildkit#1155). This is a bugfix / by design, but is a change in behavior that should be called out (changelog?)
  • document change in behavior
  • consider adding Successfully built, Successfully tagged output ([RFC] Add some output to logs for backward compat buildkit#1155)
• BuildKit does not use intermediate images for caching, but uses a dedicated build-cache. While this has many advantages, it also makes it less discoverable where disk-space is consumed by Docker (many users are familiar with "the old way", and remove dangling images to clean up disk space)
  • consider enabling BuildKit garbage-collection by default
  • improve documentation around managing build-cache and garbage-collection (related: builder: use buildkit's GC for build cache #37846)
  • (for Docker Desktop) consider adding configuration options in the Docker Desktop UI to make the garbage-collection options better discoverable, and to make it easier to configure.
• BuildKit doesn't handle Dockerfile from stdin, when using remote context #38254 BuildKit doesn't handle Dockerfile from stdin, when using remote context
• Add support for subdirectories when building from git source buildkit#1684 Add support for subdirectories when building from git source (implemented in git: support subdir component buildkit#2116)

List of options, and their status in BuildKit

Note that the current CLI always shows all options (even if not supported), but docker/cli#2123 added annotations to hide unsupported options (currently only on master)

Looking at these options:

• Some of these options may need discussion wether or not they will be supported by BuildKit, or should be marked "deprecated".
• Decide if any of these options are a "blocker" for switching the default builder, or a "nice to have" and are ok to be implemented in future updates. Note that users are still able to opt-out by setting DOCKER_BUILDKIT=0, or disabling BuildKit to be the default in the daemon configuration

Status	Option	Notes/Tracking
✅	--add-host	-
✅	--build-arg	-
✅	--cache-from	-
✅	--cgroup-parent	moby/buildkit#2430, docker/buildx#814
❓	--cpu-period	-
❓	--cpu-quota	-
❓	-c, --cpu-shares	-
❓	--cpuset-cpus	moby/buildkit#1276
❓	--cpuset-mems	-
✅	--disable-content-trust	-
✅	-f, --file	-
✅	--force-rm	no-op: not needed for BuildKit. Needs to be documented(?)
✅	--iidfile	-
❓	--isolation	windows-only (should be hidden?)
✅	--label	-
❓	-m, --memory	-
❓	--memory-swap	-
❓ 🚧	--network	only none and host, but no custom networks see moby/buildkit#978
✅	--no-cache	-
✅	-o, --output	(BuildKit-only feature)
✅	--platform	-
✅	--progress	(BuildKit-only feature)
✅	--pull	-
✅	-q, --quiet	-
✅	--rm	no-op: not needed for BuildKit. Needs to be documented(?)
✅	--secret	(BuildKit-only feature)
❓	--security-opt	-
✅	--shm-size	moby/buildkit#2384, docker/buildx#790, docker/buildx#418
🚧	--squash	Partial support: moby/buildkit#819 #38903 #39187 DOCKER_BUILDKIT=1 docker build --squash should not squash the base image #38903. This feature is still "experimental" so should not be a blocker
✅	--ssh	(BuildKit-only feature)
✅	-t, --tag	-
✅	--target	-
✅	--ulimit	moby/buildkit#2398 / docker/buildx#800

[thaJeztah]

@nebuk89 @tonistiigi @AkihiroSuda @tiborvass

[AkihiroSuda]

Maybe we can start with adding "BuildKit ad" message to the legacy docker build CLI?

[thaJeztah]

✅ Linking #40335 which looks to be a regression/difference? resolved

[tonistiigi]

The leftover items listed here are in 3 big categories. WCOW support, setting cgroups per build request and debugging with intermediate images.

WCOW - I don't think anyone is actively working on it. There has been some interest in the community (moby/buildkit#1314 ) and MSFT but no actual code to get over the first roadblocks, eg. containerd/containerd#2366 for completing snapshot API in containerd so it has feature parity with what Docker graphdrivers can do.

Cgroups flags (--cpu-period, --cpu-quota) don't really fit well to BuildKit design that enforces deduplication of parallel workloads and maintains possibility to do distributed builds. Eg. if there is a 500MB memory limit on a running build and then a new request comes in with no memory limit that has the same graph, we end up in a tricky situation. The same thing appears where a limit is set, but the build gets distributed to multiple nodes. It is unclear if there is a good way to fit these limits into the design or if setting cgroups would be something that you configure per worker in BuildKit. If any other unsupported flag is not directly a cgroup, we think support for it should be removed(eg --security-opt is replaced with entitlements model in BuildKit).

For intermediate images, we do not plan to support them because of performance and storage management issues. We do want to replace this with a builtin debug capabilities that would be much helpful than messing with intermediate images anyway.

Two other important considerations:

BuildKit was always designed with a dependency on the containerd storage libraries, and they have still not been merged to Moby. This means that the current implementation is incomplete and uses hacky adapters to mock some containerd features with older Moby libraries. Things that do not work in Moby are multi-platform images support, remote cache export from cache manifest in the registry, remote cache from client-side files, some exporters (eg. oci, registry). The instruction cache chain is a bit different for images in Moby because we can't access the digest of the image manifest. Making it default before finishing the containerd work does look like cutting corners with a lower quality end product.

The other aspect is that I think the default build experience in Docker shouldn't be equivalent to DOCKER_BUILDKIT=1 docker build . but docker buildx build . https://github.com/docker/buildx . It doesn't make much sense in developer flow to have the builder attached to the Docker daemon process that is quite hard to upgrade. Buildx comes with the possibility to always run the newest builder, with the configuration you like (moby, container, k8s), create namespaced instances of builders, and do distributed builds. All this while keeping the default experience identical to the current docker build and not requiring extra setup or relearning the UX. You can already switch the default builder to buildx today by running docker buildx install. There are also possible maintenance improvements from not needing to connect everything back to cli/dockerd. Unfortunately, the current default experience suffers from the same issues of incomplete containerd integration mentioned earlier. Resulting in behavior where not all buildx drivers are functionally equivalent, and limitations like although you can build a multi-arch image with buildx, you get an error if you try to load it into docker. Long-term, I do think buildx is the right default but not sure if we want to switch the default in every release, forcing users to learn new concepts again.

[bryanhuntesl]

Status .	Option	Notes/Tracking
	--network	only none and host, but no custom networks see moby/buildkit#978

This is going to cause lots of pain, lots of projects use the trick of accessing hosts during docker build time in order to run database tests , use apt-cache to speed up dependency fetching, and lots of other things. You can't do this without running a custom network.

[thaJeztah]

Added #40887 to the top description

[abdennour]

it was the default on my mac now..
But when i used ubuntu machine, it's not the default.

[velovix]

I really appreciate the Docker team being so careful during this migration to BuildKit. I'm sure supporting both build systems has a significant maintenance cost.

[karfau]

Is there a reliable way to detect if BuildKit is active?
I know it can be activated using /etc/docker/daemon.json (on linux) or DOCKER_BUILDKIT=1.
docker info doesn't seem to provide any information about it.

Since building an image that requires it for mounting build cache on RUN commands, when it's not active gives a syntax error that is not helpful...

Please let me know if I should create a new issue/ask elsewhere for that topic, but I thought this question could be relevant for this topic and I didn't find any answers/similar questions when searching.

[dm17]

These are currently blockers for me; I need build time networking to work in docker-compose. There's a bunch of complexity around why some of us need networks besides node, host, and default attached to a container that's building (defined in a docker-compose.yml in my case).
docker/buildx#175
moby/buildkit#978

[tekumara]

As of v23.0.0 BuildKit is the default builder on Linux.

[dboreham]

As of v23.0.0 BuildKit is the default builder on Linux.

Is there a good way to disable it (since it's not compatible with all existing use cases) through the command line?

[dboreham]

As of v23.0.0 BuildKit is the default builder on Linux.

Is there a good way to disable it (since it's not compatible with all existing use cases) through the command line?

I discovered DOCKER_BUILDKIT=0 documented in https://github.com/moby/moby/releases/tag/v23.0.0 .
Posting here since Google brings you to this page when everything breaks due to buildkit being enabled as the default.

[crazy-max]

everything breaks due to buildkit being enabled as the default

@dboreham Would you mind posting a repro with a Dockerfile and build command so we can look at your issue with BuildKit?

[dboreham]

everything breaks due to buildkit being enabled as the default

@dboreham Would you mind posting a repro with a Dockerfile and build command so we can look at your issue with BuildKit?

It turned out to be this issue: docker/buildx#1832

[dboreham]

fwiw I have also had problems in the past with BuildKit on ARM, I think to do with the container images it depends on (BuildKit's images) not published for that arch. I can try to reproduce that one again if you're interested.

[crazy-max]

fwiw I have also had problems in the past with BuildKit on ARM, I think to do with the container images it depends on (BuildKit's images) not published for that arch. I can try to reproduce that one again if you're interested.

If you're talking about the container builder, here is the list of supported platforms: https://github.com/moby/buildkit/blob/db6342f9fa9b69d0da517d776e619b8de221de2f/.github/workflows/buildkit.yml#L28

For the Dockerfile frontend it's here: https://github.com/moby/buildkit/blob/db6342f9fa9b69d0da517d776e619b8de221de2f/.github/workflows/frontend.yml#L26

Wonder if in your case arm/v6 is the one missing. Please let us know.

• improve documentation around managing build-cache and garbage-collection (related: builder: use buildkit's GC for build cache #37846)

@thaJeztah If you want to update issue description with: https://docs.docker.com/build/cache/garbage-collection/

[dm17]

As of v23.0.0 BuildKit is the default builder on Linux.

Is there a good way to disable it (since it's not compatible with all existing use cases) through the command line?

My point isn't that it is difficult to disable, just that BuiltKit should support build-time networking (not sure of the official term, but that's how I refer to moby's fuctionality allowing networking during builds, using compose especially).

[dm17]

Status . Option Notes/Tracking
--network only none and host, but no custom networks see moby/buildkit#978
This is going to cause lots of pain, lots of projects use the trick of accessing hosts during docker build time in order to run database tests , use apt-cache to speed up dependency fetching, and lots of other things. You can't do this without running a custom network.

Agreed @bryanhuntesl
What did you end up doing for this? So no way to have custom networks and docker secrets throughout build time. Also no way to have build features of compose+buildkit while also having access to custom networks. Is host networking really so popular that it doesn't make sense to support defined or parameterized networks?

[dboreham]

Agreed @bryanhuntesl What did you end up doing for this? So no way to have custom networks and docker secrets throughout build time. Also no way to have build features of compose+buildkit while also having access to custom networks. Is host networking really so popular that it doesn't make sense to support defined or parameterized networks?

Not parent, but the approach we've taken in my team is to simply disable buildkit. So now we get to see a loud warning telling us how stupid we are for not using buildkit, but at least our product still works. I'm assuming that eventually either buildkit will be fixed to have feature parity with "docker classic", or one of the open source clones/forks of docker will emerge as a usable alternative.

[dm17]

Agreed @bryanhuntesl What did you end up doing for this? So no way to have custom networks and docker secrets throughout build time. Also no way to have build features of compose+buildkit while also having access to custom networks. Is host networking really so popular that it doesn't make sense to support defined or parameterized networks?

Not parent, but the approach we've taken in my team is to simply disable buildkit. So now we get to see a loud warning telling us how stupid we are for not using buildkit, but at least our product still works. I'm assuming that eventually either buildkit will be fixed to have feature parity with "docker classic", or one of the open source clones/forks of docker will emerge as a usable alternative.

This release marks the beginning of the deprecation cycle of the classic (“legacy”) builder for Linux images. No active development will happen on the classic builder (except for bugfixes). - https://docs.docker.com/engine/deprecated/

I would hate for my build system to become entrenched with deprecated mechanisms. How many people with complex container systems or proxies for anything public facing use host networking mode - or none for that matter - the only two supported by buildkit?