Support COPY from other builds

[sheerun]

I know you are reluctant to accept changes to Dockerfile syntax, but this is important and needed feature.

This is simpler alternative for nested builds, and extension of "multiple FROM" already present in Docker.

The issues Docker currently has:

1. Builds can only be created by "inheriting" other builds, it's impossible to "compose" from other builds
   • this makes it impossible to create multi-stage build, like:
     • compiling static website in first step of build with Webpack / Jekyll (complex environment), and then packaging result in simple static server (simple environment), like nginx
     • compiling in dev environment (like go-lang program), and serving in plain busybox image (currently the recommended solution seems to be custom script that extract artifact from build with cat https://github.com/docker-library/hello-world/blob/master/update.sh)
     • people are even desperately hacking multi-FROM syntax to achieve it
2. Build sizes & layer numbers go through the roof, because final builds have usually include all development dependencies, as well as layers used for building
   • it's not possible to "drop" or "flatten" or "remove cache" of not needed layers
   • it's also impossible to depend (FROM) on another image without using all the layers it used, even if we only want small portion of it (again, inheritance vs composition)
3. It is not possible to build multiple images within one Dockerfile with shared layers
   • one could argue the solution is to split into multiple Dockerfiles, but then it's really hard to enforce dependencies between images and share layers between them
4. The current form "multiple FROM" feature is mostly useless, but it could be useful
   • "non-final images that are produced are not easily found w/o some ugly parsing of the build output. It would then also leads to people wanting to do things like add TAG commands to tag each image so they're easily used/found. "
   • The solution of tagging images was proposed multiple times (and me) and rejected without good reason
     • @shykes is concerned about "polluting global space of tags", but they can be local to Dockerfile. no need to automatically publish them unless told explicitly with -t flag
     • @tiborvass closed Add TAG command to Dockerfile #5603 because of the same reason. Again "ubuntu" can be a local-only tag and only exposed if explicitly told with -t ubuntu:ubuntu or similar syntax
     • Also from time to time someone mentions "no real use cases" or "it's not defined problem", or "we don't accept new keyword in Dockerfile" and closes issue..

Those are all real issues with docker build with use cases, and they could be all solved by three simple modifications to Dockerfile that don't sacrifice repeatability of builds:

1. Allow to TAG each layer in Dockerfile (only locally)
2. Allow to COPY from any tagged layer within Dockerfile
3. Allow to explicitly export local tags with -t flag of docker build

---

To give an example:

FROM dockerfile/nodejs
ADD ./ /app
WORKDIR /app
RUN npm install
RUN npm run build
TAG development

FROM dockerfile/nginx
COPY development:/app /var/www/html
TAG runtime # not really necessary

Then you can export both runtime and development image with:

docker build \
  -t sheerun/app_development:dev_host_tag:development \
  -t sheerun/app_development:run_host_tag

It would create sheerun/app_development:dev_host_tag host image from development local tag, and sheerun/app_development:run_host_tag host image from runtime local tag (by default we export last FROM from Dockerfile, so this change is backward compatible). development must be explicitly exported, otherwise it's ignored.

Please don't close this without a good reason, and without addressing 4 issues I mentioned 💙

[phemmer]

This sounds like #7115.
You mention nested builds in your description, but linked to a different issue, so I'm not sure if you're aware of #7115. It would be good to put your thoughts on that one.
I personally don't see much difference between the two.

[sheerun]

Actually it's the second link I've posted.

I think #7115 is more invasive in syntax (blocks, 2 keywords), it is hard to understand, doesn't solve state of multiple FROM, and encourages coupling from parent to child container (not the other way around).

[phemmer]

I still think it would be a good idea to post your comments there. The purpose of a proposal is to iron out any issues with its design. Creating a new proposal to change the design makes the issue hard to track.
It can also result in your suggestions getting missed as people follow that issue, and don't see your new proposal.

[sheerun]

I want to hear from docker maintainers before inviting people here for +1 comments. I'll link to in in future.

[pdxjohnny]

+1 I also have the problem of needing to compile for a build to work. I took this idea from docker machine, what I do is have another dockerfile just to build a compiler then mount the source in it. Then I have my main dockerfile to put the binary in a container. This makes it impossible to do automated builds unless I want to leave the whole source and compiler in the production container. Well ok not impossible but it would be very messy in comparison to something like this.

[jimmycmh]

+1 my use case is to deploy the same binaries to different clusters with different parameters. i can build separate images for each cluster, but it's hard to maintain consistency between images.

[jessfraz]

@phemmer has been around the project a long time and we all appreciate this feedback on issues and review of PRs, I agree with what he said.
Also sorry the docker file syntax is indeed frozen but not for long once we split out the builder which is an on going effort and should be completed soon.

[vbatts]

and sounds like #4933

[unclejack]

As @jfrazelle has stated above, the builder will be split out and that will enable far more than nested builds. In my opinion, that will allow a whole lot more experimenting and trying out new things, rather than attempt to add it all to Docker itself.

[sheerun]

#4933 has been closed because you decided to implement:

1. Proposal: Nested builds #7115 that has disadvantages over this solution, I explained in this comment
2. Squash build dependencies #6906 that is closed because "We are no longer accepting patches to the Dockerfile syntax"

I hope the "builder" will allow to such use case (i.e. extracting stuff from images and putting them somewhere else). By not fixing current Dockerfile syntax you sadly force people to use terrible hacks.

[sleaze]

+1 for docker multiple inheritance functionality

[sheerun]

@jfrazelle @thaJeztah @unclejack What's the status of the builder? Could you reconsider this feature?

[thaJeztah]

This is now implemented in the multi-stage builds (implemented in #31257, #32063, and #32496);

FROM node AS build-env
ADD ./ /app
WORKDIR /app
RUN npm install
RUN npm run build

FROM nginx AS prod-env
COPY --from=build-env /app /var/www/html

Using that Dockerfile, running this command would only perform the steps up to the build-env stage, and tag it as development;

docker build -t development --target build-env .

Whereas omitting the --target would build all stages, and tag the last stage as production

docker build -t production .

I'll close this issue, because I think this is addressed, but let me know if there's still something missing 👍