Separate out docker image build and deployment steps into an independent command line tool

[popoludnica]

Container image build and deployment is considered a separate process from container runtime in many enterprise environments.
It would be super-cool (liberating and sensible) if docker image build was independent of docker-engine, docker file-system storage and didn't require any special privileges on a build host.

It can be solved in a form of a command line tool responsible for the following:

• building new image layers from provided content,
• building new image manifest referencing parent layers and new layers from the step above,
• uploading new layers/blobs to the registry repository,
• mounting or uploading parent layers to the registry repository,
• uploading the new manifest to the registry

Sometimes an example is worth a thousand words:

imagebuild [PARENT]... [LAYER]... --name=IMAGE
PARENT: <IMAGE>
LAYER: <uri>
IMAGE: <registry>/<reponame>[:<tag>]

I can arrange for contribution of the code if there's any interest? Proof of concept of the above described tool already exists.

Image build split out mentioned: #18596
Partial overlap of requirements: #18585, #17982, #7115

[orent]

As a thought experiment, here is how building an image from a Dockerfile might work on the client side using an unmodified daemon. It uses the current remote API, but without the /build command.

The client-side builder executes dockerfile steps one by one. The result of each step is a new json state in the same format that can be uploaded to "PUT containers/*/create". The json state is self-contained except for any images and layers it references that are kept by the daemon.

FROM: Calls POST images/*/?fromImage to ensure the image is downloaded and resolve it to specific hash.

COPY/ADD: - create temporary container, PUT /archive, commit, save image hash, remove temporary container.

RUN: create temporary container, exec, commit, save hash and remove temporary container.

ENV/MAINTAINER/ENTRYPOINT/many other commands: just locally modify the json state without connecting to daemon.

The correctness of this client-side builder can be verified by ensuring that the resulting image history is identical and has the same hashes as that produced by the server-side build process.

Modifications to the daemon can help streamline this, without so many separate commands creating and deleting temporary containers, but I think this thought experiment demonstrates it is already possible right now.

[thaJeztah]

Although it was originally planned to move the "builder" to the client, side (also see https://github.com/docker/docker/blob/v1.11.0/ROADMAP.md#22-dockerfile-syntax), development in that area was stopped. However, all required changes to the API are implemented, so you should be able to create a "custom" builder that works from the client, for example, see https://github.com/jlhawn/dockramp

[thaJeztah]

I guess we can mark this as solved, with;

• https://github.com/moby/buildkit
• https://github.com/docker/buildx