New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend docker cp to permit copying from images #16079
Comments
This would make images mutable objects, and the whole point of images is to be immutable. I'm pretty 👎 on this, but I'd like to hear what other people think about it, specially @jlhawn. |
@calavera I think he wants to just copy stuff from images, not into them - so images would still be immutable. Its an interesting feature - I'd like to hear more about the usecases for it though. |
You can already get the desired behavior by simply creating a container using the image first.
|
As mentioned in the request, Although initially proposed to fail when the target argument referred to an image, what if
The naming conventions for the new image would assume the semantics applied by the In other words, a |
Not always. Strictly speaking Also, |
This proposal, especially with the enhanced semantics permitting the
|
I'm not hearing a lot of traction on this one. I think given the relatively easy solution that @jlhawn mentioned (just create a container - even if you have to give it a dummy command to run - and then use |
Since I found it useful, the semantics of creating/evolving images by simply copying files were incorporated into a bash script called |
+1 Use case: im am building my application with docker and then i wanted to extract the binaries to outside. actual solution: create a dockerfile, build it, and then run a container only for a few seconds to extract with docker cp. another proposal: add to COPY functionality to extract from image that is being builded to working folder in host. |
I'm looking for this exact feature to implement a two phase docker build (see #7115):
to articulate those, I need to cp binary from image build on step 1. Ability to run |
Just in case it went unnoticed, there's a bash script named |
@WhisperingChaos
Cheers! |
@mercuriete Enjoy! |
As shown here you could run cat or tar:
|
@wedesoft this assumes the image has |
Now that we have an official way to run Docker on a Raspberry Pi I wold like this to be re-opened. We have a use case where we have software running on PCs and RPis. Given that they have different architecture I CANNOT simply start a container using our image. However there are some files in the image I would like to extract so we do not have to create multiple (PC and RPi) versions of all of our containers. |
Unfortunately, one of my earlier posts suggested that dkrcp operated on running containers to implement the copy operation. It doesn't. Instead, it executes I've been using dkrcp with Docker 1.12, within an AMD64 Ubuntu VM to generate minimal images containing a golang executable targeted for the Pi. Lastly, if you encounter a problem with dkrcp, let me know and I'll address it. |
this feature is available when used to build a Dockerfile as multi-staged build, seems to me it could be exposed the the CLI as well. |
@thaJeztah you linked to the thread we are already in. I'm interested to see where you intended to link :) |
Oh, lol, wrong link on my clipboard; meant to link to #30449 |
I like this method :D |
This assumes |
No need to use docker create --name foo <image-name>
docker cp foo:/path/to/file file_from_image
docker rm foo |
@binarytemple, I agree. Another way to do this is to simply piggy-back off of container IDs by dropping the IMG_ID=$(docker create <image-name>)
... |
@rbi13 - Thanks for the pointer, I just thought I'd share my Makefile workaround for building parameterized images and copying out build artifacts. It's pretty ugly (and completely untrustworthy (shell exploit anyone?)) , but works for now.
|
I'm concerned that this multi-step process is quite a learning curve for someone getting started with docker. It takes research. Everyone's doing it differently. Docker is already hard to learn. Why not make it easier for newcomers? What's the downside? I'm not convinced that the reasons for closing this ticket are adequate enough to discount the potential benefit to the community. I strongly advocate reopening this issue. |
My use case is for Tomcat8(Java J2EE server) base image modification for the new application. I would like to see a simple command to extract file from the image, modify file, use it in a dockerfile script file.
|
Docker's relatively new multi-stage build feature obsolesces the need for this enhancement when using |
I was reaching for the same thing (in my case to add a userspace so I could It essentially does:
|
Multi-stage builds obsolete this feature for the purpose of copying in to an image (although it seems impossible to do so and also copy files to /). However, multi-stage builds definitely do not obsolete the need for this feature for copying out of an image. The image continues to be immutable for the purpose of copying a file out. Note, 'docker export' also does not work for copying out of an image. Considering the following file, 'hello.txt':
Consider the following Dockerfile:
Build an image from the above Dockerfile.
How you you suggest one should extract the file hello.txt from the image (i.e. not a running container)?
My solution was the following, which is a complete pain in the ass, and could potentially have unintended consequences. I create a stopped container from the image, which has an entry point of "/" (obviously running the container will fail). I then extract the file from the stopped container, and delete the container.
IMHO, creating a stopped container to just to copy files out of an image is a waste of resources, but that is just my HO. |
Copying from an image directly likely would do the same, as the images filesystem still has to be mounted before being able to copy |
I use the following command to copy single file to a local disk, it creates a non-running container from the image file, copies a file and then delete the named temp container. This trick is not my discovery but here and also Stackoverflow helped me to do it.
Copying all files from the image I use similar command sequence.
Not very intuitive way to achieve a simple task but once you know this black magic trick you move on to the next challenges in life. |
Extra steps. I guess it just means life is less convenient for users :P |
I think it will make sense copying file to a volume. Use case can be a multi stage build that check out a code build it into an image and just like the COPY --from=build src dest functionality i should be able to do some like:
|
@duglin @thaJeztah @tonistiigi I understand this can be done by combining If adding this feature is unacceptable for |
@AkihiroSuda discussing with @tonistiigi - if we understand the use-case correctly, the primary use would be to copy build artefacts from an built-image to the host. Given that BuildKit would allow exporting the artefact directly on the host, perhaps we should have a look at that (i.e. integrating buildkit?) |
In addition to that, I think this proposal is useful for copying binaries from pulled images. WDYT? |
@thaJeztah I think so, my use-case would be to copy a static site out of an integration-tested image to submit to a CDN. To elaborate: We use MultiStage builds to take a website -> minify etc. -> make a minimal nginx image with the static site. This image is used for integration testing. But then when that is done the actual hosting is done by a CDN. So I want to get the static site out of the image, zip the files and submit to the CDN. But I don't want to include the CDN API / curl in the "integration testing" container and run the command inside the container. |
I'll add one more use case for that. We use Docker multi-staged build to build ember app, and then host it under nginx container. So we have two build steps. But we also running QUnit tests for ember app inside of first build image, and we want to save unit test results on actual host, so CI agent could pick up it, parse and display test results in CI's UI. docker create\cp\rm would help here, but if |
I agree, docker must support such feature, to copy file/folder from an image into host. So, instead of currently doing (as @binarytemple suggested): IMG_ID=$(dd if=/dev/urandom bs=1k count=1 2> /dev/null | LC_CTYPE=C tr -cd "a-z0-9" | cut -c 1-22)
docker create --name ${IMG_ID} <image-name>
docker cp ${IMG_ID}:/path/to/file file_from_image
docker rm ${IMG_ID} We need to have something like: docker cpimage [docker-run-options] /etc/some-service.conf ./ <image-name>[:<tag>] Or copying an entire folder: docker cpimage [docker-run-options] /etc/some-service.d ./ <image-name>[:<tag>] Basically, the following syntax could work: docker cpimage [docker-run-options] <source-path-on-image> <dest-path-on-host> <image-name>[:<tag>] |
I agree with @slavikme. My use case involves using Docker to build a static binary that is then run on bare metal. In this way, any linux distribution supporting docker can be used to build the application. I am currently resorting to using: CID=$(docker create image-name:latest)
docker cp ${CID}:/opt/binary-name ./binary-name
docker rm ${CID} This approach, however is not optimal: apart from requiring scripting, it is prone to subtle breakages, because the three commands sequence is not atomic, and performs side effects on the machine it runs on. Having a single command that extracts one or more files from an image would really be a better solution. |
@muxator |
Hi @AkihiroSuda, thanks for the suggestion! The Docker version on my ubuntu 20.04 is 19.03.8 and thus it is compatible with your hint. The documentation at https://docs.docker.com/engine/reference/commandline/build/#custom-build-outputs mentions it, too. This is probably a case of getting lost in old material through search engines. For future reference, a minimal working example would be: FROM ubuntu:20.04 as build-stage
# placeholder for commands that perform the build and put the
# statically linked binary in <release_path>/<binary_name>
RUN <build_commands>
# Copy just the statically linked artifact in the root directory
# of an empty container
FROM scratch
COPY --from=build-stage <release_path>/<binary_name> /<binary_name> To run the build and put the generated binary in a directory, this command would then be sufficient: DOCKER_BUILDKIT=1 docker build --output <dest_path> <dockerfile_path> In general, that command would put the whole generated image contents in Thank you very much. p.s.: as a side note, it seems that the BUILDKIT subsystem is completely separated from the "normal" docker workflow. For example, the first time I run the build, docker had to re-run the whole Dockerfile (including preliminary package installs that supposedly where already in the cache) |
I agree with @sbussard for 100%.
This issue was opened on Sep 4, 2015. Right now it's Mar 17, 2021 and the discussion is still ongoing. Maybe it's time we look at it seriously. To put a summary(and plz correct me if I am wrong), essentially most of the purposed solutions are to
This works for sure, which solved my problem to grab a json file inside an image. As a docker beginner, whenever there are some simple stuff that I assume I can find in official docker documentation, I will go there firstly. If a docker beginner encounters the same issue, chances are they will check official docker documentation as well, so it would be really nice and user friendly when there are something like Another question I am curious about is, is that pretty hard to make it available for copying file from docker image to local file system? Is that because comparing to docker container, docker image doesn't have a file system? But essentially docker image is just an immutable image file with multiple layers right, why can't we read data from it diretly? |
Perhaps an example could be added to the docs; that said, it's not a very common scenario to copy files from an image (without also having a container related to that image)
Each of those layers has to be mounted to construct the final representation (e.g. if layer1 adds 2 files, layer2, removes one of those files, and layer3 updates on of those files, all three changes have to be applied on top of each other to know the final result); effectively, copying from an image would thus be the equivalent of |
I guess it depends on the "not very common scenario" but I'm copying files from an image when I'm using docker to only build the app artifact (e.g. deb file) not to run the app. Thus I can have reproductive builds using Docker even when a customer is not using docker to run the app (believe it or not there are quite some companies that don't run systems on Docker). |
You may be interested in enabling buildit ( Creating a project directory with some "source files" $ mkdir -p myproject/src/build
$ cd myproject
$ touch src/source-file1 src/source-file2 A multi-stage Dockerfile to illustrate a "build-stage" that produces a # Build-stage: builds "my-binary" from source
FROM busybox AS buildstage
COPY src src
RUN echo "I built this from src" > /src/build/my-binary
# Stage for --output; must only contain files to ship
FROM scratch AS artifacts
COPY --from=buildstage /src/build/my-binary / Now, with buildkit enabled; $ mkdir shipit
$ DOCKER_BUILDKIT=1 docker build --output=shipit .
$ ls -la shipit/
total 8
drwxr-xr-x 3 sebastiaan staff 96 Mar 17 17:42 ./
drwxr-xr-x 5 sebastiaan staff 160 Mar 17 17:42 ../
-rw-r--r-- 1 sebastiaan staff 22 Mar 17 17:42 my-binary
$ cat shipit/my-binary
I built this from source No image is created in this example (as we didn't tag it), but the BuildKit build-cache will be used for repeated builds |
Wow, this looks great! Thanks @thaJeztah! |
The |
Permit
docker cp
to copy from an image but fail when copying to an image.Perhaps a [
docker cp
] command that targets an image doesn't need to fail.Motivation.
Although declined, feature was implemented as bash git hub project
dkrcp
The text was updated successfully, but these errors were encountered: