Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"docker save" does not store image tags. #3877

Closed
ptramsey opened this issue Jan 31, 2014 · 31 comments
Closed

"docker save" does not store image tags. #3877

ptramsey opened this issue Jan 31, 2014 · 31 comments

Comments

@ptramsey
Copy link

If you call "docker save repo:tag", the resulting tarball will not contain the repo or tag, and there's no way that I can see to figure out what image is inside such a tarball, short of extracting the json and building the revision history yourself. IE, if you're handed a "docker saved" image, the only way to figure out what was loaded when you do "docker load" is to compare "docker images" from before and after.

Docker inspect should work on tarballs, and docker save should store repos and tags.

@tavisrudd
Copy link

It's inconsistent: docker save image-repo-name -> docker load restores name, docker save SHA -> docker load no names or tags, docker save name:latest -> docker load no names or tags

@tianon
Copy link
Member

tianon commented Feb 6, 2014

Since a single image ID can have multiple names/tags, I'd say the behavior of those first two is exactly what I'd expect (ie, if you've done something like docker tag ubuntu:12.04 my-private-repo/user/ubuntu:12.04, then you don't want docker save <ubuntu-12.04-id> to include "my-private-repo/user/ubuntu:12.04"), but that last one feels wrong. I think the bug there may be in docker load, but I haven't examined the resulting docker save tarballs recently enough to say for sure.

@ptramsey
Copy link
Author

ptramsey commented Feb 6, 2014

The problem is really that currently there's no way to know, after calling docker load, what was loaded. If you load an image, you can't then instantiate it, because you don't know its ID.

@tavisrudd
Copy link

The docs for docker save should be explicit about the semantics of these three cases. The second case makes sense when you think about it but my initial intuition from reading the docs was that it would find all tags associated with the SHA and save them. docker load should provide some feedback about what was loaded.

@ewindisch
Copy link
Contributor

👍 The third use-case needs to preserve the tag imho, and I agree with better output from load.

@jamtur01
Copy link
Contributor

jamtur01 commented Feb 7, 2014

+1 to Eric.

@techwolf359
Copy link

SHA should get the names and tags associated with that too. Also +1 for documentation in docker save that itemizes what gets stored. "Contains all parent layers, and all tags + versions" is misleading if by design, it does not save tags + versions.

@unclejack unclejack added the bug label Feb 26, 2014
@srid
Copy link
Contributor

srid commented Mar 8, 2014

i'm attempting to workaround this issue on my end using this script:

# docker-save.sh
#!/bin/bash -ex
# Wrapper for 'docker save' fixing,
# https://github.com/dotcloud/docker/issues/3877
# In addition: this script will always save exactly one image (possibly
# multiple tags).

IMAGE=$1
TARGET=$2
NAME=`echo $IMAGE | awk -F':' '{print $1}'`
ID=`docker inspect $IMAGE | scripts/json 0.id`
TAGS=`docker images --no-trunc | grep $ID | awk '{print $2}'`
DIR=`mktemp -d --suffix=-docker-save`
pushd $DIR

docker save $ID > $TARGET
echo "{\"$NAME\":{" > repositories
for TAG in $TAGS; do
        echo "\"$TAG\":\"$ID\"," >> repositories
done
echo "}}" >> repositories

tar -r -f $TARGET repositories

popd
rm -rf $DIR

however, docker load refuses to load a tar file created by this script (all it does is add a file called "repositories" to it):

$ sudo docker load < x.tar
2014/03/07 18:23:57 Error: load: open /tmp/docker-import-482638663/repo/.wh..wh.orph/json: no such file or directory
$

it seems to me that a tar file created by 'docker save' cannot be manipulated by GNU tar.

@srid
Copy link
Contributor

srid commented Mar 8, 2014

and the added 'repositories' file is missing for some reason. i see some strange files appended to it instead:

$ tar -t -f base.tar
./
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/VERSION
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/json
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/layer.tar
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/VERSION
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/json
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/layer.tar
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/VERSION
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/json
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/layer.tar
./
.wh..wh.aufs
.wh..wh.orph/
.wh..wh.plnk/
$

@srid
Copy link
Contributor

srid commented Mar 8, 2014

it seems to me that a tar file created by 'docker save' cannot be manipulated by GNU tar.

yup, the culprit turns out to be GNU tar (or docker's tar being incompatible with it). using Python's tarfile module works. here is the working wrapper script working around this issue: https://gist.github.com/srid/9424614

@crosbymichael crosbymichael added this to the 1.0 milestone May 15, 2014
@crosbymichael crosbymichael removed this from the 1.0 milestone May 16, 2014
@yukw777
Copy link

yukw777 commented Jul 24, 2014

Any development on this issue?

@vieux
Copy link
Contributor

vieux commented Jul 24, 2014

@yukw777 I think there is no issue /cc @vbatts @fkautz

@vbatts
Copy link
Contributor

vbatts commented Jul 24, 2014

What versions are you working with? There were inconsistent behavior, but has been fixed recently

@fkautz
Copy link
Contributor

fkautz commented Jul 25, 2014

The current (and correct) behavior is as follows:

docker save repo

Saves all tagged images + parents in the repo, and creates a repositories file listing the tags

docker save repo:tag

Saves tagged image + parents in repo, and creates a repositories file listing the tag

docker save imageid

Saves image + parents, does not create repositories file. The save relates to the image only, and tags are left out by design and left as an exercise for the user to populate based on their own naming convention.

@matthughes
Copy link

@fkautz The second one works for you? If I try and save a repo:tag, it fails to create the directory because it's trying to create a directory with a colon in it (well that's my guess):

docker save matt/centos:6.5.0.2014.07.21
2014/07/29 21:05:32 Error: mkdir /tmp/docker-export-083524343/matt/centos:6.5.0.2014.07.21: no such file or directory

@fkautz
Copy link
Contributor

fkautz commented Jul 29, 2014

I tried it in trunk and saw it work. What version are you running?
On Jul 29, 2014 5:09 PM, "Matt Hughes" notifications@github.com wrote:

@fkautz https://github.com/fkautz The second one works for you? If I
try and save a repo:tag, it fails to create the directory because it's
trying to create a directory with a colon in it (well that's my guess):

docker save matt/centos:6.5.0.2014.07.21
2014/07/29 21:05:32 Error: mkdir
/tmp/docker-export-083524343/matt/centos:6.5.0.2014.07.21: no such file or
directory


Reply to this email directly or view it on GitHub
#3877 (comment).

@matthughes
Copy link

I'm running 1.1.2. Have you tried with a namespace? Here's a reproduction:

vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker build -t mynamespace/myrepo:1.0.0 .
Sending build context to Docker daemon  2.56 kB
Sending build context to Docker daemon
Step 0 : FROM busybox
Pulling repository busybox
d200959a3e91: Download complete
37fca75d01ff: Download complete
a9eb17255234: Download complete
fd5373b3d938: Download complete
511136ea3c5a: Download complete
42eed7f1bf2a: Download complete
c120b7cab0b0: Download complete
120e218dd395: Download complete
1f5049b3536e: Download complete
f06b02872d52: Download complete
 ---> a9eb17255234
Step 1 : CMD echo "hello world"
 ---> Running in a1249d508391
 ---> 16a69a2920aa
Removing intermediate container a1249d508391
Successfully built 16a69a2920aa
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker save mynamespace/myrepo:1.0.0 > savedImage.tar
2014/07/29 23:23:52 Error: mkdir /tmp/docker-export-858462892/mynamespace/myrepo:1.0.0: no such file or directory
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker info
Containers: 1
Images: 20
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 22
Execution Driver: native-0.2
Kernel Version: 3.13.0-32-generic
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker --version
Docker version 1.1.2, build d84a070

@matthughes
Copy link

It appears that the namespace is actually the culprit, not the version. If I take the same example but build without a namespace, it works:

agrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker build -t myrepo:1.0.0 .
Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon
Step 0 : FROM busybox
 ---> a9eb17255234
Step 1 : CMD echo "hello world"
 ---> Using cache
 ---> 16a69a2920aa
Successfully built 16a69a2920aa
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker save myrepo:1.0.0 > myrepo.tar
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ ls
Dockerfile  myrepo.tar 

@matthughes
Copy link

Also, saving with namespace/tag (dropping version) does work. So to sum up:

namespace/tag:version -> does not work
namespace/tag              -> does work
tag                                  -> does work

@fkautz
Copy link
Contributor

fkautz commented Jul 30, 2014

I did not try it with a namespace. I'll give it a try.
On Jul 29, 2014 8:07 PM, "Matt Hughes" notifications@github.com wrote:

Also, saving with namespace/tag (dropping version) does work. So to sum up:

namespace/tag:version -> does not work
namespace/tag -> does work
tag -> does work


Reply to this email directly or view it on GitHub
#3877 (comment).

@vbatts
Copy link
Contributor

vbatts commented Aug 6, 2014

all the behaviour seen in master (042b642) is expected behaviour.

vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev version
Client version: 1.1.2-dev
Client API version: 1.14
Go version (client): go1.2.2
Git commit (client): 042b642
OS/Arch (client): linux/amd64
Server version: 1.1.2-dev
Server API version: 1.14
Go version (server): go1.2.2
Git commit (server): 042b642
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev images busybox
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
busybox             latest              a9eb17255234        8 weeks ago         2.433 MB
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save busybox | tar xO repositories
{"busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save busybox:latest | tar xO repositories
{"busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev tag busybox:latest namespace/busybox
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save namespace/busybox | tar xO repositories
{"namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save namespace/busybox:latest | tar xO repositories
{"namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev tag busybox:latest 0.0.0.0:5000/namespace/busybox
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save 0.0.0.0:5000/namespace/busybox | tar xO repositories
{"0.0.0.0:5000/namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save 0.0.0.0:5000/namespace/busybox:latest | tar xO repositories
{"0.0.0.0:5000/namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save a9eb17255234 | tar xO repositories
tar: repositories: Not found in archive
tar: Exiting with failure status due to previous errors

@vbatts vbatts closed this as completed Aug 6, 2014
@sroebuck
Copy link

The comment that closes this issue is unclear as to whether the current released behaviour is considered to be 'expected behaviour' or whether there is a fix to the current release that will appear in the next release (as a result of being in master).

Certainly I'm finding two issues when using the -o option to output to a specified output tar file:

  1. If the image to be saved is identified with a url that contains a / then the command issues a mkdir error trying to produce the /tmp directory to generate the intermediate files.
  2. If the image to be saved is identified with an explicit version tag then the resulting tar (when loaded using docker load) results in an untagged image being loaded.

I would have expected that saving an image with an explicit version would save only that image and its associated layers. When saving an image without an explicit version it is probably beneficial that it saves all of the versions and their associated layers.

@ewindisch
Copy link
Contributor

@sroebuck re:#2 - is this reproducible on the master branch?

@sroebuck
Copy link

Your question hits my concern. I'm unfortunately not in a position to say whether this works in master but I know it exhibits these issues in the current latest release 1.1.2. The closing comment is not at all clear on what should or shouldn't work and I don't know if master at the time of the comment corresponds to the current 1.1.2 or a future release with 'fixed' behaviour.

@ewindisch
Copy link
Contributor

@sroebuck this has been marked closed because it was confirmed to be fixed in 'master'. The presence of this fix will be in the next release (1.2)

@Espace2001
Copy link

Very Easy, make a df on command line, and you will see where docker has monted his container, so you can tar it.

@nuest
Copy link

nuest commented Jan 3, 2017

I've found that saving a image by id does not preserve the name and tag. Is this intentional?

I understand the digest sha overrides the tag, but I don't see why the loaded image (last command) cannot contain the original image name.

daniel@gin-nuest:~$ docker --version
Docker version 1.12.5, build 7392c3b

daniel@gin-nuest:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
fdab12439263: Pull complete 
Digest: sha256:f102731ae8898217038060081c205aa3a4ae3f910c2aaa7b3adeb6da9841d963
Status: Downloaded newer image for busybox:latest

daniel@gin-nuest:~$ docker save busybox > busybox_name.tar
daniel@gin-nuest:~$ docker load < busybox_name.tar
Loaded image: busybox:latest
Loaded image ID: sha256:1efc1d465fd6255396d0efd90a899f57aba2b7b294b02d5f55c6f5505ca9f3e5
Loaded image ID: sha256:e02e811dd08fd49e7f6032625495118e63f597eb150403d02e3238af1df240ba
daniel@gin-nuest:~$ docker save busybox:latest > busybox_name-latest.tar
daniel@gin-nuest:~$ docker load < busybox_name-latest.tar
Loaded image: busybox:latest

daniel@gin-nuest:~$ docker save busybox:latest@sha256:f102731ae8898217038060081c205aa3a4ae3f910c2aaa7b3adeb6da9841d963 > busybox_name-digest.tar
daniel@gin-nuest:~$ docker load < busybox_name-digest.tarLoaded image ID: sha256:1efc1d465fd6255396d0efd90a899f57aba2b7b294b02d5f55c6f5505ca9f3e5

daniel@gin-nuest:~$ docker save busybox@sha256:f102731ae8898217038060081c205aa3a4ae3f910c2aaa7b3adeb6da9841d963 > digest.tar
daniel@gin-nuest:~$ docker load < digest.tarLoaded image ID: sha256:1efc1d465fd6255396d0efd90a899f57aba2b7b294b02d5f55c6f5505ca9f3e5

@thaJeztah
Copy link
Member

I understand the digest sha overrides the tag, but I don't see why the loaded image (last command) cannot contain the original image name.

A single digest can be tagged under multiple names; when saving by digest, the name:tag is therefore not part of the saved image.

@attila123
Copy link

attila123 commented Mar 31, 2018

I wrote a shell script to save all the docker images with repo + tag info as .tar files, see https://gist.github.com/attila123/8f60a0ebce15b6065975ba35e3be7d75

@vipcxj
Copy link

vipcxj commented Aug 7, 2018

@thaJeztah IMO, save should save everything including name:tag. And then we can using some option to control what to load from the tar. This should be perfect.

@gixnex
Copy link

gixnex commented Nov 23, 2018

I use this one liner, may be it helps:

for i in $(docker images | awk '{if (NR!=1) {print $1":"$2","$3}}');do ID_DOCKER=$(echo $i | cut -d',' -f2); TAG_DOCKER=$(echo $i | cut -d',' -f1); echo "docker tag $ID_DOCKER $TAG_DOCKER"; done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests