Docker export creates a tar archive with a lot of files in the filesystem root

[wieslaw-gat]

Description

I'm currently working on creating a docker image for a nodejs application to use with Concourse CI. The image creates fine, works also okay, but when doing a docker export the resulting tar archive has the file system structure completely bonkers, with a ton of files in the archive root.

Steps to reproduce the issue:

1. Download the Dockerfile and package.json.
2. Run docker build . The image builds fine.
3. Use docker run to run the resulting image. The container starts fine.
4. List the root file system contents while inside the running container. Should look like a normal root, like below:

# ls /
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

5. Export the contents of the container you've just created with docker export {CONTAINER_ID} | tar xf -. The operation succeeds.
6. Now list the files you've just extracted. Here's where the magic starts ;-)

Describe the results you received:
Listing the directory to which the tar archive was extracted looks like this:

# ls
0002-gyp-apply-https-codereview.chromium.org-11361103.patch
0003-gyp-don-t-use-links-at-all-just-copy-the-files-inste.patch
2015-01-30.md
abort.js
Accordion.js
acosh
addEventListener.js
addFocusListener.js
additionalItems.json
<snip!>

(the directory was previously empty of course)

Describe the results you expected:
I'd expect the usual files in the root directory of the exported container - the same as ls shows in the point #4 in the steps to reproduce section. Something like this:

# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Additional information you deem important (e.g. issue happens only occasionally):
I'm opening this issue because I've run out of the skills needed to debug it further.

The issue happens regardless of the version of node and yarn used. I've also encountered this problem when using npm instead of yarn.
I don't think it's a yarn/npm issue, since the files are properly placed in the running container. Looks like this problem affects only the exported archive.

Also, this issue happens regardless of the image used. I've also tried my luck with Ubuntu & Alpine images - the same problem persists.

Output of docker version:

% docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Output of docker info:

% docker info                                                                   
Containers: 10
 Running: 0
 Paused: 0
 Stopped: 10
Images: 99
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-252:4-1048577-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.99 GB
 Data Space Total: 107.4 GB
 Data Space Available: 8.922 GB
 Metadata Space Used: 10.99 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.136 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge overlay host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.6.5-040605-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.508 GiB
Name: waterdeep
ID: K3W6:FNMX:V3Y6:25ML:BLRN:M6EW:FSU5:RUZQ:BHQ5:VM7H:UBHT:NF7U
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Happens on both physical machines & in AWS.

[thaJeztah]

ping @dmcgowan PTAL

[dmcgowan]

Yikes, there is definitely something weird going on here. Going to write some more tests around this to see if I can figure out what is going on. It does seem to effect files in the node_modules directory, wonder if the /#/ could be messing something up on export.

Here is an example of one of the files

docker export 0cd3187b7d26 | tar -t | grep "first.js"     
root/.cache/yarn/npm-lodash-4.16.4-01ce306b9bad1319f2a5528674f88297aeb70127/first.js
root/.cache/yarn/npm-lodash-4.16.4-01ce306b9bad1319f2a5528674f88297aeb70127/fp/first.js
first.js
usr/local/lib/node_modules/yarn/node_modules/es5-ext/array/#/first.js
usr/local/lib/node_modules/yarn/node_modules/es5-ext/test/array/#/first.js
usr/local/lib/node_modules/yarn/node_modules/lodash/first.js
usr/local/lib/node_modules/yarn/node_modules/lodash/fp/first.js
usr/src/app/node_modules/lodash/first.js
usr/src/app/node_modules/lodash/fp/first.js

[thaJeztah]

Thanks for an excellent bug-report @wieslaw-gat; having all the information and the reproduce case ready is really helpful ❤️

[dmcgowan]

Looks like it is a bug related to path length in the tar

inside container

# find / -name "first.js"
/root/.cache/yarn/npm-lodash-4.16.4-01ce306b9bad1319f2a5528674f88297aeb70127/first.js
/root/.cache/yarn/npm-lodash-4.16.4-01ce306b9bad1319f2a5528674f88297aeb70127/fp/first.js
/usr/local/lib/node_modules/npm/node_modules/node-gyp/node_modules/path-array/node_modules/array-index/node_modules/es6-symbol/node_modules/es5-ext/array/#/first.js
/usr/local/lib/node_modules/yarn/node_modules/es5-ext/array/#/first.js
/usr/local/lib/node_modules/yarn/node_modules/es5-ext/test/array/#/first.js
/usr/local/lib/node_modules/yarn/node_modules/lodash/first.js
/usr/local/lib/node_modules/yarn/node_modules/lodash/fp/first.js
/usr/src/app/node_modules/lodash/first.js
/usr/src/app/node_modules/lodash/fp/first.js

in tar

docker export 0cd3187b7d26 | tar -t | grep "first.js"
root/.cache/yarn/npm-lodash-4.16.4-01ce306b9bad1319f2a5528674f88297aeb70127/first.js
root/.cache/yarn/npm-lodash-4.16.4-01ce306b9bad1319f2a5528674f88297aeb70127/fp/first.js
first.js
usr/local/lib/node_modules/yarn/node_modules/es5-ext/array/#/first.js
usr/local/lib/node_modules/yarn/node_modules/es5-ext/test/array/#/first.js
usr/local/lib/node_modules/yarn/node_modules/lodash/first.js
usr/local/lib/node_modules/yarn/node_modules/lodash/fp/first.js
usr/src/app/node_modules/lodash/first.js
usr/src/app/node_modules/lodash/fp/first.js

[dmcgowan]

Looks like this hit golang/go#17630

We will test this with go 1.8 to see if it is addressed

Looks like importing it back into Docker works fine, but when using tar it is broken

[wieslaw-gat]

@dmcgowan Any updates?

Do you have any idea how to work around this?

[thaJeztah]

Trying to link the right issues, but it may be that a full fix was pushed back to Go 1.10 golang/go#12594