docker build hangs/crashes when useradd with large UID

[mcieslik-mctp]

When I try to add a user during a "docker build ." the process hangs for approx 2-3 min and crashes with a

$ docker build .
Uploading context  5.12 kB
Uploading context 
Step 0 : FROM ubuntu:14.04
 ---> 99ec81b80c55
Step 1 : RUN useradd -u 99900000 -g users mcieslik
 ---> Running in 3ba3c92673fd
2014/04/26 14:58:55 write /var/lib/docker/devicemapper/mnt/.../rootfs/var/log/lastlog: no space left on device

Dockerfile

FROM ubuntu:14.04 
RUN useradd -u 99900000 -g users mcieslik

If I change the above to

FROM ubuntu:14.04 
RUN useradd -u 1001 -g users mcieslik

or run

useradd -u 99900000 -g users mcieslik

in a

docker run -i -t ubuntu:14.04 /bin/bash

everything works fine

[pnasrat]

We're probably doing the wrong thing with sparse files which /var/log/lastlog is, it looks like you are using the devicemapper storage backend

Can you update this the output of docker info

I'd also be interested if you can you explain your need for such a large uid?

[mcieslik-mctp]

Thanks for your prompt answer. These user ids are given to each user (university wide) by our IT overlords. Setting a proper UID is needed to write to NAS mounts.

Containers: 58
Images: 30
Storage Driver: devicemapper
 Pool Name: docker-254:3-12587682-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 14230.2 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 11.4 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.1                                                                                                                                                                                                                   
Kernel Version: 3.14.1-1-ARCH                                                                                                                                                                                                                  
WARNING: No swap limit support   

[pnasrat]

As a work around can you try the -l or --no-log-init in the Dockerfile

RUN useradd -l -u 99900000 -g users mcieslik

[mcieslik-mctp]

Thanks! This worked.

[unclejack]

The underlying issue is that a large sparse file is created (approximately 32 GB), but it's not exactly a Docker bug. Docker could make an attempt to handle large sparse files better, but that's a subject for the #docker-dev mailing list.

I'll close this issue now. Please feel free to comment.

[rhvgoyal]

So is this an issue with "docker commit" that it can't handle sparse files and blots the file to the full size during commit. If yes, then I guess this is something which should be fixed in docker.

I created a 1G sparse file in a container (with overlay backend) and then did docker commit and top most layer size was 1G. So docker did inflate the file to 1G and that seems very inefficient.

[rhvgoyal]

@unclejack Has this issue been discussed since then any where else? I tested this with latest docker so it has not been fixed yet.

[trntv]

usermod causes the same problem and it doesn't have --no-log-init
Storage Driver: aufs

[runcom]

It's happening the same on overlay.. I'm going to reopen this to track this bug and see if Docker can better handle this scenario

[vbatts]

golang's archive/tar only supports extracting from archives with sparse files, not creating an archive with sparse files.
From my searches, only GNU tar boasts creation of archives with sparse files. Many implementations barely support extracting these archives created by GNU tar

[wontonst]

This issue still occurs on docker 1.10.2

Step 7 : RUN echo "start" && echo $(useradd -m -u $uid -g $gid rdsdb) && echo "done"
 ---> Running in d1d07c9f2316
start

done

Hangs after printing done.

Docker info:

Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 12
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 86
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-79-generic
Operating System: Ubuntu precise (12.04.5 LTS)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.58 GiB
Name: xxx
ID: xxx
WARNING: No swap limit support

[sdwolfz]

I can also confirm this is still occurring with 1.10.2

I am trying to create the following image:

FROM node:5.7.1

ARG HOST_USER_UID=1000
ARG HOST_USER_GID=1000

RUN DEBIAN_FRONTEND=noninteractive                            && \
                                                                 \
    echo 'Creating notroot user and group from host'          && \
    addgroup --gid $HOST_USER_GID notroot                     && \
    adduser --uid $HOST_USER_UID --gid $HOST_USER_GID notroot && \
                                                                 \
    echo 'Installing testing tools'                           && \
    npm install -g mocha phantomjs testem

USER notroot

WORKDIR /work

EXPOSE 9876

CMD testem --host 0.0.0.0 --port 9876

And I am building with the following command:

docker build -t testem --build-arg HOST_USER_UID=`id -u` --build-arg HOST_USER_GID=`id -g` .

The build freezes after displaying all output from RUN and writes to /var/lib/docker/aufs/diff/a66b81c2371267f6e749ecc667ed9c39dfabdb6f8d767c88d80fe07525ac6ade/var/log/lastlog until the disk is full.

docker info output:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 13
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.19.0-51-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.679 GiB
Name: cgi
ID: E4J7:MZX3:WSEP:CEVQ:HGMD:GNQT:IW5L:NJYX:GTUV:6HRV:LB3L:36EK
WARNING: No swap limit support

[sdwolfz]

I also can confirm that @pnasrat's workaround fixes this problem. I am now using:

groupadd -g $HOST_USER_GID notroot                           && \
useradd -l -u $HOST_USER_UID -g $HOST_USER_GID notroot       && \

instead of:

addgroup --gid $HOST_USER_GID notroot                     && \
adduser --uid $HOST_USER_UID --gid $HOST_USER_GID notroot

And the build succeeded.

My $HOST_USER_UID and $HOST_USER_GID contain 10 digit IDs

[AkihiroSuda]

linking golang/go#13548 to this issue

[gbraad]

Reproducible as an automated build on Docker Hub: https://hub.docker.com/r/gbraad/issue-dockerfile/builds/bsdmngknf8dhfreh8sgfmyu/ :-s

While adduser -l can work around the issue, when using something like Ansible inside the container to configure an environment or some other use of su - or sudo, will cause the same issue.

[justincormack]

Presumably disk quotas will fix the issue with the host running out of space and just leave a runtime error?