Docker daemon leaks containers when an exec'd container dies

[mohitsoni]

I am using docker 1.4.1, and very recently I encountered a consistently reproducible bug (on ubuntu and boot2docker), where docker daemon thinks the container is running, but the container is actually gone. This usually happens when a docker exec operation is in progress on a container and the main process of that container dies.

To reproduce this issue you can try following:

1. Create a script called crash.sh which sleeps for sometime and then exits with a non-zero code:

   #!/bin/sh
   sleep 10
   exit 1

2. Then write a Dockerfile:

   FROM busybox
   COPY crash.sh /tmp/
   CMD "/tmp/crash.sh"

3. Create an image:

   docker build -t crash_bug .

4. Run that image:

   docker run --name 'crash_bug' -ti crash_bug

5. Open another terminal and then exec into this container:

   docker exec -t -i 'crash_bug' /bin/sh

Once the sleep will expire, you'll be kicked out of the terminal from step #5. At this point, you won't be able to either stop or kill the container from step #4 (sending a SIGINT also doesn't work). For ex:

$ docker kill a4f003724c6e
Error response from daemon: Cannot kill container a4f003724c6e: no such process
FATA[0000] Error: failed to kill one or more containers

The only way to get rid of the leaked container is to actually restart the host. Because, when you try to restart the daemon for the first time, it doesn't come up. Restarting daemon the second time seems to clean the state.

[cpuguy83]

I belive this is fixed on master, per #9699 Can you test?
You can get a master binary from https://master.dockerproject.com/

[LK4D4]

@cpuguy83 Yup, same issue.
@mohitsoni Thanks for report!

[cpuguy83]

@jpetazzo Did you try on master?

[cwdcwd]

I'm on a mac with 1.4.1 and boot2docker and I'm experiencing this same issue.
was attached to a container with a run away for loop in a shell script. I killed the container and tried to remove the image but can't because it thinks the container is still active

$ docker rmi bcfa043029fa -f
Error response from daemon: Conflict, cannot delete bcfa043029fa because the container 04abb6cbcb06 is using it, use -f to force
Error response from daemon: No such image: -f
FATA[0000] Error: failed to remove one or more images

[thaJeztah]

@cwdcwd What you're seeing is not a bug; docker reports that the image you're trying to remove is still in use. Stopping a container doesn't remove it. If you execute docker ps -a, you'll likely see that the container you "killed", is still there but with a status "Exited".

The -f option (to "force" removing the image, even if it is still used by a container), should be before the container name/id, thus docker rmi -f bcfa043029fa (see the documentation)

Alternatively (and a bit less "brute force"), remove the stopped container first, using docker rm 04abb6cbcb06, and then remove the image using docker rmi ....

[cwdcwd]

@thaJeztah aahhhhhh thank you very much!

[thaJeztah]

Glad I could help!