Unable to remove container

[jayfk]

I wasn't able to remove a container that failed to start when running docker-compose up. I had to reboot the machine to resolve the issue.

docker-compose -v
docker-compose version 1.6.0, build d99cad6

docker -v                                                                                                              
Docker version 1.10.2, build c3959b1

docker version                                                                                                         
Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 21:37:01 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 21:37:01 2016
 OS/Arch:      linux/amd64

docker-compose ps                                                                                                      
          Name                         Command               State     Ports
------------------------------------------------------------------------------
7236570cd4_project_nginx_1   nginx -g daemon off;             Exit 0

docker-compose logs nginx                                                                                              
Attaching to 7236570cd4_project_nginx_1
7236570cd4_project_nginx_1 | Error attaching: open /var/lib/docker/containers/7236570cd4ea0443d48e944ec8bbf76b17275452ca40b44af6a69db313ab6a3f/7236570cd4ea0443d48e944ec8bbf76b17275452ca40b44af6a69db313ab6a3f-json.log: no such file or directory
7236570cd4_project_nginx_1 exited with code 0

docker-compose rm                                                                                                      
Going to remove 7236570cd4_project_nginx_1
Are you sure? [yN] y
Removing 7236570cd4_project_nginx_1 ... error

ERROR: for 7236570cd4_project_nginx_1  Driver aufs failed to remove root filesystem 7236570cd4ea0443d48e944ec8bbf76b17275452ca40b44af6a69db313ab6a3f: rename /var/lib/docker/aufs/mnt/c802f1d67f0d650f68ace7fadb65c6db2f1dd36c9a2e615defea6e01419f96fc /var/lib/docker/aufs/mnt/c802f1d67f0d650f68ace7fadb65c6db2f1dd36c9a2e615defea6e01419f96fc-removing: device or resource busy

docker-compose rm -f                                                                                                   
Going to remove 7236570cd4_project_nginx_1
Removing 7236570cd4_project_nginx_1 ... error

ERROR: for 7236570cd4_project_nginx_1  Driver aufs failed to remove root filesystem 7236570cd4ea0443d48e944ec8bbf76b17275452ca40b44af6a69db313ab6a3f: rename /var/lib/docker/aufs/mnt/c802f1d67f0d650f68ace7fadb65c6db2f1dd36c9a2e615defea6e01419f96fc /var/lib/docker/aufs/mnt/c802f1d67f0d650f68ace7fadb65c6db2f1dd36c9a2e615defea6e01419f96fc-removing: device or resource busy

[dnephin]

This is a bug in docker engine (or arguably in the aufs driver). Nothing we can do about this from Compose. I think there is an open issue for it already.

[tholu]

docker-compose -v
docker-compose version: 1.5.1

docker -v
Docker version 1.10.3, build 20f81dd

docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:48:06 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:48:06 2016
 OS/Arch:      linux/amd64

Same thing happened to me. Will try the reboot solution. @dnephin To which issue were you refering?

[dnephin]

moby/moby#21704

[tholu]

Tried restarting docker, worked as well.

[einarf]

This issue disappeared for me when I stopped naming containers with container_name and moved from using depends_on to network aliases. Now replacing/upgrading a container with docker-compose up -d <service name> works perfect every time.

[mdzidic]

Hello,
I don't know is it related but I'm getting this error since starting ntpd service on the host servers (CentOS 7)...
Server Version: 17.05.0-ce

$ docker-compose rm collectd

Going to remove 59479d7093d4_collectd
Are you sure? [yN] y
Removing 59479d7093d4_collectd ... error
ERROR: for 59479d7093d4_collectd b'Driver devicemapper failed to remove root filesystem 59479d7093d474354d44bebf96d3dc261e2cf8118b4ca15689b31b8b36422f04: failed to remove device b87081c5e7b1d6be11ba4951a96f25961e405f39ac3402c2ff2133d89f5c25a5: Device is Busy'

$ find /proc/*/mounts | xargs grep b87081c5e7b1d6be11ba4951a96f25961e405f39ac3402c2ff2133d89f5c25a5

/proc/14263/mounts:/dev/mapper/docker-202:1-2101643-b87081c5e7b1d6be11ba4951a96f25961e405f39ac3402c2ff2133d89f5c25a5 /var/lib/docker/devicemapper/mnt/b87081c5e7b1d6be11ba4951a96f25961e405f39ac3402c2ff2133d89f5c25a5 xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0

$ ps aux | grep 14263

ntp 14263 0.0 0.0 29408 2116 ? Ss Jun26 2:13 /usr/sbin/ntpd -u ntp:ntp -g

$ sudo kill 14263
$ docker-compose rm collectd

Going to remove 59479d7093d4_collectd
Are you sure? [yN] y
Removing 59479d7093d4_collectd ... done

[odg0318]

Do you know why happened? @mdzidic

[mdzidic]

@odg0318 nope, it seems like this problem is gone by new Docker version. 😕