Docker 1.12.3 freezes when VMware VDP does a backup of the Docker Host VM

[hauihau]

Description
We had docker 1.9.1 running on OpenSUSE Leap 42.1 without any problems. After updating docker to first 1.12.1 and afterwards 1.12.3, we're encountering complete system freezes like in /issues/21179. The system is also a VM runnung in a VMware 6.0U2 envorinment. It looks like the issue occurs, when VMware Data Protection does its backups. Neither syslog and systemds journal have any relevant entries.

Steps to reproduce the issue:

1. Install docker 1.12.3 in a OpenSUSE 42.2 VM wich is backuped by VDP
2. Start a container
3. Wait until VDP does the backups

Describe the results you received:
VM totally freezes. No new ssh or console logins possible. Running ssh sessions also freeze when doing e.g tab completion in the shell.

Describe the results you expected:
VM does not freeze when backuped by VDP

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   8eab29e
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   8eab29e
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-8:2-133300-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.187 GB
 Data Space Total: 107.4 GB
 Data Space Available: 15.35 GB
 Metadata Space Used: 1.729 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.03.01 (2015-05-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge overlay null host
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Security Options: apparmor
Kernel Version: 4.4.36-5-default
Operating System: openSUSE Leap 42.2
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953 GiB
Name: www-v212
ID: AGPD:3ICB:OQMB:67IA:EKAZ:USDB:FKEU:RTOA:3CQX:JUDV:NNIZ:M2UM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://www-cache.uni-mannheim.de:3128/
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
Insecure Registries:

Additional environment details (AWS, VirtualBox, physical, etc.):

Please let me know what else I should provide.

[cyphar]

Have you tried using the btrfs graphdriver? Since you're on openSUSE most people will have their root filesystem formatted with btrfs.

[hauihau]

I do not use (no experience/knowledge) btrfs in production. / is an ext4 filesystem.

[justincormack]

If the VM freezes, this seems likely to be a VMWare bug (or possibly a kernel bug) rather than a Docker bug, I would file an issue with them.

[hauihau]

I did already. But as I said: docker 1.9.1 didn't had that issue. So I thought asking here would be also a good idea.

[hauihau]

Excluding the VM from VDP stopped the freeze. Are there changes regarding IO between 1.9.1 and 1.12 which may interfere with the VDP quiescing the FS?

[mbudnick]

I have this problem with OpenSUSE Leap 42.2, filesystem as ext4 and docker 1.12.3.
After using btrfs as filesystem the freeze is gone away.

[mibrault]

I also have this problem, with CentOS 7.3 (last kernel 3.10.0-514.2.2.el7) and Docker 1.12.5 and vmware tools 10.0.9.55972.
As a workaround I stopped vmware-tools client on the VM.

[hauihau]

FYI: vmware/open-vm-tools#122 (comment) says it i an open-vm-tools issue. I've closed the issue.