Live restore breaks in-use check on volume rm

[dsanders11]

Description

docker volume rm was allowed to proceed on an in use volume with no warning or error. Everything continue to function as normal until a system restart stopped the container, and a subsequent start failed with: Cannot start service foobar: get 53c78ad9781aa1f45f9ce84cc324625504096b937ed22118a038a65af4210bfc: no such volume.

It appears that the reason this worked is because the system has "live restore" enabled, and the docker daemon had been restarted (due to upgrade from 18.06.0-ce to 18.06.1-ce).

Steps to reproduce the issue:

0. "live restore" must be enabled
1. Restart docker daemon
2. Run docker volume rm with an in-use volume

Describe the results you received:

Volume removed without any complaints about in-use status.

Describe the results you expected:

Error prohibiting removal due to in-use state. Removal attempts before restarting the daemon triggered this error successfully.

Additional information you deem important (e.g. issue happens only occasionally):

This is definitely related to "live restore", I've tested it a few times now. It seems that the check for in-use is bypassed if the daemon is restarted while the container stays running. I have not tried stopping the container and starting it after the daemon is restarted to see if that somehow reconnects the check.

Output of docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:03 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:25:29 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 7
 Running: 6
 Paused: 0
 Stopped: 1
Images: 16
Server Version: 18.06.1-ce
Storage Driver: devicemapper
 Pool Name: docker2-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Udev Sync Supported: true
 Data Space Used: 8.236GB
 Data Space Total: 12.23GB
 Data Space Available: 3.999GB
 Metadata Space Used: 3.387MB
 Metadata Space Total: 125.8MB
 Metadata Space Available: 122.4MB
 Thin Pool Minimum Free Space: 611.3MB
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.11.6.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.7GiB
Name: ip-172-31-22-109
ID: HS2S:5VZ6:KMAR:QCGS:U22B:LM4J:QMZM:BCVH:NH53:6FFK:XTR6:W7LH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.):

[dsanders11]

@thaJeztah, this probably warrants a highish priority since "live restore" is not an experimental feature, the bug can lead to data loss, and it's consisten to reproduce.

[dsanders11]

Tried to create a minimal reproduce case for this using the alpine image, but no luck. I have a system where it readily reproduces on, but I was not able to create a reproduce case on another system.

Things I know about the situation it reproduces in:

• The volume was created via VOLUME a b c d in a Dockerfile (note that a, b, c and d are four different volumes being created in one line)
• The volume has no name, just the long hexadecimal identifier

If more info can be extracted from the system I've got where it reproduces, lemme know and I'll try to provide that.

[thaJeztah]

Thanks for reporting! 🤔 sounds like the in-memory state is being reset somewhere when restarting the daemon.

ping @anusha-ragunathan @cpuguy83 is this information persisted on disk as well (so that this can be reloaded?); If not persisted separately, I assume this can be obtained from each container's configuration.

[cpuguy83]

I would have to dig in and see if something changed.
When the daemon is started it should be re-populating the volume references even for live restore. Surprised there isn't a test for this, I thought there was.

[vvoland]

This is fixed by: #44231

Let me close this issue.