"mount destination not absolute: unknown." Seems to be tied to Upgrade from Ubuntu 18.04 to Ubuntu 20.04

[sosukeinu]

Description

All of my docker images are having some issues. I've used Docker everyday for years, and have never had any issues. However, today (2021-06-07) for the first time, I am now having a very peculiar issue with permissions, it seems. I issue the command docker run -it -v ~/Documents/notebooks:/notebooks -p 8899:8888 myimage:my-version This regularly worked without issue. Now, however, I'm getting the response:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:[/notebooks] Type:bind Source:/var/lib/docker/volumes/ca4b81f5d5366bbb09b71fcd2353fcbde46bb96ff4e3ca7895fb60d51993fd3a/_data Options:[rbind]}: mount destination [/notebooks] not absolute: unknown.
ERRO[0000] error waiting for container: context canceled 

So, I looked to see if /var/lib/docker/volumes/ca4b81f5d5366bbb09b71fcd2353fcbde46bb96ff4e3ca7895fb60d51993fd3a/_data existed. It does, but it's permissions are off. I notice inconsistent permissions when I ls -al /var/lib/docker/volumes:

drwx-----x 34 root root  61440 Jun  7 11:55 .
drwx--x--x 16 root root   4096 Jun  7 11:31 ..
drwxr-xr-x  3 root root   4096 Jun  7 08:08 05de6b1c9a0fff7313b7efcbba1c4c36fe96a1c87f2f5fd79872bc052fa2fcc9
drwxr-xr-x  3 root root   4096 Jun  7 08:32 1131e3748469d003fa43445380e437ce121004c97f31e371b07a16163b8f1fce
drwxr-xr-x  3 root root   4096 Jun  7 08:32 168c6431448042efe9576b890cf67e7d0b3da0d86018570708578355f56ba044
drwxr-xr-x  3 root root   4096 Jun  7 07:57 2a28be34888a198381c70c85675fb6bcb1f67ac7c6792e214e8f06dd1a0fc917
drwxr-xr-x  3 root root   4096 Jun  7 08:05 548b4143bf75d319347240e1926f6d3d966f6a959be80ea7c8280c8aa2d783be
drwxr-xr-x  3 root root   4096 Jun  1 13:09 552b5cf49694caabea2564353ae0404ef61455a46cab8a0f9048bd6bf50ea0f2
drwxr-xr-x  3 root root   4096 Jun  7 07:53 5f96b984188413631b88d8e44938a071310e27f29db1f75a1b5aa5baa31fa0af
drwxr-xr-x  3 root root   4096 Jun  7 08:16 68573af0a30fda7a84cd693dda7096647769a1bf8e2cb1b679bf55c25fdb7447
drwxr-xr-x  3 root root   4096 Jun  7 10:43 6c47d07b287b222573fb1acd91dd13ad86973b031738ca9989bc08f6e1bcdfd9
drwx-----x  3 root root   4096 Jun  7 10:44 87a4235ef1cfe5a850740b351885d15dc89b1fbf50da3635ffa13d76e459cf84
drwx-----x  3 root root   4096 Jun  7 11:55 ca4b81f5d5366bbb09b71fcd2353fcbde46bb96ff4e3ca7895fb60d51993fd3a
...

I tried updating the permissions sudo chmod -R 755 /var/lib/docker/volumes/{volume-hash} but whenever i restart the daemon, or re-issue the run command, the permissions revert, or a new volume is created that also has the wrong permissions.

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:38 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:50 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 20
  Running: 0
  Paused: 0
  Stopped: 20
 Images: 1078
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-74-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.26GiB
 Name: jonathanbatteas-OMEN-by-HP-Laptop
 ID: ZTKV:OWN7:ZLVN:7YQB:2O7I:U5YF:H7AI:5PW2:APZM:IOP5:3VS2:GKNX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: sosukeinu
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

[thaJeztah]

Looks related to opencontainers/runc#2928, containerd/containerd#5547, and opencontainers/runc#3004

Are you seeing the same issue if you start the image without the bind-mount (without ~/Documents/notebooks:/notebooks)?

If it still fails with that, could you do a docker image inspect of your image and search for /notebooks ? I'm wondering what the image defines as path for the volume.

[sosukeinu]

Thank you for the response. As a sidenote, you were correct, after downgrading containerd.io to 1.4.4-1, the container began working without an issue. It does still fail without the bind-mount. My inspect looks like this:

[
    {
        "Id": "sha256:250238b1cff2f51e2405ea098a2ec7dfcb13c715e01154a987781f5156c5aa89",
        "RepoTags": [
            "jupyter-notebook:markdown-journal"
        ],
        "RepoDigests": [],
        "Parent": "sha256:852e2fcde8490666c9033175283626cc6104b017287ec6361fea9ca712e9f09d",
        "Comment": "",
        "Created": "2021-06-07T11:57:50.105899972Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "notebook",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8888/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LIBRARY_PATH=/lib:/usr/lib",
                "SHELL=/bin/bash",
                "GOPATH=/go"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) COPY file:83c449108e4c6e705971b3b450358371cf9ca4d84061d1ce5e95470089a10597 in /etc/jupyter/jupyter_notebook_config.py "
            ],
            "Image": "sha256:852e2fcde8490666c9033175283626cc6104b017287ec6361fea9ca712e9f09d",
            "Volumes": {
                "[/home/notebook]": {},
                "[/notebooks]": {}
            },
            "WorkingDir": "/notebooks",
            "Entrypoint": [
                "docker-entrypoint"
            ],
            "OnBuild": null,
            "Labels": {
           ...
            }
        },
        "DockerVersion": "20.10.7",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "notebook",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8888/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LIBRARY_PATH=/lib:/usr/lib",
                "SHELL=/bin/bash",
                "GOPATH=/go"
            ],
            "Cmd": null,
            "Image": "sha256:852e2fcde8490666c9033175283626cc6104b017287ec6361fea9ca712e9f09d",
            "Volumes": {
                "[/home/notebook]": {},
                "[/notebooks]": {}
            },
            "WorkingDir": "/notebooks",
            "Entrypoint": [
                "docker-entrypoint"
            ],
            "OnBuild": null,
            "Labels": {
                ...
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 522557683,
        "VirtualSize": 522557683,
        "GraphDriver": {
            "Data": {
                ...
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                ...
            ]
        },
        "Metadata": {
            ...
        }
    }
]

[Turbocube644]

I had the same issue but with docker-compose (see containerd/containerd#5547 (comment)).
Removing stopped containers before running them without the bind worked for me searching for the cause.

[thaJeztah]

Looks like that config is incorrect, but that problem was previously ignored;

            "Volumes": {
                "[/home/notebook]": {},
                "[/notebooks]": {}
            },

In the config, there's two volumes defined for [/home/notebook] and [/notebooks]. note that those are literal paths.

Here's an example to reproduce the same:

FROM alpine
VOLUME [/notebooks]
VOLUME [/home/notebook]

(note that the correct volumes likely had to be ["/notebooks"] and ["/home/notebook"] (in JSON format), but the quotes were either forgotten or stripped somewhere)

Building the above:

docker build -t foo .

And run ls in the container; notice that there's a directory named [ at the root:

docker run -it --rm foo ls -l /
total 72
drwxr-xr-x    4 root     root          4096 Jun  7 21:37 [
drwxr-xr-x    2 root     root          4096 Oct 21  2020 bin
...

And when showing whats in it (with tree);

docker run -it --rm foo sh -c 'apk -q add --no-cache tree && tree /['
/[
├── home
│   └── notebook]
└── notebooks]

3 directories, 0 files

[sosukeinu]

(note that the correct volumes likely had to be ["/notebooks"] and ["/home/notebook"] (in JSON format), but the quotes were either forgotten or stripped somewhere)

Excellent! thank you so very much

[cweiske]

We had the volume definition in Dockerfile with single quotes. After changing it to double quotes, it works again:

- VOLUME ['/var-data/']
+ VOLUME ["/var-data/"]

[thaJeztah]

This issue should be fixed by opencontainers/runc#3004, which relaxed the validation in runc, and is part of runc v1.0.0, which is included in the containerd.io v1.4.7 (and up) packages.

Static binaries of docker do not yet include that version of containerd and runc, but should become available with the next docker patch release.

Let me go ahead and close this ticket because of the above, but feel free to continue the conversation