container error after upgrade to containerd.io 1.4.6-1

[KeyserSoeze]

System: docker-ce version 20.10.6 on Debian buster 10.9

Bug: After last update of conainerd.io from version 1.4.4-1 to 1.4.6-1 docker container immediately stop working.

Error messages from docker status:

cleanup: failed to delete container from containerd: no such container
failed to start container ... task 64c... already exists: unk

Restarting docker did not solved the problem.

Workaround: Downgrade containerd.io to previous version (sudo apt-get install containerd.io:amd64=1.4.4-1) and container immediately worked again.

[nilsglow]

I also have problems after upgrading from 1.4.4-1 to 1.4.6-1.

I get the following error:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:[...] Type:bind Source:/var/lib/docker/volumes/ ... /_data Options:[rbind]}: mount destination [...] not absolute: unknown.

Downgrading to 1.4.4-1 also helped me immediately.

[murach]

Same here, needed to downgrade from 1.4.6 to 1.4.4.

[goodspark]

Same thing! Had to downgrade. It fails even for a simple Dockerfile, such as:

FROM ubuntu:18.04

RUN echo asd

docker build -t asd .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:18.04
 ---> 81bcf752ac3d
Step 2/2 : RUN echo asd
 ---> Running in 71fc8a0abb62
OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: no such file or directory: unknown

docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

On Ubuntu 18:04

[mesquita]

I was getting:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:: Type:bind Source:/var/lib/docker/volumes/9e1805936631f35e4895a18840a0f56c21669099b5967954b176585e9af7a2da/_data Options:[rbind]}: mount destination : not absolute: unknown.

Downgrading as pointed out by @KeyserSoeze worked.

[thaJeztah]

@KeyserSoeze not sure what happened in your case; do you happen to be using the "live restore" option on Docker?

I tried reproducing by:

• installing docker 20.10.6 and containerd v1.4.4 + runc v1.0.0-rc93
• start some containers
• upgrade containerd to 1.4.6 + runc v1.0.0-rc95
• start some containers

wget -q -O- "https://download.docker.com/linux/debian/gpg" | apt-key add -qq -
echo "deb [arch=amd64] https://download.docker.com/linux/debian buster stable" > /etc/apt/sources.list.d/docker.list

# install docker 20.10.6 and containerd v1.4.4 + runc v1.0.0-rc93
apt-get update && apt-get install -y -qq --no-install-recommends \
    docker-ce-cli=5:20.10.6~3-0~debian-buster \
    docker-ce=5:20.10.6~3-0~debian-buster \
    containerd.io=1.4.4-1

# run some containers
docker run -d --name web nginx:alpine
docker run -it ubuntu:18.04

# upgrade containerd
apt-get install -y containerd.io=1.4.6-1

# try starting the "web" container
docker start web

# try running an ubuntu container
docker run -it --rm ubuntu

Does upgrading docker to 20.10.7 solve your issue?

w.r.t. #5547 (comment) and #5547 (comment)

@nilsglow @mesquita

That looks related to a breaking change in runc (see opencontainers/runc#2928). We applied changes in Docker 20.10.7 to work around that change (see moby/moby#42370), as well as changes in BuildKit (moby/buildkit#2123) but it's possible that (e.g. in situations where you're running a swarm service which was already created) that this is causing issues.

I'm not sure if the runc maintainers want to revert that change for v1.0.0 (but I can try)

@goodspark I tried reproducing your issue (which looks to be separate from the above), but wasn't able to: what distro are you running on? (perhaps you can edit your comment and add the output of docker info ?)

[cdalvaro]

I'm having this same issue with Github workflows using ubuntu-latest:

Current runner version: '2.278.0'
- Operating System
  Ubuntu
  20.04.2
  LTS
- Virtual Environment
  Environment: ubuntu-20.04
  Version: 20210531.0
  Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20210531.0/images/linux/Ubuntu2004-README.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20210531.0

docker version
- Client:
  Version:           20.10.6+azure
  API version:       1.41
  Go version:        go1.13.15
  Git commit:        370c28948e3c12dce3d1df60b6f184990618553f
  Built:             Fri Apr  9 17:01:36 2021
  OS/Arch:           linux/amd64
  Context:           default
  Experimental:      true

- Server:
  Engine:
    Version:          20.10.6+azure
    API version:      1.41 (minimum version 1.12)
    Go version:       go1.13.15
    Git commit:       8728dd246c3ab53105434eef8ffe997b6fd14dc6
    Built:            Fri Apr  9 22:06:18 2021
    OS/Arch:          linux/amd64
    Experimental:     false
  containerd:
    Version:          1.4.6+azure
    GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
  runc:
    Version:          1.0.0-rc95
    GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
  docker-init:
    Version:          0.19.0
    GitCommit:

(I have already tried downgrading to moby-containerd=1.4.4+azure-1, but the error persists)

This is the error output:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:[ Type:bind Source:/var/lib/docker/volumes/361b5350807ea92c892fdc87e911040d484c24db5b72b74bf5a865374e416224/_data Options:[rbind]}: mount destination [ not absolute: unknown.
Error: Process completed with exit code 125.

The error is reproducible by running the following command in a GitHub workflow:

# Create configuration files
mkdir -p /tmp/config/
cat > /tmp/config/salt-api.conf <<EOF
external_auth:
  pam:
    salt_api:
      - .*
      - '@runner'
      - '@wheel'
      - '@jobs'
EOF

# Run test instance
docker run --rm --detach --name saltstack_master \
  --publish 4505:4505 --publish 4506:4506 --publish 8000:8000 \
  --env 'SALT_API_SERVICE_ENABLED=true' \
  --env 'SALT_API_USER_PASS=4wesome-Pass0rd' \
  --platform linux/amd64 \
  --volume /tmp/config:/home/salt/data/config:ro \
  cdalvaro/docker-salt-master:latest

# Wait for salt-master bootup
sleep 60

[thaJeztah]

Hm... in your case, I wonder if you made a typo somewhere, or if some script produced an incorrect output;

mount destination [ not absolute

Seems to indicate you're attempting to mount to [ inside the container.

[cdalvaro]

Hm... in your case, I wonder if you made a typo somewhere, or if some script produced an incorrect output;

mount destination [ not absolute

Seems to indicate you're attempting to mount to [ inside the container.

Thank you for your reply, @thaJeztah!

Initially, that was my first thought. But then I realized that these same settings were running well until a dependabot PR where checks began to fail. I haven't approved that PR yet, but checks are failing now in the main branch.

[thaJeztah]

Yeah, it's definitely odd. The "source" of that mount (/var/lib/docker/volumes/361b5350807ea92c892fdc87e911040d484c24db5b72b74bf5a865374e416224/_data) looks like it's an "anonymous" volume (which would be if the container's image has a VOLUME specified). The error message is weird though, as it would mean that somehow the VOLUME definition in the image that's used has [ as target.
That could be if (e.g.) it was built from a Dockerfile that uses the JSON notation for volumes, something like:

VOLUME ["/some/path/", "/some-other/path/", "/yet/another/path/"]

If that JSON was invalid (e.g., using single quotes instead of double quotes, or a missing comma) during building the image, Docker will fallback to treating that JSON as a literal string (which would explain the error message).

Are you able to do a docker inspect of the cdalvaro/docker-salt-master:latest image and the container that was started?

[Turbocube644]

I ran into the same mounting error after upgrading from 1.4.4-1 to 1.4.6-1.
Mine was indeed caused by an incorrect VOLUME declaration which worked before.
Before it was:

VOLUME [ ${DATA_DIR} ]

As the quotation marks are missing, this is an incorrect declaration of a JSON array. Changing it to the following (and removing stopped containers beofre running build) solved it:

VOLUME [ "${DATA_DIR}" ]

[cdalvaro]

You are my hero @thaJeztah 🎉

By doing docker inspect cdalvaro/docker-salt-master:latest there are two wrong volumes:

"Volumes": {
  "/home/salt/data/3pfs": {},
  "/home/salt/data/config": {},
  "/home/salt/data/keys": {},
  "/home/salt/data/logs": {},
  "/home/salt/data/srv": {},
  "[": {},
  "]": {}
},

I had a multiple directories VOLUME command without commas. I have separated those directories with commas and it has worked like a charm! (cdalvaro/docker-salt-master@7a03130)

Thank you again for your help!

[yomofo2s]

Looks like this is really a common problem with the mount volume while using docker image by nginx

$docker run --name spa-web -v $(pwd):/usr/share/nginx/html:ro -d nginx

I have have similar issue with OCI runtime create failed: invalid mount while mounting my mount file. I observed a new folder was created which is totally different from my current directory where my html files are..This is absolutely strange to me.

the spa was what i specified in my code while spa;C kept popping up after i run my command.

any help would be greatly appreciated

Docker version 20.10.7, build f0df350
windows 10

[yomofo2s]

Okay so i figured out a workaround. I changed my terminal and stopped using bash script in interi,m. So i tried powershell instead because of windows os. However, something changed. in my code, () now {}

$docker run --name spa-web -v ${pwd}:/usr/share/nginx/html:ro -d nginx

[theAkito]

Downgrading worked for me, too.
That said, I wanted to elaborate on my specific scenario.
In my case I was reviving a script from 2019 that worked perfectly back then.
The first time the script worked fine. Then, when forcefully removing the container and re-creating it with the exact same script, I encountered the same error as @mesquita:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:: Type:bind Source:/var/lib/docker/volumes/9e1805936631f35e4895a18840a0f56c21669099b5967954b176585e9af7a2da/_data Options:[rbind]}: mount destination : not absolute: unknown. 

It just wouldn't budge.
Downgrading "solved" it, for now.

Why did it work the first time, though?
That's another question that will hopefully be answered when the real fix will have been implemented.

[thaJeztah]

Why did it work the first time, though?

Hard to tell. If you have a link to the script, perhaps that provides more info as to what the script is doing (and why it failed)

[theAkito]

@thaJeztah

The entire script consists of setting bash variables to passwords and paths for a docker run command. Once they are set, there is the docker run command.

So, there is absolutely nothing else in the script, besides the action of running a single Docker container.

[JamesGallant]

@theAkito

I had the same issue
For some reason its got to do with the volume mount paths the docker-compose.yml, at least in my case. Either specify a volume or have a relative mount. Changing these paths fixed it for me and I did not need to downgrade the containerd, I couldn't figure out how to downgrade it on windows running docker desktop. Perhaps this helps someone out.

services:
  frontend:
    build:
     context: ./frontend
     dockerfile: ./Docker/Dockerfile
    image: react_frontend_service
    container_name: frontend
    restart: always
   command: npm start
    volumes:
      - ./frontend/:/usr/src/app/ # it wants this
     - usr/src/app/ # it does not want this
    ports:
      - "8000:3000"
    env_file:
      - frontend/environments/.develpment.env
    depends_on:
      - postgres_database

[thaJeztah]

 - usr/src/app/ # it does not want this

That line only has a single path, in which case it's considered the path inside the container to create a volume for. When using the shorthand (-v <host-path or volume name>:<container-path>) syntax, if no host-path is specified, then an anonymous volume is created and mounted at <container-path> inside the container. That path (inside the container) must be an absolute path, so /usr/src/app/ in your case.

However, it looks like you specify two volumes/mounts for the same container path;

• a bind-mount to mount your local ./frontend/ directory at /usr/src/app/ inside the container
• an anonymous volume to also mount at /usr/src/app/ inside the container

I think this could potentially lead to a race-condition, where (depending on which mount gets mounted first), the bind-mount masking the content of the anonymous volume, or vice-versa.

Was is intentional to have to mounts? I have a feeling only the first one is needed?

[DarwinSurvivor]

I've run into a similar problem with the latest version.

In my Dockerfile I have the following

VOLUME /path/to/dir # My Comment

When I try to start a container using that image, I get the following error:

OCI runtime create failed: invalid mount {
Destination:My
Type:bind
Source:/var/lib/docker/volumes/<long_hash_here>/_data Options:[rbind]
}: mount destination My not absolute: unknown

Note that it appears to be grabbing the first word of the comment "My" as part of the volume definition. This was not the case before the update. Moving the comment above the VOLUME line as depicted below resolved the error, but requiring such formatting contradicts my understanding of the documentation regarding Dockerfile comments https://docs.docker.com/engine/reference/builder/#format.

# My Comment
VOLUME /path/to/dir

[thaJeztah]

Yes, The Dockerfile syntax only supports comments on their own line, and does not support trailing comments.

Docker treats lines that begin with # as a comment, unless the line is a valid parser directive.
A # marker anywhere else in a line is treated as an argument (...)

In case of VOLUME, the Dockerfile syntax allows multiple volumes to be defined in a single VOLUME instruction;

VOLUME /one /two /three

So what happens if you add the trailing comment;

VOLUME /path/to/dir # My Comment

Is that it defines four volumes: /path/to/dir, #, My, and Comment; # is a valid name for a Directory on Linux, so it's valid as a mount-point as well for a volume.

You can see this when building an image;

FROM busybox
VOLUME /path/to/dir # My Comment

Then, inspecting the image, you can see the 4 volumes in the configuration:

DOCKER_BUILDKIT=1 docker build -t myimage .
docker image inspect --format '{{json .Config.Volumes}}' myimage
{"#":{},"/path/to/dir":{},"Comment":{},"My":{}}

[unleashit]

Just to add to the fray, the update also affected me. I don't doubt that the error is on my part, but so far I just haven't had the time to spend more time than I already have troubleshooting this particular minor app. It worked fine until one day my dev server crashed with about 10 apps on it (unattended upgrades had updated containderd.io which broke the app and caused nginx to panic since it could no longer find the crashed app) but it sounds like the upgrade isn't to blame... just that it now enforces proper syntax, which should have probably failed before.

So until I have some free time, I'll probably keep the last "working" version pinned.

[thaJeztah]

This issue should be fixed by opencontainers/runc#3004, which relaxed the validation in runc, and is part of runc v1.0.0, which is included in containerd v1.5.3 (and up) and containerd v1.4.7 (and up).

[pdhar-tibco]

Building this Dockerfile on the latest Docker 3.5.2 mac Os also shows the problem. Is this fixed on mac ?

ARG IMAGE_TAG="undefined"

RUN apt-get update && apt-get install -y jq vim

VOLUME [ /usr/src ]
WORKDIR /usr/src
CMD [ "bash" ]

#COPY content /opt/
RUN { \
  set -vx; \
  echo "done"; \
}

build command

docker build -t test-docker .

result

...
Removing intermediate container 8ac750f81f58
 ---> f35985ea2242
Step 4/7 : VOLUME [ /usr/src ]
 ---> Running in a7f9b7cfdd1f
Removing intermediate container a7f9b7cfdd1f
 ---> 884b15bc5c03
Step 5/7 : WORKDIR /usr/src
 ---> Running in 9a499880fb40
Removing intermediate container 9a499880fb40
 ---> d85cdfeb5a16
Step 6/7 : CMD [ "bash" ]
 ---> Running in 2145cb2fb967
Removing intermediate container 2145cb2fb967
 ---> ef3b45f40dee
Step 7/7 : RUN {   set -vx;   echo "done"; }
 ---> Running in ca93d27d5241
OCI runtime create failed: invalid mount {Destination:[ Type:bind Source:/var/lib/docker/volumes/77f6f42bf4284e2ad943b57cc4d6abd87ea8fb91b0e58f97df2e27b733c4cc4c/_data Options:[rbind]}: mount destination [ not absolute: unknown```

[thaJeztah]

Upcoming patch release of Docker Desktop will include runc v1.0.0

But in your case looks like there's a mistake in your Dockerfile;

VOLUME [ /usr/src ]

Is not valid JSON, so will create 3 volumes: [, /usr/src, and ]

Change it to:

VOLUME ["/usr/src"]

(with quotes)

or

VOLUME /usr/src

[pdhar-tibco]

Tested and it works. Thanks for your explanation.

[yanjunli76]

Downgrading worked for me, too.
That said, I wanted to elaborate on my specific scenario.
In my case I was reviving a script from 2019 that worked perfectly back then.
The first time the script worked fine. Then, when forcefully removing the container and re-creating it with the exact same script, I encountered the same error as @mesquita:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:: Type:bind Source:/var/lib/docker/volumes/9e1805936631f35e4895a18840a0f56c21669099b5967954b176585e9af7a2da/_data Options:[rbind]}: mount destination : not absolute: unknown. 

It just wouldn't budge.
Downgrading "solved" it, for now.

Why did it work the first time, though?
That's another question that will hopefully be answered when the real fix will have been implemented.

Hi @theAkito, did you figure out why it worked at the first time. I also faced the same issue. I could run a container without any issue at the first time. After a while I tried to restart the container, then it showed the invalid mount error. The error is expected since I use the containerd(1.5.2-0ubuntu120.04.2) and runc(1.0.0rc95-0ubuntu1~20.04.2). But I couldn't figure out why it worked at the first time.

[theAkito]

@yanjunli76

I'm sorry, didn't figure it out and I doubt I will be able to.