Swarm stack using short-syntax for config doesn't work

[markfqs]

Description

Deploying a swarm stack using configs with short-syntax fails to launch container.

OCI Runtime (containerd) complains about not using an absolute mount destination. This behaviour has appeared after containerd upgrade from 1.4.4 to 1.4.6:

May 28 13:55:04 dock1 dockerd[629]: time="2021-05-28T13:55:04.709159795+02:00" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: invalid mount {Destination:redis.conf Type:bind Source:/dock/local/containers/3acbb6eba08475597b389fb46e025b3b77a6daf3439af8f50e7f7971e782819c/mounts/secrets/n39yd0pjjvqnj1w5j1c1ndtwe Options:[rbind ro]}: mount destination redis.conf not absolute: unknown" module=node/agent/taskmanager node.id=irqb65g5x8kwnz0kdy2y6r5ia service.id=omsa1oe5ens2wnh5nd6pglip6 task.id=h0h7ea9msv2unf6c1ru6h6f23

Steps to reproduce the issue:

1. Create a compose YAML file using short-syntax config as simple as:

---
version: '3.5'
configs:
  redis.conf:
    file: configs/redis.conf
services:
  cache:
    image: redis:alpine
    command:
      - redis-server
      - /redis.conf
    configs:
      - redis.conf

2. Deploy the stack

Result:

• Service never deploys, container/task not created
• Dockerd logs show the reason:

`May 28 13:55:04 dock1 dockerd[629]: time="2021-05-28T13:55:04.709159795+02:00" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: invalid mount {Destination:redis.conf Type:bind Source:/dock/local/containers/3acbb6eba08475597b389fb46e025b3b77a6daf3439af8f50e7f7971e782819c/mounts/secrets/n39yd0pjjvqnj1w5j1c1ndtwe Options:[rbind ro]}: mount destination redis.conf not absolute: unknown" module=node/agent/taskmanager node.id=irqb65g5x8kwnz0kdy2y6r5ia service.id=omsa1oe5ens2wnh5nd6pglip6 task.id=h0h7ea9msv2unf6c1ru6h6f23

`
Results expected:

• Service deployed
• Container/task having the config on / as usual

Additional details

• This was working before.
• Documentation shows short syntax as valid
• This behaviour has appeared after containerd upgrade from 1.4.4 to 1.4.6
• Either containerd issue or problem on how docker manages the container runtime.
• The obvious (tested good) workarround is to use long-syntax and explicitily set a target with an absolute path:

configs:
  - source: redis.conf
    target: /redis.conf

Docker version:

Docker information
Client: Docker Engine - Community
Version: 20.10.6
API version: 1.41
Go version: go1.13.15
Git commit: 370c289
Built: Fri Apr 9 22:47:17 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.6
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8728dd2
Built: Fri Apr 9 22:44:17 2021
OS/Arch: linux/arm
Experimental: true
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Docker info

Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.7.0)

Server:
Containers: 7
Running: 7
Paused: 0
Stopped: 0
Images: 8
Server Version: 20.10.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local nfs
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: xrgsutkuqfwgsif3xmiqgeewe
Is Manager: true
ClusterID: j5ooqivbvl6hxwt972fc8ne09
Managers: 5
Nodes: 5
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.24.231.246
Manager Addresses:
10.24.231.246:2377
10.24.231.247:2377
10.24.231.248:2377
10.24.231.249:2377
10.24.231.250:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.83-v7
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 926.1MiB
Name: dock2
ID: 575M:62SE:WWJ6:3FA7:OYOD:IXMZ:JDTU:J3E2:T4ST:V7GR:TVXF:5UD5
Docker Root Dir: /dock/local
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
l3jane.fleet.host=dock2
l3jane.fleet.env=prod
l3jane.fleet.size=4.micro
l3jane.fleet.storage=mmc
Experimental: true
Insecure Registries:
raft:5000
127.0.0.0/8
Registry Mirrors:
http://fleet:5050/
Live Restore Enabled: false
Default Address Pools:
Base: 172.20.0.0/16, Size: 24

[peron]

We have the same issue in our Swarms, including a Production Swarm, which are now failing to deploy containers. Apart from the mentioned containerd version change, we also have a change in runc version, in case that is of any relevance.

[thaJeztah]

This looks to be due to a breaking change in runc; a fix/workaround for this was merged in #42370, which will be included in the upcoming docker 20.10.7 patch release. As a workaround, you can either downgrade to an older version of the containerd.io package, or specify the target path for the config to be an absolute path

[thaJeztah]

this was fixed in 20.10.7, and the validation was relaxed for runc v1.0.0 (through opencontainers/runc#3004)

closing this one