Compose does not mount volume if I use relative path

[KES777]

Description

I think this is related to: #3715

When I use relative path at configuration it does not work, but if I use absolute path, then everything is OK

Steps to reproduce the issue:
Working compose file:

version '3.7'
services:
  grafana:
    image: grafana/grafana:latest
    container_name: chimera_grafana
    restart: "no"
    ports:
      - "8082:3000"
    user: '1000'
    volumes:
      - chimera-grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=asdfadsfafadf
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_DOMAIN=kes

volumes:
  chimera-grafana-data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/home/kes/work/projects/sidnet/chimera/Chimera-DevEnv/data/grafana'
      # device: './data/grafana' <<<< If I use this instead. Then I get errors

Describe the results you received:

...
compose.cli.verbose_proxy.proxy_callable: docker create_container <- (environment=['GF_SECURITY_ADMIN_PASSWORD=OGg5Pb4dvHMMJ2Z5', 'GF_USERS_ALLOW_SIGN_UP=false', 'GF_SERVER_DOMAIN=kes'], image='grafana/grafana:latest', ports=[('3000', 'tcp')], user='1000', volumes={'/var/lib/grafana': {}}, name='chimera_grafana', detach=True, labels={'com.docker.compose.project': 'chimera-devenv', 'com.docker.compose.service': 'grafana', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/home/kes/work/projects/sidnet/chimera/Chimera-DevEnv', 'com.docker.compose.project.config_files': 'docker-compose.yml', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.25.0', 'com.docker.compose.config-hash': 'b9fcaefc1104351753faad8037b33f6941e6ccfd43e77b2ff88ec455d0198899'}, host_config={'NetworkMode': 'chimera-devenv_default', 'RestartPolicy': {'Name': 'no', 'MaximumRetryCount': 0}, 'VolumesFrom': [], 'Binds': ['chimera-devenv_chimera-grafana-data:/var/lib/grafana:rw'], 'PortBindings': {'3000/tcp': [{'HostIp': '', 'HostPort': '8082'}]}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'chimera-devenv_default': {'Aliases': ['grafana'], 'IPAMConfig': {}}}})
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/containers/create?name=chimera_grafana HTTP/1.1" 500 192
compose.parallel.parallel_execute_iter: Failed: ServiceName(project='chimera-devenv', servCreating chimera_grafana ... error
compose.parallel.feed_queue: Pending: set()

ERROR: for chimera_grafana  Cannot create container for service grafana: failed to mount local volume: mount ./data/grafana:/media/kes/WORK-DATA/docker_images/volumes/chimera-devenv_chimera-grafana-data/_data, flags: 0x1000: no such file or directory
compose.parallel.parallel_execute_iter: Failed: <Service: grafana>
compose.parallel.feed_queue: Pending: set()

ERROR: for grafana  Cannot create container for service grafana: failed to mount local volume: mount ./data/grafana:/media/kes/WORK-DATA/docker_images/volumes/chimera-devenv_chimera-grafana-data/_data, flags: 0x1000: no such file or directory
ERROR: compose.cli.main.main: Encountered errors while bringing up the project.

Describe the results you expected:

...
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/containers/b0fbb89a8254ff654e6368bdd7d6825332c00b089f082daaafdeb5b1207b523a/start HTTP/1.1" 204 0
compose.cli.verbose_proxy.proxy_callable: docker start -> None
compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='chimera-Creating chimera_grafana ... done
compose.parallel.feed_queue: Pending: set()
compose.parallel.parallel_execute_iter: Finished processing: <Service: grafana>
compose.parallel.feed_queue: Pending: set()
Attaching to chimera_grafana
compose.cli.verbose_proxy.proxy_callable: docker events <- (filters={'label': ['com.docker.compose.project=chimera-devenv', 'com.docker.compose.oneoff=False']}, decode=True)
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.38/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dchimera-devenv%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker events -> <docker.types.daemon.CancellableStream object at 0x7f9ece388fd0>
chimera_grafana | Error checking server process execution privilege. error: could not get current OS user to detect process privileges
chimera_grafana | logger=settings t=2022-02-19T20:39:44.83+0000 lvl=info msg="Starting Grafana" version=8.4.1 commit=53f5c6a44c branch=HEAD compiled=2022-02-18T11:14:07Z

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

$ docker-compose --version
docker-compose version 1.25.0, build unknown

Output of docker info:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 20
  Running: 8
  Paused: 0
  Stopped: 12
 Images: 23
 Server Version: 20.10.12
 Storage Driver: btrfs
  Build Version: Btrfs v5.4.1 
  Library Version: 102
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.16.9-051609-generic
 Operating System: Linux Mint 20.3
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.514GiB
 Name: work
 ID: R4Y6:HHMO:QFLS:ALS3:RWAT:CBOX:PDNL:NQOH:5Q3S:DB5R:GE7P:XP55
 Docker Root Dir: /media/kes/WORK-DATA/docker_images
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

[KES777]

What I noticed, that if I use relative path, then it looks like:

drwx-----x 1 root root    28 Feb 19 22:46 chimera-devenv_chimera-grafana-data

drwxr-xr-x 1 root root  0 Feb 19 22:46 _data
-rw------- 1 root root 89 Feb 19 22:46 opts.json

But when I use absolute path, then:

drwx-----x 1 root root    28 Feb 19 22:52 chimera-devenv_chimera-grafana-data

drwxrwxr-x 1 kes  kes   62 Feb 19 22:29 _data
-rw------- 1 root root 141 Feb 19 22:52 opts.json

So this is some problem with permissions

[KES777]

If I use relative path without named volumes then also all works fine:

    volumes:
      - ./data/grafana:/var/lib/grafana

compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/containers/fa909831e3fb985d85c270b7a66b671394c961a40a2ba56343ab77a7d54065b2/start HTTP/1.1" 204 0
compose.cli.verbose_proxy.proxy_callable: docker start -> None
compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='chimera-Creating chimera_grafana ... done
compose.parallel.feed_queue: Pending: set()
compose.parallel.parallel_execute_iter: Finished processing: <Service: grafana>
compose.parallel.feed_queue: Pending: set()
Attaching to chimera_grafana
compose.cli.verbose_proxy.proxy_callable: docker events <- (filters={'label': ['com.docker.compose.project=chimera-devenv', 'com.docker.compose.oneoff=False']}, decode=True)
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.38/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dchimera-devenv%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker events -> <docker.types.daemon.CancellableStream object at 0x7fc4bc75ee50>
chimera_grafana | Error checking server process execution privilege. error: could not get current OS user to detect process privileges
chimera_grafana | logger=settings t=2022-02-19T21:02:33.76+0000 lvl=info msg="Starting Grafana" version=8.4.1 commit=53f5c6a44c branch=HEAD compiled=2022-02-18T11:14:07Z
chimera_grafana | logger=settings t=2022-02-19T21:02:33.76+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations"
chimera_grafana | logger=settings t=2022-02-19T21:02:33.76+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled."
chimera_grafana | logger=settings t=2022-02-19T21:02:33.76+0000 lvl=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini

Permissions are next:

drwx-----x 1 root root    28 Feb 19 23:02 chimera-devenv_chimera-grafana-data

drwxr-xr-x 1 root root   0 Feb 19 23:02 _data
-rw------- 1 root root 141 Feb 19 23:02 opts.json

Here, despite folder is root:root everything is working. But it seems require kes:kes if I use named volumes

[ndeloof]

This is expected: as you define volume driver option, those aren't processed by docker compose but passed to docker engine, and this one won't accept relative path (as there's no "working directory" in this context on engine side)

You don't have to declare a volume to bind mount a local path, just use :

    volumes:
      - ./data/grafana:/var/lib/grafana

[KES777]

I want to give name because want to see normal names when listing:

$docker volume ls
local     volume-name
local     volume-name

If I do not give name I get:

$ docker volume ls
DRIVER    VOLUME NAME
local     3a759cdf0d57b77b853d1fa2c35a4ba97a0d2d5314a4b3733ed419e9761c1d98
local     4a2c77592c52cd5b3e626ad5f4ead58fda005d1c7bd9c52af0fd1161d2833dab
local     07bb318bf0943bbfc9a71fed6bf6ed12f665491273bdbc6c46c1e6a3d3d05254
local     49eb61d926fe03b35c50e6866f43699d445a7d6ff25f2a1298ab3839184eebf4
local     096aaea991f1fc279198a11889cdf8f1b28dbba2974c825f3f83549b2633d084
local     392cfbe247032a6a17e0808f700d8bebc87dd5281d227b1d39602218051246a8

And this is unreadable =(

@ndeloof: So before passing to docker engine please expand relative path to full

[ndeloof]

before passing to docker engine please expand relative path to full

I don't think we want to include volume driver-specific code into docker-compose

Seems to me you get some confusion between volumes and bind-mount. While "local" driver allows you do do such a hack, a volume should be an abstract storage, so not declared with a local path. Either you want a volume (abstract) or you want to mount some local files, and then obviously volume ls won't be aware

[KES777]

I want to see readable names for mounts/volumes.

I use here local because I did not find other way to name this mount point

[ndeloof]

I got your point, but that's basically a corner-case we don't want to support.
A possible workaround is to use $PWD variable:

volumes:
  chimera-grafana-data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '${PWD}/data/grafana'

[Bessonov]

I got your point, but that's basically a corner-case we don't want to support.

Well, this was supported in the tool chain docker-compose -> docker all the time. Run in this issue after setup new environment with newest docker and docker-compose version. I still have the old env for proof/testing.

The workaround is OK and it works for me, but be aware that it breaks if you aren't in the same folder like your config and point to it with -f.

[qazhub]

Relative path was actually supported with the legacy docker-compose (the Python version), so this is a breaking change in docker compose plugin, is it documented somewhere?

[ndeloof]

Considering docker compose V1 had dedicated code for this use case, I've proposed a fix: compose-spec/compose-go#332