Docker Swarm on Docker Machine VirtualBox VM and Windows Path Issue

[Jitsusama]

Description

When running a Docker Swarm cluster on a docker-machine created VirtualBox VM, using Windows 7 as the Docker client OS, I have issues when using bind mounts in a docker-compose.yml file that is being consumed by docker stack deploy -c docker-compose.yml <service_name>.

What happens is that the relative path name for the bind mount defined in the docker-compose.yml file is not properly modified so that the VirtualBox boot2docker VM can map it to a shared volume. Instead, it looks like a standard Windows path which can't be handled.

If I feed this same docker-compose.yml file into docker-compose, it handles the path translation for me and the bind mount works fine.

Steps to reproduce the issue:

1. Create a swarm with the typical commands for a Windows 7 PC running Docker Machine via a VirtualBox VM.
2. Create a docker-compose.yml file with the following contents:

version: "3.3"

services:
  example:
    image: nginx
    volumes:
      - ./mount-me:/mount-me

2. Create a mount-me directory in the same folder as the docker-compose.yml file.
3. Run docker stack deploy --compose-file docker-compose.yml test

Describe the results you received:

C:\test>docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
rgepu13o9erq        test_example        replicated          0/1                 nginx:latest
C:\test>docker service ps test_example
ID                  NAME                 IMAGE               NODE                   DESIRED STATE       CURRENT STATE             ERROR                              PORTS
of40p5khc6sd        test_example.1       nginx:latest        local-docker-swarm-1   Ready               Rejected 3 seconds ago    "invalid bind mount source, mu…"
7zz8lfnsvnnn         \_ test_example.1   nginx:latest        local-docker-swarm-1   Shutdown            Rejected 8 seconds ago    "invalid bind mount source, mu…"
woan2400jv8r         \_ test_example.1   nginx:latest        local-docker-swarm-1   Shutdown            Rejected 13 seconds ago   "invalid bind mount source, mu…"
C:\test>docker inspect -f '{{json .Status}}' of40p5khc6sd
{"Timestamp":"2017-09-27T15:20:32.120107356Z","State":"rejected","Message":"resolving controller failed","Err":"invalid bind mount source, must be an absolute path: C:\\Users\\joel.gerber\\test\\mount-me","ContainerStatus":{},"PortStatus":{}}
C:\test>docker inspect  -f '{{json .Spec.ContainerSpec.Mounts}}' of40p5khc6sd
[{"Type":"bind","Source":"C:\\Users\\joel.gerber\\test\\mount-me","Target":"/mount-me"}]

Describe the results you expected:

I expected the container to come up like it does with docker-compose.

Here's a working example using the same docker-compose.yml file as was illustrated in the replication steps:

C:\test>docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating network "test_default" with the default driver
Creating test_example_1 ...
Creating test_example_1 ... done
C:\test>docker-compose ps
     Name              Command          State   Ports
------------------------------------------------------
test_example_1   nginx -g daemon off;   Up      80/tcp
C:\test>docker inspect  -f '{{json .Mounts}}' test_example_1
[{"Type":"bind","Source":"/c/Users/joel.gerber/test/mount-me","Destination":"/mount-me","Mode":"rw","RW":true,"Propagation":"rprivate"}]

Additional information you deem important (e.g. issue happens only occasionally):

The issue is 100% reproducible. If I manually create the service with docker run, while specifying the mount path, I have to write it out as /c/Users/joel.gerber/test/mount-me in order for it to work. I believe the issue is that docker stack deploy, when utilizing the relative path specified in the docker-compose.yml file, does not properly modify the path before sending it to the boot2docker Docker daemon. As a result, he cannot load the path, which causes service creation to fail.

Output of docker version:

C:\test>docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:30:30 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:51:55 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

C:\test>docker info
Containers: 3
 Running: 2
 Paused: 0
 Stopped: 1
Images: 24
Server Version: 17.06.0-ce
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 52
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: 479d9m5cxfpwo7k9v69rjg36o
 Is Manager: true
 ClusterID: eh5ca6gcyx4vcwxi3sr47zpdi
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 10.0.2.15
 Manager Addresses:
  10.0.2.15:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.74-boot2docker
Operating System: Boot2Docker 17.06.0-ce (TCL 7.2); HEAD : 0672754 - Thu Jun 29 00:06:31 UTC 2017
Architecture: x86_64
CPUs: 1
Total Memory: 1.955GiB
Name: local-docker-swarm-1
ID: 2SA3:PKTC:5VMU:CYQN:5SMD:XSZ7:MT77:43DD:EQNW:JJHS:WYH3:SFNP
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 45
 Goroutines: 151
 System Time: 2017-09-27T15:23:32.824962567Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Client OS: Windows 7 Business Edition
Docker OS: boot2docker established with docker-machine running on VirtualBox

[Jitsusama]

I'm sure you're all quite busy, but is there a chance that someone could take a look at this? This behaviour makes it impossible to have a docker-compose.yml file that can be shared amongst members of my development team when I want to test operation within a local swarm instance, as we all use Windows as our development platform.

[thaJeztah]

/cc @dnephin @vdemeester

[dnephin]

Oh, I realized it's because this is running from a windows client on a linux swarmkit worker.

I'm not sure where this translation is supposed to happen, but the client really doesn't have enough information to translate paths. It has no idea what platform will receive the task.

[friism]

@dnephin I suspect this is a carry-over from docker-compose up - in that case, this behavior is kind-of what you want.

I agree it generalizes poorly to docker stack deploy.

cc @dgageot who's deep in this lore.

[dnephin]

I guess this is something that would need to be translated by Docker for Windows. That's really the only place it makes sense.

[friism]

cc @ebriney who might better remember how this part of the system works

[thaJeztah]

I think the problem is roughly that;

• bind-mounts are performed on the host where the container runs
• for docker-compose, which historically targets "developers", it is assumed that the daemon either runs on the same host as the client (or boot2docker/docker for mac/docker for windows), and the local path matches a path on the daemon's host to make it feel "as if" the container is running locally and it just works.

docker stack deploy is targeted at production, and for a swarm, which means the daemon will basically never run on the same host; also when deploying a stack, you don't know up front on which the containers will run. Translating a local path to an absolute path, and "hoping" the same path exists on the host that the container is deployed may thus be the wrong assumption for that.

Not sure what the best approach is for that; make it behave the same as docker compose, or document that absolute paths are the recommended approach when deploying to a swarm (even more so, because when using services, the host path is not automatically created)

[Jitsusama]

@thaJeztah, the problem with requiring absolute paths is that the development environment is no longer reproducible.

I'm in the situation where I want my development environment to be swarm, so I can perform configuration/secret sharing and various other swarm only features without manual hacking being required by the local developer, thus keeping the production and development environment consistent, which is supposed to be one of the main selling points of Docker.

[tdaniely]

Seeing same issue, just trying to up a local haproxy on DfW in single node swarm.

Funny thing is when I inspect the service I see the right path:

"Mounts": [
  {
    "Type": "bind",
    "Source": "/host_mnt/c/Workspace/devops/Services/haproxy/etc",
    "Target": "/usr/local/etc/haproxy",
    "ReadOnly": true
  }
],

But on the container it's a windows path:

"Mounts": [
  {
    "Type": "bind",
    "Source": "C:\\Workspace\\devops\\Services\\haproxy\\etc",
    "Target": "/usr/local/etc/haproxy",
    "ReadOnly": true
  }
],

The yaml:

version: "3"

services:
  haproxy:
    image: haproxy:alpine
    ports:
      - 80:80
      - 443:443
      - 25:25
    volumes:
      - .\etc:/usr/local/etc/haproxy:ro

I tried all kinds of paths, windows paths, linux paths, path in the moby box. None of them actually work. I get what stack does and what it's intended for but IMO there's no reason that this shouldn't just work.

[dazinator]

Same issue on windows server 2019, plus docker desktop, plus swarm, plus linux containers.
Trying to bind mount x:/ into a linux container where x:/ is mapped to an azure file share on every node in the swarm. It works fine with docker compose. Fails with docker stack deploy

version: "3.2"
services:
  rabbit:
    image: rabbitmq:3-management
    container_name: rabbit
    ports:
      - "15672:15672"
      - "5672:5672"
    volumes:
      - x:/conf/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf

There are 2 nodes in the swarm, both windows server 2019. The manager is in windows container mode when I created the swarm. The worker is in linux container mode, when I joined it to the swarm.
The x:/ drive has been added as an allowed "shared drive" under docker desktop on both nodes when switching to linux container mode in docker desktop.
I've tried everything I can think of regarding path formats etc.. I'm now in the process of ditching windows server 2019 and trying out windows 10 as I have a feeling this may not be an issue with the WSL2 based backend.
I also tried to use the platform attribute in the docker compose file but that doesn't seem to be a valid attribute for version 3.X of the compose spec, or if it does, it wasn't added until version "3.9" - and sadly docker stack commands don't support compose version "3.9" on this setup.

[thaJeztah]

@dazinator docker compose, when using Docker Desktop may be converting Windows paths to their corresponding path inside the wsl2 environment.

If the daemon is running on wsl2, you probably need to access the drive through some /mnt/xx location where the daemon can access it; https://superuser.com/questions/1128634/how-to-access-mounted-network-drive-on-windows-linux-subsystem/1261563

[dazinator]

@thaJeztah

I have just rebuilt 2 x fresh windows 10 vm's on Azure, using latest docker CE install with WSL2 backend to create a simple swarm, and I have the same issue. What follows is more information about the setup, along with all the different variations of path formats I tried and the error for each

VM's

OS: Windows (Windows 10 21h1-pro)
Azure VM Size: Standard D2s v3 (2 vcpus, 8 GiB memory)
Both VM's have their own E:/ drive attached, with a file located at E:/a/a.conf
WSL2 installed on both.
Windows containers enabled on both via:

Enable-WindowsOptionalFeature -Online -FeatureName $("Microsoft-Hyper-V", "Containers") -All

Installed docker desktop - using wsl2 backend.
- Enable experimental features in docker desktop for docker compose v2
- Enable experimental features in docker desktop -> docker engine.

The swarm was then initialised on VM 1 by:-

• Switching it to "Windows Container" mode
• Running docker swarm init --advertise-addr X.X.X.X
  So VM 1 became the manager node.

VM 2, which has docker desktop still running in "linux container" mode, was then joined to the swarm successfully, after making sure windows firewall ports were open on both machines.

Failed attempts

I am running the following steps on the manager node to try to deploy a compose file to the swarm that uses a readily available linux container, and tries to mount a file from the host into it. With each attempt, I repeat the following:-

• Update the compose.yaml located at E:/compose.yaml to set the 'source' argument in the volume binding to some attempt at mounting the file that exists on both windows hosts, at E:/a/a.conf.
• From a powershell terminal, with working directory as E:/
  • execute docker stack deploy -c compose.yaml test
  • then check the logs for errors with docker stack ps --no-trunc test
  • make a note of the volume mount error
  • delete the stack with docker stack rm test

Here is the compose.yaml

version: "3.9"
services:
  rabbit:
    image: rabbitmq:3-management
    container_name: rabbit
    ports:
      - "15672:15672"
      - "5672:5672"
    networks:
      - workers  
    volumes:
      - type: bind
        source: E:/a/a.conf
        target: /etc/rabbitmq/a.conf    

networks:
  workers: {}

Here is a summary of all the values I have tried for the source path along with the errors.

Source	Error
E:\a\a.conf	invalid bind mount source, must be an absolute path: E:\a\a.conf
E:/a/a.conf	invalid bind mount source, must be an absolute path: E:/a/a.conf
E\a\a.conf	invalid bind mount source, must be an absolute path: E:\E\a\a.conf
E/a/a.conf	invalid bind mount source, must be an absolute path: E:\E\a\a.conf
/E:/a/a.conf	invalid mount config for type "bind": bind source path does not exist: /E:/a/a.conf
/E/a/a.conf	invalid mount config for type "bind": bind source path does not exist: /E/a/a.conf
/mnt/e/a/a.conf	"invalid mount config for type "bind": bind source path does not exist: /mnt/e/a/a.conf"
a\a.conf	invalid bind mount source, must be an absolute path: E:\a\a.conf
a/a.conf	invalid bind mount source, must be an absolute path: E:\a\a.conf
/a/a.conf	invalid mount config for type "bind": bind source path does not exist: /a/a.conf
//E/a/a.conf	invalid mount config for type "bind": bind source path does not exist: //E/a/a.conf

Outcome

Can anyone please explain if what I am trying to do is supported, and if so, what the working volumes specification is to mount this file? Maybe there are paths I have not attempted.

[dazinator]

@thaJeztah
I think I've got to the crux. I've created a new issue here: #42837
When you have a hybrid swarm, containing linux and windows nodes, the ability to docker stack deploy windows containers or linux containers that use a bind mount, is impacted by whether the current leader is a windows node or a linux node. If the current leader is a linux node, then you cant deploy any windows containers that bind mount from x:/ for example, even if that path exists on all the windows nodes that the container would be placed on. If you force a windows leader, it succeeds, but then deploying linux containers with bind mounts from /x fails with the same issue. In this instance x:/ on windows nodes is mapped to a file share, and /x on linux nodes is mapped to the same file share (smb) but I don't think that aspect really matters, I think its more about how paths get translated depending on what the current leader is, impacts what workloads can presently be deployed - windows or linux.