On 12.1, swarm service running on one node cannot resolve hostname of service running on another node

[anthonyserious]

On 12.1, swarm service running on one node cannot resolve hostname of service running on another node

With a simple 2-node (including the one master node), and services running on the same overlay network, services containers running on one node can't resolve the hostname of containers running on the other node. I have tried several different overlay network subnets including 10.x/24, 192.168.x/24, 172.21.x/24.

1st node is docker 12.1 on Ubuntu, 2ns is docker 12.1 on Mac OS X El Capitan.

NOTE: the choice of mongo and redis images here is arbitrary - it's the same behavior experienced with proprietary node.js services running on centos:latest.

Steps to reproduce the issue:

• Create overlay network:
  docker network create -d overlay --subnet=10.127.127.0/24 test6
• Start a redis container named "redis" on the swarm master node:
  docker service create --network test6 --name redis --constraint "engine.labels.location == cloud" redis:latest
• Start a mongodb service named "mongo" on a non-master node
  docker service create --network test6 --name mongo --constraint "engine.labels.location == local" -p 27017:27017 mongo:latest
• ping redis from the mongo container
  From the second node:

➜ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
d17fabae45dd        mongo:latest        "/entrypoint.sh mongo"   6 seconds ago       Up 5 seconds        27017/tcp           mongo.1.3ctt7lmtdhefjrr3qj2jkj0f5
➜ docker exec -it d17fabae45dd /bin/sh
# ping redis
ping: unknown host
#

Describe the results you received:

# ping redis
ping: unknown host
#

Describe the results you expected:
The redis hostname should be resolved.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:
Master

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

Secondary

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7.1
 Git commit:   6f9534c
 Built:        Thu Sep  8 10:31:18 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 17:52:38 2016
 OS/Arch:      linux/amd64

Output of docker info:

Master

Containers: 19
 Running: 1
 Paused: 0
 Stopped: 18
Images: 6
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-202:1-525000-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.715 GB
 Data Space Total: 107.4 GB
 Data Space Available: 105.7 GB
 Metadata Space Used: 3.609 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge null host
Swarm: active
 NodeID: 6vwftswwb08n3j6a4gwfi8ax7
 Is Manager: true
 ClusterID: 43icrrhvmh9yhzchfta75dcrn
 Managers: 1
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.8.0.76
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-91-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.676 GiB
Name: ip-10-8-0-76
ID: ASN5:L337:YWJW:7JRT:5COD:7CQQ:IC2Z:FXTS:E7SR:VWOL:XBF2:ZHMX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: anthonyserious
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 location=cloud
Insecure Registries:
 127.0.0.0/8

Secondary

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 40
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 161
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: active
 NodeID: 3r5m8lmbrjjg9cxxaeb8ugrx1
 Is Manager: false
 Node Address: 192.168.65.2
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.20-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952 GiB
Name: moby
ID: VHJC:OVDV:6EY3:XCRJ:VYXL:I2S3:XNPL:FU4U:H4OR:7Z7N:ECLS:L7MQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 40
 Goroutines: 104
 System Time: 2016-10-07T17:46:19.131463959Z
 EventsListeners: 2
No Proxy: *.local, 169.254/16
Username: xxx
Registry: https://index.docker.io/v1/
Labels:
 location=local
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
1st node is docker 12.1 on Ubuntu 14.04 LTS, 2nd is docker 12.1 on Mac OS X El Capitan.

[mrjana]

@anthonyserious The problem is more likely to be the combination of using a Ubuntu VM and Docker4Mac. Can you provide more information about the environment i.e how the docker swarm was initalized like what were the exact command line options used in docker swarm init and docker swarm join? Also can you attach daemon logs from both the nodes?

[garthk]

Smells like #25266 to me.

[anthonyserious]

Thanks for the replies. I'm still trying some things and will get the requested info - apologies for the delay. @mrjana yes, it appears to be very much specific to docker4mac native, since the "equivalent" setup with a node on a Linux laptop works fine.

[xiaods]

yes, need testing PTAL!

[mrjana]

ping @anthonyserious. Any update on this? It seems trying to form a swarm cluster between a linux VM and docker4mac is not going to work out of the box. Not sure what we do with this bug. May be mark it as platform/desktop? /cc @justincormack

[anthonyserious]

Hi @mrjana, sorry for the delay here. We tried upgrading Linux and docker4mac to docker 12.2 beta, still having the same issue. Rebuilt the swarm and overlay network multiple times...same thing.

Something interesting, perhaps, is that for the overlay network, running docker network inspect test6 shows different results on each node. On the Linux host, it shows only IP addresses for the containers running there, and same for the docker4mac host - shows only the IP address for the one container running on that node. I would expect that I should see IP addresses for all containers using that network.

I cannot even ping the IP address for a container running on a different node, so it's not just the resolver failure. I was going to ask if using an external K/V store could possibly work, but given that hosts are unreachable across the overlay network even by IP address then I think it's not worth it.

One thought - is it possible that since docker4mac is running behind a hypervisor that I need to explicitly advertise the underlying IP address for docker? I'm not even sure how to modify commandline arguments for docker4mac, so would appreciate any guidance so I can play around with that.

Thanks!

[mavenugo]

@anthonyserious yes. this is the expected behavior for Docker4mac. I havent tried this myself though (which I think I should).

ping @justincormack has anyone tried such a setup with docker4mac and external linux system participating in a swarm cluster with a public facing --advertise-addr and direct port plumbing from mac -> docker daemon running in d4mac ?

@anthonyserious Also, docker network inspect showing just the local containers is a known limitation.

[thaJeztah]

Let me close this ticket for now, as it looks like it went stale.