New swarm overlay network issue

[Noodle05]

I'm testing docker 1.12 swarm, and having some issue with overlay network.

In three nodes (one manager, two workers) swarm environment, all running on ubuntu 16.04 with docker 1.12.
Create two alpine ssh server:

$ docker service create --name ssh1 --network ingress soniclidi/alpine-ssh
$ docker service create --name ssh2 --network ingress soniclidi/alpine-ssh

Found out which node they are running, and go to the node (ssh1 for example), execute command "ip addr show" shows container got two ip addresses from ingress network.

$ docker exec -it f15fac50e9e6 ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:0a:ff:00:07 brd ff:ff:ff:ff:ff:ff
    inet 10.255.0.7/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.255.0.6/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:aff:feff:7/64 scope link 
       valid_lft forever preferred_lft forever
22: eth1@if23: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.3/16 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:3/64 scope link 
       valid_lft forever preferred_lft forever

And docker inspect shows IP address is 10.255.0.7

$ docker inspect --format '{{.NetworkSettings.Networks.ingress.IPAMConfig}}' f15fac50e9e6
{10.255.0.7  []}

But internal DNS resolve ssh1 to 10.255.0.6

$ docker exec -it f15fac50e9e6 nslookup ssh1
Server:    (null)
Address 1: ::1 localhost
Address 2: 127.0.0.1 localhost

Name:      ssh1
Address 1: 10.255.0.6

And from ssh2, I can ping 10.255.0.7, cannot reach 10.255.0.6. But since DNS resolve ssh1 to 10.0.255.0.6, ssh2 cannot access ssh1.

I tried to create another overlay network, got same result.

Anybody know what I did wrong?

[michaelkrog]

This seems to be precisely what I experience too since I started testing swarm mode from rc4. I have not yet been able to get a swarm working on our cluster.

I think it must be a duplicate of #24234, but this description may be more to the point.

[vdemeester]

/cc @mavenugo @mrjana

[mavenugo]

@Noodle05 this works just fine when I tried your example. Can you please make sure the appropriate tcp/udp ports are opened across the cluster to make sure the overlay networking works fine ?

[Noodle05]

@mavenugo I'm sure it's not firewall issue. I even tried to turn off firewall. Also, if it's firewall issue, they shouldn't be able to ping each other at all. It's reachable, just on different IP.

If you looks at output of "ip addr show" inside container, two IPs it get have different netmask, 10.255.0.7/16 and 10.255.0.6/32. 10.255.0.7 is reachable, 10.255.0.6 is unreachable.

@mavenugo May I know you try to reproduce it, so I can give it a try?

My environment is: three Ubuntu 16.04 (latest packages) with docker 1.12.0 (Release version). I do have ufw, but port 2377/tcp, 4789, 7946 are opened. I also tried stop ufw, same thing.

[Noodle05]

Is this same issue as: #23855? If so, it's probably by design? Let me try to go to port directly, will report back.

[mavenugo]

@Noodle05 thats correct. I missed the ping able to reach the individual container-ip case. this is not a firewall issue. BTW, the 2 IPs that you see in the container is not causing this issue. It was added to satisfy another requirement.

The issue here seems to be more of IPVS not load-balancing correctly. Can you share the ipvsadm output from the service-sandbox ? The sandboxes are stored in /var/run/docker/netns. You can use nsenter into the service-sandbox and get the ipvsadm output.

am using latest docker master (equivalent to 1.12.0) in 14.04 ubuntu. Can you share your daemon logs with debugs enabled ? I have to assume that there is some issue with ipvs configurations.

[Noodle05]

Quick update.

1. I'm still working on getting ipvsadm (not familiar with it yet).
2. The 2 IPs looks ok, I guess it's by design.
3. "docker inspect " and "docker service inspect " give different IPs, service IP not pingable, but can be used to connect to specific port.
4. But the problem is: containers that connect to same overlay network randomly reachable/unreachable to each other. Sometime, internal DNS server not working (cannot lookup by service name), sometime cannot reach an alive container.

I setup multiple overlay networks, for example: application service has three connections: connect to backend (for database), ldap-backend (for ldap server) and frontend (for proxy),
proxy connect to frontend and ingress, and so on.
Whenever this randomly unreachable happened, the only way to make it work again is restart docker service. But restart docker service could make other container unreachable.

@mavenugo If you can provide a link to document on how to get ipvsadm, it will be great helpful.

Thanks

[jhmartin]

@Noodle05 https://www.server-world.info/en/note?os=Ubuntu_14.04&p=lvs shows 'apt-get install ipvsadm', and on CentOS 7 it is 'yum install ipvsadm'.

[Hosweay]

I am seeing a similar issue with an overlay network and services added to it.

docker network create -d overlay --subnet 10.10.0.0/16 redis_net
docker service create --name redis --replicas=1 --network redis_net redis
docker service create --name flask --replicas=1 --network redis_net --with-registry-auth -p 5000:5000 internal_registry:flask:1.0

The redis port of 6379 is reachable from the flask container only when using the redis container IP. When trying to reach via the VIP or the VIP's dns record, it returns no route to host.

# curl redis:6379
curl: (7) Failed to connect to redis port 6379: No route to host
# curl 10.10.0.3:6379
-ERR wrong number of arguments for 'get' command

Should the exposed port of the redis container be available over the VIP? It seems like it should.

All hosts in the swarm are running 1.12.1

[root@ip-172-18-55-74 ~]# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

[garthk]

Similar: after internal DNS failure resolving service names, I restarted the Docker service and DNS worked again. Docker 1.12.1 (23cf638) in swarm mode.

[Hosweay]

I have restarted the service before and it didn't help. Also, to clarify the DNS resolves to the VIP just fine. DNS resolution isn't broken. It's that the routing of the service's port using the VIP is not working. Even bypassing DNS and using the VIP address directly has the same no route to host.

[garthk]

… and again.

$ docker exec -ti 3b13024ff420 su - $USER -c 'ssh hg@$CONTAINERNAME'
ssh: Could not resolve hostname $CONTAINERNAME: Name or service not known

All three hosts are at ten days' uptime. The target container has been up for six days. The other containers have been up for 47 hours. They could resolve and reach the target back then. Now two of the three can't.

@mavenugo can you make any sense of the following? I ran it on an affected host. The target container has 10.255.0.4 on ingress and 10.0.0.6 on its --network. The service container trying to reach it can ping 10.0.0.6, but can't resolve it by name. dig reports SERVFAIL.

# for net in `ls -1 /var/run/docker/netns` ; do /tmp/nsenter --net=$net ipvsadm ; done
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  258 rr
  -> 10.255.0.4:0                 Masq    1      0          0         
FWM  261 rr
  -> 10.255.0.9:0                 Masq    1      1          0         
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  256 rr
  -> 10.0.0.2:0                   Masq    1      0          0         
  -> 10.0.0.4:0                   Masq    1      0          0         
FWM  263 rr
  -> 10.0.0.4:0                   Masq    1      0          0         
  -> 10.0.0.10:0                  Masq    1      0          0         
FWM  271 rr
  -> 10.0.0.6:0                   Masq    1      0          0         
FWM  275 rr
  -> 10.0.0.12:0                  Masq    1      0          0         
FWM  276 rr
  -> 10.0.0.11:0                  Masq    1      0          0         
FWM  278 rr
  -> 10.0.0.13:0                  Masq    1      0          0         
FWM  280 rr
  -> 10.0.0.4:0                   Masq    1      0          0    

Meanwhile, after restarting the Docker daemon: the service container on the other affected host can resolve the target container, but is getting the wrong IP address for it. Restarting Docker again doesn't help. Restarting Docker on the host with the log output above… ok, now it's getting the wrong address, too. Frustrating, but consistent.

The daemons on all three hosts are binding to the same network for listening. I'll try adding listen-addr anyway in case there's some magic.

[garthk]

• Rebuilt entire Swarm with --listen-addr and --advertise-addr set during swarm init and swarm join
• docker network create for the shared overlay
• docker service create of all the services again
• On same host as the target service container, docker exec -ti $CID ssh $USER@$TARGET and ping $TARGET works with the CID of the origin service container
• On one other host, the ssh works. The ping gets Destination Host Unreachable reported from the origin service container IP address. I don't need ping, though, so it's fine.
• On the last host, the ssh fails with Could not resolve hostname; ping reports unknown host; ping to the target container IP address gets Destination Host Unreachable; ssh to the target service container IP address reports No route to host.
• service docker restart… now the ssh succeeds.

You announced that swarm is "production-ready". Righto, then: hit me with the troubleshooting steps that'll quickly isolate what surely must be a configuration error. What information do you need?

[UPDATE: struck out jibe.]

[larsha]

Same issue here, Docker 1.12.1 and 1 manager and 2 nodes running Ubuntu 16.04. Sometimes services can connect and resolve to each others hostnames, sometimes not. Restart of docker and server sometimes works.

[michaelkrog]

Also same here. Ubuntu 16.04, docker 1.12.1, 1 manager and 2 workers om digital ocean. Communicating via private network. Overlay network is encrypted.

It works only sometimes. And when it works I still see a few requests time out between services (like 1 of 30 or so).

Edit
I should add that I have a very simple setup consisting of 2 services both based on nginx. The one service acts as a proxy for the other which serves a plain html site. When it works I will see most requests for html/JavaScript/images go through except some random few request that will time out because the proxy cannot connect upstream.

Also, when I first created the swarm I did not have the firewall setup correctly for overlay networking between the nodes. I fixed it, but it could not heal itself. I then recreated the swarm, saw overlay networking being replicated in the swarm, but to no avail. Issues persist.

Edit 2
The issues I hade with requests timing out between services is lkely caused by nginx's caching of DNS Records. The open source edition does NOT honor DNS TTL.

When using proxy_pass to an URL this is what happens in Nginx:

NGINX caches the DNS records until the next restart or configuration reload, ignoring the records’ TTL values.

Source

[bargenson]

Same thing here. I have two Swarm clusters running on Google Compute Engine with Docker 1.12.1 and an overlay network. The Operating System is Ubuntu 16.04 Xenial. Each cluster run one master and two workers.

When I started my services right after the clusters' creation, everything was running fine. After 5 days, my first cluster started to complain that some services were not able to communicate with each other. After 11 days, same issue with my second cluster.

Remove and recreate the services don't fix the communication issue. Restart the Docker service on each node doesn't resolve the issue (at least, not every time).

My only service publishing ports outside of the cluster is an Nginx. The others are mostly Java Web Applications and MongoDB.

[cpuguy83]

@bargenson Curious... are these nodes' time in sync with each other?

[thaJeztah]

ping @bargenson ^^

[bargenson]

@cpuguy83 Yes they are.
It happened again yesterday on my cluster. I just restarted all my nodes and it fixed the issue.
It just happened again today and restart didn't fix the issue. Remove and recreate my services didn't fix the issue. Removing and recreating my services AND my network seem to have fix the issue for this time.
I gonna try to switch from vip to dnsrr to see if I get the same issue.

[xiaods]

need testing.

[garthk]

@cpuguy83 date +%H%M%S.%N' on the three nodes:

165354.445382800
165354.493114787
165354.493211239

I trust a 5ms delta is acceptable.

What's our next step for data gathering? I'll put in effort to help nail this down if the maintainers point me in the right direction.

[garthk]

Forgot to mention: the reason I'm back is that it has, of course, happened again… this time shortly after a reboot of all three nodes. Restarting the affected daemon resolved the incident but not, of course, the issue.

[thaJeztah]

ping @mrjana PTAL!

[mrjana]

@garthk If it is a test setup, would you mind testing with 1.12.2-rc3 here: https://github.com/docker/docker/releases/tag/v1.12.2-rc3 ?

It fixes a number of potential issues.

[garthk]

@mrjana I've upgraded to 1.12.2 on all three nodes. I haven't seen anything break in the last hour. We'll have to leave the ticket open a couple of weeks given the unpredictable nature of the problem and lack of any diagnostic steps to nail it down, e.g. "X happens after Y, so watch for another Y and see if X recurs".

[mrjana]

@garthk Thanks for upgrading and reporting back. I am fine with keeping the issue open for some time. And yes we could've gone and debugged the issue step by step, but given that a lot of people hit the same kind of issues which are fixed in 1.12.2 it would just be unscalable to debug all them in an individual basis especially since the problem happened at a very low level.

For sure this means that we probably need better metrics exposed by docker to scale troubleshooting and we will look into that more closely in the future.

[virtuman]

looks like issue is back. Also running centos 7.2 on 1.12.2 and docker services cannot communicate between each other on same overlay network

[xiaods]

Oops, do you have a try upstream branch.

[ghost]

Those of you updating to docker 1.12.2, is that the client only? Or the client and various daemons? I'm finding when I update the daemons my services are unable to start.

Would love to figure out a solution to this problem. It's killing me that my services can't talk to each other when they are on different nodes.

[bourquep]

I am facing this issue as well, with Docker version 1.12.3, build 34a2ead (via CoreOS 1235.0.0).

I created a 3-node CoreOS cluster using their Alpha CloudFormation template, then edited the EC2 security group to allow all inbound traffic to all ports (TCP+UDP) from that security group.

Then, pretty much followed the steps in this MongoDB replica set tutorial to create the swarm, network, volumes and services.

I now have my 3 services running, one on each node. When I docker exec -it bash into any of these containers, I get Destination Host Unreachable when trying to ping any of the 2 other containers. Hostname resolution to vip seems to work.

I'll be happy to provide any diagnostic logs/info you might need.

[thaJeztah]

@bourquep the CoreOS fork of Docker contains various modifications, which can play a role here, so it would be worth to report that with them. Also note, that if your setup is using encrypted overlay (--opt encrypted), that protocol 50 (ESP) is open, see https://docs.docker.com/engine/swarm/swarm-tutorial/#/open-ports-between-the-hosts

When you ping, are you pinging to a container or to the service? Pinging the VIP across hosts may not work, but you should be able to connect (e.g. curl)

[bourquep]

@thaJeztah I have all ports opened between the hosts. I was trying to ping a service from within a container. After rebooting all 3 nodes, I was able to initiate the MongoDB replicaset and have the 3 mongos processes talking to each other. Still can't ping though, but at least they connect.

I'll let the CoreOS team know about this.

[michaelkrog]

@bourquep When you open 'all ports' then you refer to the TCP layer. But for encrypted overlay network Protocol 50 of the IP layer needs to be open too.

[MaxDesiatov]

I have this issue reproduced on Ubuntu 16.04, Docker 1.12.4. I have only one encrypted overlay network, services are unreachable. DNS seems to work though, but the IPs can't be pinged and services are unavailable on their respective ports.

[thaJeztah]

@explicitcall could you open a new issue with details? Perhaps it's a different issue, and this thread is getting lengthy; may be good to open a "fresh" issue

[sanimej]

@explicitcall Do you have the service reachability issue only if the network is encrypted ? Can you try with the service connected to an unencrypted network ?

[MaxDesiatov]

@sanimej great suggestion, can't reproduce with unencrypted network. Will keep an eye on it for the next few days in the light of previous reports about the issue being sporadic.

[aboch]

@explicitcall As suggested by @michaelkrog (also check #26523), please make sure ip protocol 50 packets (ESP) can be freely exchanged across your hosts. Make sure your firewall allows them.

[MaxDesiatov]

@aboch does this work differently with encryption enabled/disabled? I have this working without encryption without any other to changes to firewall settings or anything else

[aboch]

It does not work differently. The encryption option causes the dataplane to be encrypted. But for that to work, the encrypted packets (ESP) need to not to be blocked by your firewalls.
If you do not need traffic across your containers to be encrypted, then you are good.

[mavenugo]

@thaJeztah I agree that this is becoming a catch-all issue and i think we should consider closing and locking it and encourage folks to open new issues.

[thaJeztah]

@mavenugo agreed

For anyone arriving here; have a look at the discussion above if there's a solution mentioned for your situation. If you're still having issues, please open a new bug report with as much details as possible (following the bug report template; https://raw.githubusercontent.com/docker/docker/master/.github/ISSUE_TEMPLATE.md), so that we can look into your issue.

There are many possible causes for overlay networking not working; many are related to configuration, but if you think it's not a configuration issue but a bug, open a new issue.