I can only connect to published port on ip where service is running

[B-RH]

Hi,

I'm current running swarm in development with one manager + 3 workers. Manager (M) and 1 worker (W1) are on google cloud. 1 worker (W2) on hostingpalvelu virtual server and 1 on aws (W3). There is one overlay network where all services are connected to. All os are centos7. All behaving as same with docker 17.5 and 17.12 ce. I have previously this same problem, it solved, maybe update on some docker or centos package. Now I have it again.

I started one service on M. Container starts at M. Connect from browser works ok to M and W1 (both on google). From W2 and W3 I cannot connect. On those hosts I can run some small app, which is using same port and on that I can connect ok.

I drained M and it moves to W1. All works same as above.

I drained W1. service moved on W2. I cannot connect anymore from M or W1 and not still W3 ip+port. Same small app as above, cannot run anymore on W2 - port reserved. On W2 ip+port works ok.

I drained W2 and service moved to W3 and all works same as W2 above. Just connection from W3 p+port

Then I update M & W1 back to active and drained W3. Service moved to M and connection from M & W1 are ok, but W2 and W3 cannot been connected. Small app can run on W2 and W3, and it responsing on same port, which should be swarm?

At same time on workers, there is problem with docker dns ex. on W3, I cannot ping/connect service by name from container, which is running on M or W1 or W2.

[B-RH]

Hello again,

I made a new test and add all steps here. Hope, somebody can confirm this?

host 1 (manager):

    * google cloud
    * machine type: small
    * CentOS 7 x86_64 built on v20180129
    * ports at project-level:
        - TCP port 2377
        - TCP and UDP port 7946
        - TCP and UDP port 4789
        - TCP port 15672
    * yum update
    * yum install yum-utils device-mapper-persistent-data lvm2
    * yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    * yum install docker-ce
    * systemctl start docker
    * docker swarm init --advertise-addr host-1-public-address:2377
    * docker network create -d overlay test_net

host 2 (worker):

    * amazon ec2
    * machine type: t2.micro
    * centos7-201711141630 - ami-084a9b70
    * ports at security-group:
        - TCP port 2377
        - TCP and UDP port 7946
        - TCP and UDP port 4789
        - TCP port 15672
    * yum update
    * yum install yum-utils device-mapper-persistent-data lvm2
    * yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    * yum install docker-ce
    * systemctl start docker
    * docker swarm join --token worker-join-toke host-1-public-address:2377

On host 1:

  * docker node ls 
       ID                            HOSTNAME                                      STATUS              AVAILABILITY        MANAGER STATUS
       0rkz707epfk6joyo88kwdy7wo     ip-xx-xx-xx-xx.us-west-2.compute.internal     Ready               Active
       lt6kt5m3yboywtnt73inlgzdq *   t-docker                                      Ready               Active              Leader
   * docker service create --network test_net -p 15672:15672 -p 5672:5672 -p 5671:5671 --name rabbitmq rabbitmq:3-management
   * docker ps
       CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
       417c0b89794f        rabbitmq:3-management   "docker-entrypoint.s…"   5 minutes ago       Up 5 minutes        4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   rabbitmq.1.9epvb2sn8zpvaor3ujkoifyur

On host 2:

    * docker ps
        CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                                                 NAMES

With some pc:

    * http://host-1-ip:15672/ --> ok
    * http://host-2-ip:15672/  --> This site can’t be reached host-2-ip refused to connect.

On host 1:

    * docker node update --availability drain t-docker
    * docker ps
        CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                                                 NAMES

On host 2:

    * docker ps
        CONTAINER ID        IMAGE                   COMMAND                  CREATED              STATUS              PORTS                                                 NAMES
        00bd1bc5e3f5        rabbitmq:3-management   "docker-entrypoint.s…"   About a minute ago   Up 35 seconds       4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   rabbitmq.1.x3m1ud1bilsgnmjmbixq595yt

With some pc:

  * http://host-1-ip:15672/ --> This site can’t be reached host-1-ip refused to connect.
  * http://host-2-ip:15672/  --> ok