static ipv4 addresses not working when using overlay network and swarm mode (v1.13+)

[a-jung]

Dear all,
my docker swarm mode cluster keeps ignoring "ipv4_address:" definitions set in version-3 compose files. I try to get a consul cluster running on my swarm mode cluster. Consul needs to advertise an ip address, and you also need to define an ip address for the "consul workers" to join ("-retry-join="). With docker ignoring the fixed ipv4 addresses I set in the compose definition, you end up with consul containers advertising ip addresses they do not own. This will never lead to a working consul cluster.

Steps to reproduce the issue:

1. Docker Swarm running in Swarm Mode, 3 manager-worker nodes and 2 worker-only nodes
2. Deploy the following v3-yml-file:

version: "3"

services:

  px-consul-dsm01:
    image: consul:0.7.5
    command: [agent, -server, -bootstrap-expect=3, -advertise=10.0.111.3, -client=0.0.0.0]
    networks:
      px-consul-net:
        ipv4_address: 10.0.111.3
    volumes:
      - consul-px-tst:/consul/data

  px-consul-dsm02:
    image: consul:0.7.5
    command: [agent, -server, -bootstrap-expect=3, -advertise=10.0.111.4, -client=0.0.0.0]
    networks:
      px-consul-net:
        ipv4_address: 10.0.111.4
    volumes:
      - consul-px-tst:/consul/data

  px-consul-dsm03:
    image: consul:0.7.5
    command: [agent, -server, -bootstrap-expect=3, -advertise=10.0.111.5, -client=0.0.0.0]
    networks:
      px-consul-net:
        ipv4_address: 10.0.111.5
    volumes:
      - consul-px-tst:/consul/data

networks:
  px-consul-net:
    driver: overlay
    ipam:
      driver: default
      config:
        - subnet: 10.0.111.0/26

volumes:
  consul-px-tst:

Describe the results you received:

• The network px-consul-net is being created as expected, it comes up with the defined address range.

• The 3 consul containers are created and start up as well.

• Unfortunately they come up with random ipv4 addresses from the range defined in the compose file. - Not with the addresses I set in the "services" section.

• As the result, I get f.e. one consul container with the IP 10.0.111.7, but advertising the ip 10.0.111.3; a second consul container with the IP 10.0.111.5, advertising the IP 10.0.111.5 (accidentally the right one) and a 3rd consul container with the IP 10.0.111.3, advertising the IP 10.0.111.4. This will never work.

• Any -retry-join=<root agent ip> statement in the service definition file will never work.

Describe the results you expected:
I expected each of the 3 consul containers coming up with the ipv4 address I set in the service definition file, for example:

networks:
      px-consul-net:
        ipv4_address: 10.0.111.5

Additional information you deem important (e.g. issue happens only occasionally):
Happens each and every time I tried. Upgrading from docker engine v1.13.1 to the latest available did not help. I also installed all patches for the host system (apt-get update and apt-get upgrade followed by a system reboot). Did not help. Behaviour did not change.

Output of docker version:

Docker version 17.03.0-ce, build 60ccb22

I already tried this with docker engine 1.13.1. - Behaviour regarding the randomness of the ipv4_address: statement was the same.

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.03.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 7
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: hyk1yifcdibctr5pnytdg8hb7
 Is Manager: true
 ClusterID: k2dzk1kjcby74hrs2vshbtq2u
 Managers: 3
 Nodes: 5
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.123.123.37
 Manager Addresses:
  10.123.123.37:2377
  10.123.123.38:2377
  10.123.123.39:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-66-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 13.69 GiB
Name: t-azeurn-dsm01
ID: SOIK:RX3Q:AILR:KD4C:SFK2:X75S:27R3:W3VG:UV5D:MC6A:AP4W:3K62
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: ajunglsy
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
All nodes of this swarm are Azure VMs, size DS3_v2, running Ubuntu Linux 16.04.2 LTS:
Output of uname -a:

Linux t-azeurn-dsm01 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Output of lsb_release -a:

root@t-azeurn-dsm01:~/containerconfigs/consul-px# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial

Please investigate. This is really annoying. Thank you very much!
console_output_consul.txt
docker-stack-deploy.txt

[westsouthnight]

@a-jung IMHO you must to check -advertise= and try to use -advertise-wan= instead of advertise or together. Try to set debug option to you consul containers for get more information. Next, if i will be in same situation, i has been to try to check all env (firewalls, etc). And also, consul sometimes have different random situations which can get random results. You check that issue with other services or images?

[thaJeztah]

Static IP addresses for services are not yet supported, see #25303 (and #24170, #29816)

[thaJeztah]

Let me close this issue, because this is already tracked through the ones I linked, but feel free to add your use case there