SWARM Mode - Docker binds ingress ports always on 0.0.0.0

[eyenx]

Description

We have multiple hosts using swarm mode.

Some of this hosts have two or more external IP addresses. Docker swarm services with published ports always bind to 0.0.0.0:. Even after launching dockerd with --ip=xxx.xxx.xxx.xx.

Container not started in swarm mode, are binding to the correct IP. Services in Swarm are always binding published ports to every IP known by the Host (eno1,eno2 and lo). We cannot use the second IP with a port already taken by docker swarm.

Steps to reproduce the issue:

1. Create a Docker Service in Swarm with published Ports
2. netstat -utelpna | grep
3. Port is bound on 0.0.0.0:

Describe the results you received:

Started dockerd with --ip argument:

root 4869 1.1 10.0 2749972 102920 ? Ssl 12:36 0:12 /usr/bin/dockerd -H fd:// --ip=139.59.215.247 --swarm-default-advertise-addr=139.59.215.247

Swarm services bind published ports to every IP known by the system (0.0.0.0).

Describe the results you expected:

Expected swarm services to bind published ports only on IP 139.59.215.247.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 29
 Running: 4
 Paused: 0
 Stopped: 25
Images: 65
Server Version: 17.09.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 554
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: lbvavrurdt9edmfb44eshjbqk
 Is Manager: true
 ClusterID: 3uthkj04fn1rv2htzljm2imo9
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 139.59.215.247
 Manager Addresses:
  139.59.215.247:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-4-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.6MiB
Name: dev1
ID: CDVW:ZY3R:JMYQ:AW3L:CC6K:F7RL:JRZX:GPXQ:FOD6:4VTI:7TZV:7JDC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[cpuguy83]

The --ip option is for docker run -p, not swarm, and swarm binds on every node in the cluster.
You can use host-mode binding instead of ingress and specify an IP address in the binding.

See https://docs.docker.com/engine/swarm/services/#publish-a-services-ports-directly-on-the-swarm-node

Closing as this is expected behavior and is documented, but feel free to discuss.
Thanks! 👼

[eyenx]

Hi, Thanks.

So, there isn't a method where I can deploy a service in the swarm which binds on host1 to IP X and on host2 to IP Y? It has to bind to all addresses available on the host?

[cpuguy83]

@eyenx That seems correct as the service could be scheduled anywhere on the cluster.

[thaJeztah]

There's a tracking issue that shows which options are not (yet) supported for swarm services here: #25303. Issues about --ip can be found in #24170 (duplicated by #29816)

For publishing ports (slightly related) the -p <ip-address>:<host-port>:<target-port> option is not supported for services, see #26696 for that one

[eyenx]

Okay. Thank you very much.

[janober]

I am a little bit confused now. Checked now a lot of tickets but could not find what I was looking for. So hope somebody here can help me as it seems very closely related.
What I want is that all services of a swarm node do not listen to all IP addresses and to a specific one instead.
So my server has the IP 10.0.0.1 and 192.0.0.1 I want that it only responds to 10.0.0.1 . That would be for all services running on that node. I can understand that it is complicated on a per service level but for the whole node that should be not that hard.
Can that be done or is that also not possible?

[thaJeztah]

@janober on recent versions of docker you can specify a --data-path-addr when initialising the swarm (docker swarm init), or joining the swarm (docker swarm join). That option allows you to specify either an IP-address, or interface to use for networking data (separate from the "control" networking, that's used by managers to control the swarm)

It was implemented in this pull request: #32717

Note that this option can only be set on init or join, and cannot be modified afterwards

[janober]

@thaJeztah thanks a lot for the fast reply. That sounds like exactly what I am looking for. Sadly does it not work for me. I am running Docker version 17.09.0-ce, build afdb6d4. So that option is already there.

However when I start swarm like this:

docker swarm init --advertise-addr=x.x.x.x--data-path-addr=x.x.x.x

and then a container, for example ngnix like that:

docker service create \
  --name nginx \
  --publish 80:80 \
  nginx

I can not just access the web page via http://x.x.x.x I can still access it via the other IP address http://y.y.y.y

Do I understand or do something wrong?

[thaJeztah]

Let me double-check with @fcrisciani

[alex-sainer]

@janober just tried jour solution:

docker swarm init --advertise-addr=192.168.43.171 --listen-addr=192.168.43.171 --data-path-addr=192.168.43.171

also by specifying the interface-name:

docker swarm init --advertise-addr=eth1 --listen-addr=eth1 --data-path-addr=eth1

but docker-swarm is also still listening on the other interface eth0...

[fcrisciani]

@janober the flag data-path-addr is used only to separate data-path traffic from control-plane traffic, meaning that all the overlay traffic (VXLAN) if going to exit and enter on the interface specified by the data-path-addr knob. The flag does not change the listening address of swarm

@alex-sainer
the iptables rules that are configured for the ingress network are not scoped per interface:

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:10001
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:10001
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:30022
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30022

so will match any packet coming to that dst port.

[janober]

@fcrisciani Thanks for clarifying!
So then back to my rephrased original question. Is there really no way to change the listening address?

[janober]

Maybe also my later question is not very clear. I am also not talking about the "--listen-addr" because also that one does not change the "listening" I am talking about. I want to change the IP the docker services respond to.

[fcrisciani]

@janober at the moment there is no real way to avoid that, the listen-addr is a configuration parameter for swarmkit nodes on which IP/interface listen for connections coming from other peers.
What is the use case of it? One workaround that I can think of, that is definitely not ideal, but can work, is to add iptables rules in the DOCKER-USER chain that DROP all traffic going to a specific IP that you want to mask and to the ports that are being exposed.
Pro: it is feasible to implement. Cons: will prevent you from using services that are using the same port.

As an idea for a future improvement would be to create the iptables rules using the listening address if specified

[janober]

@fcrisciani Thanks for the answer. My use case is that my servers are part of two networks. An internal private network which can not be accessed from the internet (in which all servers are in and I via VPN) and an external one which can be accessed via the internet. I wanted to have a very simple but still secure way of accessing docker swarm services in the private network only (without having to deal with the complicated IP table stuff).

[surfdude75]

I have the same problem.

my solution was to run in host mode.

To be sure it would bind in the address I want in each docker node.

I have to remove docker_gwbridge from each node before join swarm and do something like

docker network create -o "com.docker.network.bridge.host_binding_ipv4"="192.168.1.151" docker_gwbridge

192.168.1.151 is the ip I want to expose in this node

On compose I was using

ports:
  - target: 80
    published: 80
    protocol: tcp
    mode: host

It would be GREAT have something like that for ingress.

And it sounds really simple to implement.

Hope we have it.

[jaguardev]

found a workaround:
sudo iptables -t mangle -A PREROUTING -i ens3 -p tcp -m tcp --dport 22222:22288 -j DROP

publish docker services ports to the "service" range (i.e 22222:22288) and drop packets on these ports for a public interface

now i can configure my frontend to map to the local interface
HAProxy in my case:

    acl app_1_acl hdr(host) -i example1.org
    acl app_2_acl hdr(host) -i example2.org

    use_backend app_1_bck if app_1_acl
    use_backend app_2_bck if app_2_acl

backend app_1_bck
    server app_1 127.0.0.1:22222

backend app_2_bck
    server app_2 127.0.0.1:22223

if i need to expose any port to public from the swarm (i.e. 25 - smtp), i can just use port out of "service" range...

fuh.... dirty hack

[ppetermann]

just to be sure: there is still no way of defining what IP ingress is listening for?

[thaJeztah]

@ppetermann correct; it's being tracked in #26696

[tuananh]

im not sure if my problem is related.

i can access all the services from every node via curl localhost:svc_port. however, it doesn't work with the LAN IP address

[MichaelBrenden]

time just slip-slidin away ... just to be sure: there is still no way of defining what IP ingress is listening for?

[PigNatovsky]

im not sure if my problem is related.

i can access all the services from every node via curl localhost:svc_port. however, it doesn't work with the LAN IP address

I have exactly the same issue - overlay network via publish. I was trying with and without host mode. In host mode docker-proxy binding is 0.0.0.0:port, otherwise dockerd is *:port. But still doesn't work.

I can connect from other hosts or locally via 0.0.0.0 or localhost:. Other services on the same node also cannot connect to published one.