Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

17.06 L4 ingress network not accessable #34003

Closed
Cas-pian opened this issue Jul 7, 2017 · 4 comments
Closed

17.06 L4 ingress network not accessable #34003

Cas-pian opened this issue Jul 7, 2017 · 4 comments
Labels
area/networking area/swarm version/17.06

Comments

@Cas-pian
Copy link

Cas-pian commented Jul 7, 2017

Description
Since I fresh install docker-ce 17.06, I found one issue of ingress network.
Cause there's a conflict of ingress network in 10.255.x.x with my environment, I recreate the ingress network, and then create my own overlay network named myoverlay for service internal communication. A container of a service got 4 interface, one for ingress network, one for myoverlay, one for docker_gwbridge and lo.
Then I found that across the service containers, ping is ok on ingress network ip , but L4 TCP connection is timeout.

Steps to reproduce the issue:
1.install docker17.06 on 2 machine, swarm init...
2.recreate ingress network, create another overlay network

docker network rm ingress
docker network create -d overlay --ingress --subnet=10.10.0.0/16 --gateway=10.10.0.1 ingress
docker network create -d overlay --subnet 172.29.0.0/16 myoverlay
docker service create -d --name test-conn --replicas 2 --network serviceoverlay -p 3333:9999 tmp.domain/ubuntu_tools:16.04 httpserver -p 9999
  1. check the connection between 2 containers in the service
    network of one container(lo is ingored)
eth0      Link encap:Ethernet  HWaddr 02:42:0a:0a:00:05  
          inet addr:10.10.0.5  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:11 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:762 (762.0 B)  TX bytes:853 (853.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:ac:12:00:03  
          inet addr:172.18.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2576 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:529433 (529.4 KB)  TX bytes:1972 (1.9 KB)

eth2      Link encap:Ethernet  HWaddr 02:42:ac:1d:00:03  
          inet addr:172.29.0.3  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:36 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3069 (3.0 KB)  TX bytes:3027 (3.0 KB)

the network on anther container(lo is ingored) :

eth0      Link encap:Ethernet  HWaddr 02:42:0a:0a:00:06  
          inet addr:10.10.0.6  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:238 (238.0 B)  TX bytes:238 (238.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:ac:12:00:03  
          inet addr:172.18.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:648 (648.0 B)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 02:42:ac:1d:00:04  
          inet addr:172.29.0.4  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:35 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2978 (2.9 KB)  TX bytes:2908 (2.9 KB)
  1. check network connection:
    on container1:
# ping 10.10.0.6
PING 10.10.0.6 (10.10.0.6) 56(84) bytes of data.
64 bytes from 10.10.0.6: icmp_seq=1 ttl=64 time=0.695 ms
#ping 172.29.0.4
PING 172.29.0.4 (172.29.0.4) 56(84) bytes of data.
64 bytes from 172.29.0.4: icmp_seq=1 ttl=64 time=0.807 ms

but the level 4 is not ok on ingress network:

# nc -vz 172.29.0.4 9999
[172.29.0.4] 9999 (?) open
#nc -vz 10.10.0.6 9999
10.10.0.6: inverse host lookup failed: Unknown host
(UNKNOWN) [10.10.0.6] 9999 (?) : Connection timed out
#curl http://172.29.0.4:9999/test
ok
#curl http://10.10.0.6:9999/test
curl: (7) Failed to connect to 10.10.0.6 port 9999: Connection timed out

Describe the results you received:
network not accessable on ingress L4

Describe the results you expected:
the network is ok in both ingress and user-defined overlay network

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 8
 Running: 2
 Paused: 0
 Stopped: 6
Images: 94
Server Version: 17.06.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 161
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: xllggtgi6jh2a4u8ph44pv3le
 Is Manager: true
 ClusterID: b5tgcqm3x9ufct1crguthsuc0
 Managers: 2
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 192.168.98.166
 Manager Addresses:
  192.168.98.166:2377
  192.168.99.41:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.10.0-26-generic
Operating System: Linux Mint 18.1
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.701GiB
Name: vostro
ID: Q5QZ:5FOG:MORS:J5TR:PXQ3:UBGZ:MS5P:LFJ7:ELLJ:HLON:Z34K:FNUL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://registry.docker-cn.com/
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
os info:
fNo LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
uname -r : 4.4.0-34-generic

the ufw is disabled
@thaJeztah
Copy link
Member

This is by design; The ingress network cannot be used for container-container communication. The ingress network is used for the routing mesh (i.e., routing external traffic arriving at "node A" to a container running on "node B").

Only containers attached to the same custom network are allowed to communicate; given that the ingress network is automatically attached when publishing ports, allowing communication over the ingress network would be a security issue.

I'll close this issue because this is not a bug, but feel free to continue the conversation

@Cas-pian
Copy link
Author

@thaJeztah Great thanks for your reply!
I got it, but I found that the interface order in container is random, for example, a service with exposed port 9999 and network useroverlay, the interface order in these networks are random: userovelay, ingress, docker_gwbride, why not a fixed order?

@thaJeztah
Copy link
Member

That's a known issue; IIRC networks are connected in alphabetical order (thus determining the interface order), but there's many situations where this may not be the case.

Also see #25181, and docker/compose#4645

With the "csv" style syntax for networks being worked on (see #31964, and PR's docker/cli#62 (for services), and docker/cli#156 for docker run), perhaps it would be possible to add an additional configuration option to specify the interface name

@Cas-pian
Copy link
Author

@thaJeztah Thanks! I will take a look at that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking area/swarm version/17.06
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thaJeztah @Cas-pian @GordonTheTurtle