Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port forwarding on host not working anymore since 20.10.15 #43603

Open
brainexe opened this issue May 16, 2022 · 3 comments
Open

Port forwarding on host not working anymore since 20.10.15 #43603

brainexe opened this issue May 16, 2022 · 3 comments
Labels
area/networking version/20.10

Comments

@brainexe
Copy link

brainexe commented May 16, 2022 •
edited

Description
Since a upgrade to 20.10.15 we have strange issues with port forwarding. A downgrade to 20.10.14 is fixing the issue. On 20.10.16 is still happens.

Steps to reproduce the issue:

  • we start a socat container which opens a port on the host to be proxied to one of the containers.
  • From the host 0.0.0.0:49183 should get mapped to socat:2002 which is then forwarded. But it never reaches the socat process
  • inside the container everything works, a curl to the target port+the local socat port is working
  • on the host we just get curl localhost:49183 curl: (56) Recv failure: Connection reset by peer also via 0.0.0.0.
  • This other Stackoverflow report MIGHT be related. Same symtoms and also after 20.10.15 upgrade

Describe the results you received:

  • no other container is running at this time, also docker restart, purging all networks etc did not help.

Here are more logs, all ones from the host:

docker ps
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED              STATUS                        PORTS                                                                                                                                                                                                                                                                          NAMES
5b16a0d2680b   ourproxy:5000/alpine/socat:latest               "/bin/sh -c 'socat T…"   About a minute ago   Up About a minute             0.0.0.0:49185->2000/tcp, :::49185->2000/tcp, 0.0.0.0:49184->2001/tcp, :::49184->2001/tcp, 0.0.0.0:49183->2002/tcp, :::49183->2002/tcp, 0.0.0.0:49182->2003/tcp, :::49182->2003/tcp, 0.0.0.0:49181->2004/tcp, :::49181->2004/tcp, 0.0.0.0:49180->2005/tcp, :::49180->2005/tcp   socat
03a4684a9358   ourproxy:5000/jaegertracing/all-in-one:latest   "/go/bin/all-in-one-…"   12 minutes ago   Up 12 minutes             5775/udp, 5778/tcp, 14250/tcp, 14268/tcp, 6831-6832/udp, 16686/tcp                                                                                                                                                                                                             9yqcmrujhvwt_jaeger_1
...

The docker-proxy processes for the 49183 port:

ps aux | grep 49183
matze    14878  0.0  0.0   6144   892 pts/9    R+   12:28   0:00 grep --color=auto 49183
root     19489  0.0  0.0 1370352 7228 ?        Sl   11:57   0:01 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 49183 -container-ip 172.21.0.7 -container-port 2002
root     19495  0.0  0.0 1148388 3376 ?        Sl   11:57   0:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 49183 -container-ip 172.21.0.7 -container-port 2002

Strace of the docker-proxy. so the proxy kinda get a connect...but not more:

sudo strace -p 19489 -p 19495 -p 19650 -F
[pid 19502] accept4(4,  <unfinished ...>
[pid 19500] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 19503] epoll_pwait(6,  <unfinished ...>
[pid 19502] <... accept4 resumed> 0xc000048bd8, [112], SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
[pid 19503] <... epoll_pwait resumed> [{EPOLLIN|EPOLLOUT, {u32=3467071096, u64=140105300266616}}], 128, 0, NULL, 0) = 1
[pid 19502] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>
[pid 19503] epoll_pwait(6,  <unfinished ...>
[pid 19502] <... socket resumed> )      = 5
[pid 19500] <... nanosleep resumed> NULL) = 0
[pid 19502] connect(5, {sa_family=AF_INET, sin_port=htons(2002), sin_addr=inet_addr("172.21.0.7")}, 16 <unfinished ...>
[pid 19500] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 19502] <... connect resumed> )     = -1 EINPROGRESS (Operation now in progress)
[pid 19500] <... nanosleep resumed> NULL) = 0
[pid 19502] epoll_ctl(6, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3467070856, u64=140105300266376}} <unfinished ...>
[pid 19500] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 19502] <... epoll_ctl resumed> )   = 0
[pid 19502] futex(0xc00003ed50, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 19500] <... nanosleep resumed> NULL) = 0
[pid 19500] futex(0x5609d3690238, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 19503] <... epoll_pwait resumed> [{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP|EPOLLRDHUP, {u32=3467070856, u64=140105300266376}}], 128, -1, NULL, 0) = 1
[pid 19503] futex(0x5609d3690238, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 19500] <... futex resumed> )       = 0
[pid 19503] getsockopt(5, SOL_SOCKET, SO_ERROR,  <unfinished ...>
[pid 19500] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 19503] <... getsockopt resumed> [ECONNREFUSED], [4]) = 0
[pid 19503] epoll_ctl(6, EPOLL_CTL_DEL, 5, 0xc00004db2c) = 0
[pid 19503] close(5)                    = 0
[pid 19503] write(2, "2022/05/16 12:27:29 Can't forwar"..., 128) = 128
[pid 19503] epoll_ctl(6, EPOLL_CTL_DEL, 3, 0xc00004de24) = 0
[pid 19503] close(3)                    = 0
[pid 19503] epoll_pwait(6, [], 128, 0, NULL, 824634040320) = 0
[pid 19503] epoll_pwait(6,  <unfinished ...>
[pid 19500] <... nanosleep resumed> NULL) = 0

 docker exec -it 5b16a0d2680b ps
PID   USER     TIME  COMMAND
    1 root      0:00 socat TCP-LISTEN:2005,fork,reuseaddr TCP:master_1:8080
    8 root      0:00 socat TCP-LISTEN:2000,fork,reuseaddr TCP:postgres_1:5432
    9 root      0:00 socat TCP-LISTEN:2001,fork,reuseaddr TCP:redis_1:6379
   10 root      0:00 socat TCP-LISTEN:2002,fork,reuseaddr TCP:jaeger_1:14268
   11 root      0:00 socat TCP-LISTEN:2003,fork,reuseaddr TCP:socketserver_1:80
   12 root      0:00 socat TCP-LISTEN:2004,fork,reuseaddr TCP:socketserver_1:80
   30 root      0:00 ps

From socat container:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:2000            0.0.0.0:*               LISTEN      8/socat
tcp        0      0 0.0.0.0:2001            0.0.0.0:*               LISTEN      9/socat
tcp        0      0 127.0.0.11:35857        0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:2002            0.0.0.0:*               LISTEN      10/socat
tcp        0      0 0.0.0.0:2003            0.0.0.0:*               LISTEN      11/socat
tcp        0      0 0.0.0.0:2004            0.0.0.0:*               LISTEN      12/socat
tcp        0      0 0.0.0.0:2005            0.0.0.0:*               LISTEN      1/socat

Output of docker version:

docker version
Client: Docker Engine - Community
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Thu May 12 09:17:38 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.16
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.10
  Git commit:       f756502
  Built:            Thu May 12 09:15:44 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.4
  GitCommit:        212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc:
  Version:          1.1.1
  GitCommit:        v1.1.1-0-g52de29d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)

Server:
 Containers: 13
  Running: 7
  Paused: 0
  Stopped: 6
 Images: 48
 Server Version: 20.10.16
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc version: v1.1.1-0-g52de29d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.7.0-0.bpo.2-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 10.77GiB
 Name: XXXXX
 ID: XXXXXXX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

  • debian10 with 5:20.10.(15|16) debian package
@brainexe brainexe changed the title Port forwardning on host not working anymore since 20.10.15 Port forwarding on host not working anymore since 20.10.15 May 16, 2022
@cpuguy83
Copy link
Member

I am unable to reproduce this.
That said, there was exactly 1 change to the proxy in 20.10.15: #43360
I'm not sure how it relates.
Here's the diff: https://github.com/moby/libnetwork/compare/64b7a4574d1426139437d20e81c0b6d391130ec8..339b972b464ee3d401b5788b2af9e31d09d6b7da

@brainexe
Copy link
Author

we tested it with 20.10.17 and the same broken behavior than described above. Switching back to 20.10.14 fixed it again

@ralyodio
Copy link

Still happens for me on Manjaro 20.10.14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking version/20.10
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ralyodio @cpuguy83 @thaJeztah @brainexe