cant access published port from anywhere except localhost on some containers

[alec-jensen]

Description

On some containers, I can only connect with the published port from localhost. Doesn't work if I try connecting from the local IP.

Reproduce

1. run a container with a published port

Expected behavior

I should be able to connect with the port

docker version

Client: Docker Engine - Community
 Version:           26.1.0
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        9714adc
 Built:             Mon Apr 22 17:07:06 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       c8af8eb
  Built:            Mon Apr 22 17:07:06 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.1.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 28
  Running: 22
  Paused: 0
  Stopped: 6
 Images: 234
 Server Version: 26.1.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-18-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.54GiB
 Name: debian
 ID: 6KIY:SEKL:HWB7:QNQS:CRK7:Z75X:TJLW:GVTK:YIBN:FV6B:JZEI:QCP3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

The server where I am encountering this issue is running Debian 12

[akerouanton]

Hi @alec-jensen, thanks for reporting.

Could you try enabling debug mode as described here and paste all logs related to iptables please?

You can also try https://github.com/akerouanton/iptables-tracer. It will show you all the iptables rules matched by a packet and which one is dropping them.

[alec-jensen]

Nothing in the docker logs after enabling debug. I'm not sure what logs relate to iptables. Here's what the iptables-tracer output:

INFO[0000] Waiting for trace events...                  
        mangle PREROUTING NFMARK=0x0 IN=wlp2s0 (changed by last rule)
                DEFAULT POLICY
                => ACCEPT
        nat PREROUTING NFMARK=0x0 
                MATCH RULE (#1): -m addrtype --dst-type LOCAL -j DOCKER
                => DOCKER
        nat DOCKER NFMARK=0x0 
                MATCH RULE (#34): ! -i br-f4096ac58599 -p tcp -m tcp --dport 580 -j DNAT --to-destination 172.15.0.3:80
                => DNAT: --to-destination 172.15.0.3:80
        mangle FORWARD NFMARK=0x0 IN=wlp2s0 OUT=br-0c0339c8ab11 (changed by last rule)
                DEFAULT POLICY
                => ACCEPT
        filter FORWARD NFMARK=0x0 
                MATCH RULE (#2): -j DOCKER-ISOLATION-STAGE-1
                => DOCKER-ISOLATION-STAGE-1
        filter FORWARD NFMARK=0x0 
                DEFAULT POLICY
                => ACCEPT
        mangle POSTROUTING NFMARK=0x0 
                DEFAULT POLICY
                => ACCEPT
        nat POSTROUTING NFMARK=0x0 
                DEFAULT POLICY
                => ACCEPT```