Bug [26.0.1, build d260a54] - Cannot access service

[osinnowo]

Description

I upgraded to the latest build (Docker version 26.0.1, build d260a54) and when I run docker compose up, everything looks fine, service(s) is up, but I can't access these service over the browser or any other means. On the browser, it keeps loading for a long time and then finally stop, it seems as though request isn't hitting docker engine.

Reproduce

See sample docker-compose.yml:

version: '3.8'
services:
  webserver:
    image: nginx:latest
    ports:
      - "8080:80"
    volumes:
      - ./html:/usr/share/nginx/html
    networks:
      - webnet

networks:
  webnet:

When docker compose up is ran, it setup the service successfully with no crash but then the webserver service would be unreachable, this is also applicable my current project that have been running on docker for quite a while now, stopped just until now.

Any thought why this is happening?

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           26.0.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        d260a54
 Built:             Thu Apr 11 10:53:25 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       60b9add
  Built:            Thu Apr 11 10:53:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 26.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: lll4mt0gap31z4vpzjjf4hlxy
  Is Manager: true
  ClusterID: 0fx7kz55i081dx2dbc5qpf649
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 104.131.102.60
  Manager Addresses:
   104.131.102.60:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-174-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.759GiB
 Name: ubuntu-s-1vcpu-2gb-nyc3-01
 ID: ab97309c-968a-4630-a99e-213d7da5a34c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: footballbuzz
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional Info

No response

[tkimball83]

This issue is not exclusive to @osinnowo as the problem hit me as well. I downgraded docker to the latest 25.0 release and all is well.

[mdima]

Same here, I am having network issues accessing the containers after the upgrade to Docker version 26.0.1, build d260a54 on Ubuntu 22.04.4.

I can provide additional information if needed.

Thanks,
Michele

[robmry]

Hi all - thank you for reporting. We'll need some more info to understand whether you're all seeing the same issue, and where the problem or problems are.

@osinnowo ... in the nginx example:

• Are you trying to access the web server from the docker host as http://127.0.0.1:8080, or different address or host? (Does the localhost address work?)
• Can you access the web server from the docker host, using the container's own address (from docker inspect) and port 80?
• Can you ping the nginx container's address from the docker host?
• Are you using a firewall, perhaps "ufw"?
• Are there any suspicious messages in the nginx container's log, or the docker engine logs?
• If you can share any logs, and output of docker container inspect, docker network inspect, ip a, iptables-save, iptables -nvL, iptables -t nat -nvL perhaps there will be a clue in there somewhere.
• (We might need to do some tcpdump-ing to see where packets are going missing.)

@tkimball83, @mdima - similar questions ... what does your environment look like, what commands are you using? Are you using IPv4 like @osinnowo, or IPv6 like @thomascrha? If you can share logs or other diagnostic info, that might help.

@thomascrha ... can you share any info about the IPv6 routing issues? How have you configured IPv6, what works from where etc? (Info as above, including ip6tables versions of the commands, might be useful.)

[mdima]

Hello @robmry ,
well, I checked and I have no related logs inside the containers. The issue I am facing is that at random intervals the containers are not reachable from the LAN, the reverse proxy cannot reach them. Before version 26 I did not have any issue.
The issue involves IPv4 and Ubuntu 22.04.4. The firewall is disabled.
If you tell me what specific information you need I can provide it.

Thanks,
Michele

[robmry]

Thanks @mdima - early versions of 25.0.x had an issue with containers getting duplicate MAC addresses, which caused intermittent connectivity - although I'd have expected that to cause problems in 25.0.x too. It might be worth checking docker container inspect output for duplicate MAC addresses anyway - deleting and re-creating the containers is the way to solve that problem.

Just to double-check, the failures are intermittent over the lifetime of a single container? (It's not that intermittently newly instantiated containers don't work?)

When there's an outage, and containers aren't accessible from the LAN - are they accessible from the docker host that's running them?

Which sort of networking are you using, are the containers on a custom bridge network (as they would be in the original post's compose/nginx example)?

If you're running multiple containers(?), do they all see the outage at the same time?

Which version did you upgrade to 26.0.1 from? Did you try reverting to a 25.0.x version, and did that help?

What sort of frequency/duration do the outages have? (If it works for a few seconds then fails for a few, that should be a bit easier to track down than a short failure in a long time.)

If possible, it'd be good to capture the output of the commands I mentioned above when it's working and when it's broken for comparison.

[mdima]

Dear @robmry , yes, you're right. I inspected my containers and I found two MAC collisions (involving 4 containers in total).
Now I fixed them, I will wait for the next connectivity issue on the containers to follow your check-list.
In the while I can just say that I don't remember the version 25.0.x I upgraded from and I did not revert.

Thanks,
Michele

[robmry]

Ah, great! Thank you for the update, hopefully that'll have solved the problem in your case.