dockerd fails to start if there are routes for all private networks although a custom bip is configured

[coder-hugo]

Description

All our hosts have configured routes for the following networks:

10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

So the default networks that dockerd tries to use for the docker0 bridge are already in use. Therefore I specify a custom bip which docker should use instead.

Steps to reproduce the issue:

1. Configure routes for the networks given in the description on a host
2. Start dockerd with a bip which isn't part of one of the networks

Describe the results you received:

Relevant part of the logs:

Error starting daemon: Error initializing network controller: list bridge addresses failed: no available network

If I configure the docker0 bridge manually dockerd starts sucessfully.

Describe the results you expected:

dockerd shouldn't check for available networks if a custom bip is specified.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:04:27 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:04:27 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.05.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Kernel Version: 4.9.0-0.bpo.3-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.799GiB
Name: xxx
ID: NSV4:YWNW:67GF:7SQN:YUD6:2NSE:U6VH:KNQW:L45F:S5PM:ZQWW:4TSW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[jhaprins]

Hello,

Today a colleague of mine asked me if I had changed something on the network because his Docker configuration was suddenly giving a lot of problems. At first I did not know what he was talking about but after some questions, it slowly became clear to me that he had problems starting his docker environment when his VPN connection to the office was online. I looked at the error message that he received and I saw the following: "Error initializing network controller: list bridge addresses failed: no available network". This was very strange because the network he had configured in his daemon.yaml looked like this:
{
"default-address-pools":
[
{"base":"10.10.0.0/16","size":24}
]
}

In our corporate network we have a lot of RFC1918 networks, a few in the 10.x.x.x/8 range, a lot in the 172.16.0.0/12 and 192.168.0.0/16 ranges. But nothing that collides with above ranges, and even if something would collide, it was all local on his workstation where he was developing and testing some monitoring systems, and he is completely free to use whatever network he wants to use locally, as long as he doesn't interfere with the corporate network. On the VPN router I have a default set of routes set for RFC1918 networks pointing towards the corporate routers, so everyone can reach the internal corporate networks without having to worry about anything. The firewalls will take care of the rest.

I started debugging the error message and did some Google searches and I found a lot of people complaining about exactly this same problem. Some example tickets:
docker/for-linux#123
#35121
#33925 (this ticket, currently open, so I leave a comment here.)

At first the error didn't make any sense to me because:

• a network is available
• the configured network is not directly connected so docker is not able to say that it should not use it.
• even if an overlapping network is used somewhere else, a more specific route would be configured locally and this should prevent any routing issues.

But then I thought about something. What if the docker code, searching for free networks, takes the local routing table and checks the configured network against EVERY route in the routing table. If something matches or overlaps the route in the routing table it gives this error. At first I thought that this couldn't be true because this would always fail because a default route of 0.0.0.0/0 would always match. But what if this default route is filtered out in the code for this specific reason. Then this hypothesis could be the truth.

I started testing locally on my own system, first I reproduced the error:

• Setup my docker daemon with the same configuration
• Had my normal local routing table without VPN.
• Started docker and this worked fine.

The resulting routing table:
default via 192.168.178.1 dev enp62s0u1u1 proto static metric 1024
10.10.0.0/24 dev docker0 proto kernel scope link src 10.10.0.1 linkdown
192.168.178.0/24 dev enp62s0u1u1 proto kernel scope link src 192.168.178.74 metric 100

Then I started my VPN. The result was 3 extra routes:
10.0.0.0/8 via 192.168.2.1 dev tap0 proto static metric 50
172.16.0.0/12 via 192.168.2.1 dev tap0 proto static metric 50
192.168.0.0/16 via 192.168.2.1 dev tap0 proto static metric 50

I then stopped my docker daemon and tried to start it again, and indeed I received the same error. So I could reproduce the problem, now for my hypothesis: "Does the code check EVERY route in the routing table, filtering out the default route."

To test this I did the following:
I removed the default route and replaced it by 2 more specific routes that are together the whole internet:
0.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1
128.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1

My routing table then looks like this:
0.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1
128.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1
192.168.178.0/24 dev enp62s0u1u1 proto kernel scope link src 192.168.178.74 metric 100

The only difference between this state and a clean state of my system, is not having a default route, but having two routes that are together the default route of my system. Now I tried to start the docker daemon again. If the daemon starts fine my hypothesis is wrong and I have to continue my search. If the daemon fails then my hypothesis must me correct because the default route is the only difference in my local configuration.

And indeed, I received the same error again. Now I'm sure there is absolutely no reason to give this error because:

• I don't have the 10.10.0.0/16 network anywhere in my home network
• I have a routing table that only routes for 192.168.178.0/24 and the internet

This also proves my hypothesis that every route in the routing table is being checked against the configured network, filtering out the default route. If any route matches the configured network, the configuration is rejected.

This is a bug in the docker code. The code should be changed to only match routes with "scope link" because these routes are directly connected and would be a problem when you start a docker daemon with an overlapping network configuration. Any route that is not "scope link" should be ignored because those routes could be:

• Injected by DHCP
• Injected by a routing protocol
• Injected by a VPN config.
• Less specific behind a router somewhere remote

There is one corner case where you could give a warning or maybe an error. This is when there is an equal or more specific route that is not "scope link". Because this could result in routing issues to other systems. But even then, I would make it configurable because it could very well be that this is intentional.

I'm not a developer but a network and systems engineer, so I am not able at the moment to provide a patch for this problem, but one of my colleagues thought that he had already found some parts of the code. So maybe ........

The version I have tested this with is: Docker version 19.03.13, build 4484c46d9d

Cheers,
Jan Hugo Prins

[jhaprins]

One of my colleagues has done some more digging, and he thinks the reason the code actually works at the moment, the way it works, is because the code that is going through the routing table is not able to interpret the default route entry because it is not an IP address/netmask. The result is a NIL entry in the routing table object, or whatever it is called in Go, and this in turn results in that the default route is not interfering with the network selection. If he default route would have returned 0.0.0.0/0 for example, then the code would have failed from the beginning.

He also found that it is probably very simple to put a filter in the code that iterates through the routing table, and make sure that it only takes routes with a link scope. If you do this, a lot of people will be a lot happier.

[mpictor]

@GordonTheTurtle still an issue with more recent versions. Please add tag(s) to match.

$ docker --version
Docker version 19.03.13, build 4484c46

also with

$ docker --version
Docker version 20.10.0-rc1, build 5cc2396

[vincentbernat]

The check seems to have been introduced in b0d046a. It seems to be a refactoring and there is no mention of such a change. Maybe it the check should not be done at all. It is not robust as a "conflicting" network can appear after Docker has started, making the debug all the more difficult (sometimes it work, sometimes not). It seems difficult to be smart about that. I would suggest that Docker stays dumb and just use the expected network. Less code, less problems.

PR from @deepy would also fix my use case, but it relies on the coincidence that this check would solve most issues while link-scoped links have no direct relations with the importance of not shadowing the route. A static route, like 192.168.115.0/24 could be equally as important to not shadow it. I mean, it opens the door to adding even more code to try to fix a problem that maybe people even didn't have.

[pdolinic]

IT appears Docker currently is unable to fulltunnel in OpenVPN due to routes - as mentioned here several times, and this should be seriously worked upon? The solution is to get your admin to switch to splittunnel.

1. Log onto your OpenVPN Server
2. open /etc/openvpn/server/server.conf
3. Swap from fulltunnel to splittunnel via via ; this: ;push redirect-gateway def1 bypass-dhcp
4. systemctl restart openvpn-server@server.service
5. Whatever didn't work before should work now