Error starting daemon: Error initializing network controller: list bridge addresses failed: no available network

[hubotx]

Description

I have physical machine with Gentoo as host OS for Docker containers. I have compiled kernel using instructions on page https://wiki.gentoo.org/wiki/Docker#Kernel and I have installed Docker from Gentoo repository (see on the section Additional environment details (AWS, VirtualBox, physical, etc.)). I have set following USE flags:

>=app-emulation/docker-17.03.2 pkcs11 overlay device-mapper container-init btrfs aufs

I have emerged Docker and added it to boot level default in OpenRC init system. After compiling kernel and Docker I wanted to check if Docker is working so I typed docker info in terminal and I got error. I have decided to check what is wrong and I need your help with solving issue.

Steps to reproduce the issue:

1. Issue the docker version command.
2. Try get Docker system-wide informations using docker info.
3. Check Docker daemon status.
4. Check Docker logs.

Describe the results you received:
In the output of docker version (see below) you can see error Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. The same message appears if I try get Docker system-wide informations. The same error appears if I try run the same command prepending by sudo, so this error applies to daemon. I tried to check if there a mistake in Docker daemon privileges. Based on these messages I am able to say that maybe Docker daemon not running. I checked daemon status to make sure. Docker daemon is crashed. To see the reason, I looked at the logs.

Output of cat /var/log/docker.log:

pecan@tux ~ $ cat /var/log/docker.log 
time="2017-10-07T14:52:13.178261811+02:00" level=info msg="libcontainerd: new containerd process, pid: 32311" 
time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph migration to content-addressability took 0.00 seconds" 
time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your kernel does not support cgroup blkio weight" 
time="2017-10-07T14:52:14.434423960+02:00" level=warning msg="Your kernel does not support cgroup blkio weight_device" 
time="2017-10-07T14:52:14.434759986+02:00" level=info msg="Loading containers: start." 
time="2017-10-07T14:52:14.437180876+02:00" level=info msg="Firewalld running: false" 
Error starting daemon: Error initializing network controller: list bridge addresses failed: no available network

Describe the results you expected:

docker info should return Docker system-wide informations instead of Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

Expected output of docker version:

pecan@tux ~ $ docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.9.1
 Git commit:   f5ec1e2
 Built:        Sat Oct  7 14:50:59 2017
 OS/Arch:      linux/amd64

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

pecan@tux ~ $ docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.9.1
 Git commit:   f5ec1e2
 Built:        Sat Oct  7 14:50:59 2017
 OS/Arch:      linux/amd64
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of docker info:

pecan@tux ~ $ docker info
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of sudo docker info:

pecan@tux ~ $ sudo docker info
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of sudo service docker status:

pecan@tux ~ $ sudo service docker status
 * status: crashed

Additional environment details (AWS, VirtualBox, physical, etc.):
I am using Gentoo as Host OS for Docker containers. I have compiled kernel using instructions on page https://wiki.gentoo.org/wiki/Docker#Kernel and I have installed Docker from Gentoo repository.

Host system informations:

pecan@tux ~ $ uname -a
Linux tux 4.12.12-gentoo #8 SMP Sat Oct 7 13:58:47 CEST 2017 x86_64 Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz GenuineIntel GNU/Linux

I have disabled iptables and ip6tables because firewall is not actually properly configured. I am connecting to internet through VPN and I am using 8.8.8.8 and 8.8.4.4 DNS providers. I have running Tor and Privoxy daemons and I am using OpenRC init system.

[thaJeztah]

This is a duplicate of the issue you opened in moby/moby#35121 (comment) - let me close this one as well, but feel free to continue the conversation in the linked issue

[hubotx]

Sorry, my mistake. I wanted to place this issue here but I was something bad clicked.

[kinglion811]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

[Alan-Penkar]

For interested parties- @kinglion811 's solution worked perfectly for me on Ubuntu 17.10

[MiladMohebnia]

this woked for me

[rafaelsortoxo]

Anyone else having issues to install docker on Ubuntu 18.04 by following the official docs should try this to fix it because it helped me fix my installation and start docker daemon.

The error message could be very cryptic and show this

ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)

But after executing the binary manually you will see some errors like this

Error starting daemon: Error initializing network controller: list bridge addresses failed: no available network

And adding a docker0 bridge interface fixes the problems, like described above.

Regards,
Rafael

[manimkv]

This worked for me as well. Awesome.

[GabLeRoux]

I was getting the exact same error. Turned out my kernel got updated, rebooting the system worked.

[ScottKaiGu]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

why adding a docker0 bridge fixes the problems?

[yinrong]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

why adding a docker0 bridge fixes the problems?

+1 And why document does not include this

[ghost]

this solution worked. fedora 30.

[honkiko]

/reopen

[patlher]

had same issue ... no bridge and systemctl/journalctl report very cryptic on Debian 9.9
checking /var/log/syslog confirmed no docker0 bridge !
Kernel updated but not rebooted!
Guess what? reboot solves the issue lol
Debian doc should mention IMHO to reboot after apt update IF any kernel update occured ;-)

[jordanwalsh23]

@kinglion811's solution worked on Ubuntu 18.04.3 LTS (bionic) - dual boot windows/ubuntu. Hours of searching for this one!

[tlannigan]

We really need somebody to explain why @kinglion811's solution worked. To add, it worked for me.

[emmanuelnk]

Had this same issue. @kinglion811 solution also wored for me on Ubuntu 18.04 but im curious as to why?

[someonegg]

had same issue ... the ip route table was misconfigured

ip route

• 172.16.0.0/12 via xxx dev bond0 proto static metric 300
• 192.168.0.0/16 via xxx dev bond0 proto static metric 300

docker netutils.FindAvailableNetwork would failed with 'no available network'

[ryanisn]

In my case
I changed default-address-pools in /etc/docker/daemon.conf file in order to fix conflict IPs with VPN without having to specify network for each container, then I ran into this issue after reboot, as I didn't create enough IP pools for the existing docker networks
{ "bip": "192.168.1.1/24", "default-address-pools": [ {"base":"192.168.2.0/23","size":24} ] }

I then fixed the pool
{"base":"192.168.64.0/18","size":24}
used the fix from @kinglion811 to get docker running, ran 'docker network prune' to remove the old networks and recreated now ones, now everything works.

[hdhruna]

Suggestion from @kinglion811 worked on Manjaro 19.0 as well

[1beb]

It's not clear why this is closed if it still requires a solution (from so many people). Is this something that is not within docker's scope to fix?

[ccjjmartin]

I was able to fix this error by making sure the IP address range I gave for the pool wasn't already claimed. This is likely the issue for anyone else encountering it, either a VPN, you router, some other bridge is claiming that IP address range, pick a different one and restart and you will hopefully find success as I did.

The tricks mentioned above only gave temporary relief from the issue (creating a docker0 bridge manually) as soon as I restarted the service the errors returned again. So I don't recommend them.

The most helpful command I ran was: dockerd --debug

I did delete all of my existing networks using the tricks above (get it working then delete) but I now wonder if that was necessary.

[ToonSpinISAAC]

I don't know that IP addresses can be "claimed".

I am on Linux and for me the issue is that there is a route that routes 10.0.0.0/8 through a VPN. A subnet of it is available for use, but Docker refuses to use it because there is already a route there, but it's welcome to override it - except I can't tell it to do so.

I think there should be an option for Docker to just go ahead and "claim" the IP addresses and add routes. If anybody knows of one I'd be interested to learn about it.

[kity-linuxero]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

Works for me.
rpi4/Raspbian/4.19.118-v7l+

[Snagnar]

Had the same issue, applied @kinglion811 's magic and boom it worked. Could someone please explain, how exactly this fixes the issue?

[ccjjmartin]

@Snagnar the default docker network is attempting to use an IP address range that are already used by another device on the network so it recognizes that and fails (because two devices can't have the same IP). The magic above changes the default network to use a different IP address range and then it works. Another solution people seem to be overlooking is what @ryanisn posted on this thread. It is a better solution to me as it uses dockers native config setup. Here are the docs: https://docs.docker.com/network/bridge/#configure-the-default-bridge-network

[ccjjmartin]

@ToonSpinISAAC Why not pick another private IP address range like 192.168.x.x or 172.16.x.x? I very commonly use the 192 range for my routers, 10 range for my VPNs and 172 for my VM / docker setups. https://www.arin.net/reference/research/statistics/address_filters/

[ToonSpinISAAC]

@ccjjmartin

Why not pick another private IP address range like 192.168.x.x or 172.16.x.x?

I have three answers to that:

1. Why not have Docker do basic routing correctly?
2. How we divide up our RFC-1918 IP address space is not relevant to the issue. It's also actually none of anybody's business but ours.
3. Having said that, it's because we use 172.x.x.x for other internal networking stuff, which is why we use a 10.x.x.x range for Docker in the first place, and I'd rather not use the 192.168.x.x range because people's home modems/routers often use those, and there is a bit of a work-from-home thing happening in our organization at the moment.

This issue is super clear, at least from where I'm sitting: you can have Docker use a custom IP range but it refuses to use any of them in certain circumstances for no valid reason. It's a clearly a bug, and if I knew how to write Go I would have submitted a PR long ago. I've actually considered learning it for this reason.

[ccjjmartin]

@ToonSpinISAAC

1. Docker is following basic routing principles. The most basic example here is to think about this scenario, you give both docker and your VPN the permission to assign devices to 10.0.0.2, where does your router send the packets when two devices on the same network have the same IP? This issue is related to creating a bridge between both docker and your local network, so the IP address ranges cannot conflict.
2. Your address space is relevant to your issues and everyone else on this thread, private networks / VPNs conflict with assigned ip addresses by docker.
3. I agree with the not using 192.168 range is possible due to home routers.

I would expect that even if you learned go and wrote a patch that allowed IP address ranges to conflict that it wouldn't get merged.

And to give you my perspective, I am not a docker guru, I am not a networking guru. I am just a guy that spent tons of my person time learning basic networking principles in order to over come this specific issue on my personal setup not my corporate setup. So I was in your shoes a few months ago and am chiming in here because I know fighting through this error sucks if you don't have the knowledge to know what is happening here.

If you really wanted to learn go and submit a patch, a good approach to solving this would be to have docker find an available ip address range if the assigned one is not available.

[jhaprins]

@ccjjmartin, sorry but you are not right.
@ToonSpinISAAC, you are 100% right.
This is an issue with the way the docker daemon thinks that it should search for available space.
The following is a very clear example. In my corporate network I push a route to 10.0.0.0/8 and 172.16.0.0/12 from the VPN server to all the connecting clients (some people work from home these days :-( )
This results in a client routing table looking like this:

10.0.0.0/8 via 192.168.2.1 dev tap0 proto static metric 50
172.16.0.0/12 via 192.168.2.1 dev tap0 proto static metric 50
192.168.0.0/16 via 192.168.2.1 dev tap0 proto static metric 50

This should not be any problem because these routes are always least specific in RFC1918.
When I configure my docker daemon with the following config file:

{
"default-address-pools":
[
{"base":"10.10.0.0/16","size":24}
]
}

The daemon complains that it can not configure this because it can't create the bridge interface, but when I look at the routing on my local machine when the docker daemon is started before the VPN comes up, the routing for the docker daemon looks like this:
10.10.0.0/24 dev docker0 proto kernel scope link src 10.10.0.1 linkdown

This route is more specific then the routing pushed by the VPN, so it should never be a problem.
The reason this must be a bug in the docker code is because I also have a route:
default via 192.168.178.1 dev enp62s0u1u1 proto dhcp metric 100
This route is by definition less specific then any other route on the system, and includes the whole network, so the fact that it doesn't complain about that one and it does complain about a less specific route being smaller then the default route is rather strange and makes me think that there are some strange things going on.

The docker code should just create the bridge interface and the route and make the routing on the system sort it out.

Jan Hugo Prins
Network administrator.

[ToonSpinISAAC]

Hi @jhaprins,

You said it better than I could. I would add (and I assume you will agree) that even if this is intended behavior, I disagree with Docker's philosophy because I feel provisioning IP ranges and preventing conflicts is not Docker's responsibility but the network admin's. I think that's a big difference between @ccjjmartin's point of view and ours.

Edited to add that I want to be clear and I highly doubt that this behavior is what the Docker developers want. I don't think this is intended behavior.

Chris, think of it this way: I think what me and Jan Hugo are saying, is that the point of having this setting in daemon.json is for users to tell Docker which IP ranges are safe to use, and not the other way around. I am capable of partitioning my own network, and want to be able to tell Docker where its sandbox is. It's none of Docker's business that the sandbox happens to be in a playground with other kids playing in other sandboxes.

[jhaprins]

Hello,

I have found an other reason why this must be a bug in the docker code.
I have a default route looking like this:
default via 192.168.178.1 dev enp62s0u1u1 proto dhcp metric 100

I should be able to split this route in my routing table, without any side effects into the following 2 routes:
0.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1
128.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1

Before I do this split in my routing table docker start just fine with the following config:
{
"default-address-pools":
[
{"base":"10.10.0.0/16","size":24}
]
}

After this split the docker daemon fails to start with the following error:
Oct 01 14:54:34 capetown. dockerd[27900]: failed to start daemon: Error initializing network controller: list bridge addresses failed: PredefinedLocalScopeDefaultNetworks List: [10.10.0.0/24 10.10.1.0/24 10.10.2.0/24 10.10.3.0/24 10.10.4.0/24 10.10.5.0/24 10.10.6.0>
Oct 01 14:54:34 capetown. systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 14:54:34 capetown. systemd[1]: docker.service: Failed with result 'exit-code'.
Oct 01 14:54:34 capetown. systemd[1]: Failed to start Docker Application Container Engine.

My complete routing table looks like this:
[root@capetown docker]# ip ro
0.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1
128.0.0.0/1 via 192.168.178.1 dev enp62s0u1u1
172.16.0.0/24 dev vmnet2 proto kernel scope link src 172.16.0.1
172.16.1.0/24 dev vmnet3 proto kernel scope link src 172.16.1.1
172.16.2.0/24 dev vmnet4 proto kernel scope link src 172.16.2.1
172.16.3.0/24 dev vmnet5 proto kernel scope link src 172.16.3.1
172.16.4.0/24 dev vmnet6 proto kernel scope link src 172.16.4.1
172.16.5.0/24 dev vmnet7 proto kernel scope link src 172.16.5.1
172.16.11.0/24 dev vmnet8 proto kernel scope link src 172.16.11.1
172.16.83.0/24 dev vmnet1 proto kernel scope link src 172.16.83.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
192.168.178.0/24 dev enp62s0u1u1 proto kernel scope link src 192.168.178.74 metric 100

Conclusion:
This is a bug in docker.

Jan Hugo Prins
Network administrator

[jhaprins]

I have found an open ticket in the moby project that matches this same error (also some closed tickets) and I have given a full explanation in this ticket why this should be treated as a bug: moby/moby#33925

[3lf]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

thanks. Worked

[blacklight]

The ip cli solution is a workaround that can make things work until the next reboot. For a more stable workaround, I've resorted to a netctl profile for the Docker interface:

Description="Docker bridge interface"
Interface=docker0
Connection=bridge
MACAddress=00:11:22:33:44:55
IP=static
Address='172.17.0.1/16'
SkipForwardingDelay=yes

Note however that this is still a workaround. It SHOULD NOT be up to the user to set up the bridge interface required by an application! Docker should definitely take care of it automatically! And this is completely counter-intuitive if you look at the returned error:

stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: list bridge addresses failed: PredefinedLocalScopeDefaultNetworks List: [172.17.0.0/16 172.18.0.0/16 172.19.0.0/16 172.20.0.0/16 172.21.0.0/16 172.22.0.0/16 172.23.0.0/16 172.24.0.0/16 172.25.0.0/16 172.26.0.0/16 172.27.0.0/16 172.28.0.0/16 172.29.0.0/16 172.30.0.0/16 172.31.0.0/16 192.168.0.0/20 192.168.16.0/20 192.168.32.0/20 192.168.48.0/20 192.168.64.0/20 192.168.80.0/20 192.168.96.0/20 192.168.112.0/20 192.168.128.0/20 192.168.144.0/20 192.168.160.0/20 192.168.176.0/20 192.168.192.0/20 192.168.208.0/20 192.168.224.0/20 192.168.240.0/20]: no available network

What that error means is that ALL of those subnets should be tested, and Docker should pick the first available subnet if a network interface isn't available already and create a bridge on it.

What happens, instead, is that if none of those subnets is already assigned to an available network bridge, and some of them are already in use from another interface, Docker fails, believing that for some reason NONE of those addresses is available. Can you guys please motivate how this is not supposed to be a bug?

I'm appalled by the fact that after so many years and reports the development team hasn't yet addressed this issue, and the issue is still closed despite so many reports. Please address this, or at least motivate why this isn't a bug!

[ToonSpinISAAC]

In my opinion, none of the ranges should be tested if they are in daemon.json. Again, Docker needs to not tell me how to partition my network. Instead it needs to listen to me as a network administrator because I, not it, decides what IP ranges are available to it.

[jhaprins]

I have linked open tickets with an exhausting analysis of this bug.
I would really like to have this fixed in Docker / Moby / any other derivatives.

[abhinava]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

This comment from @kinglion811 works! I'm running Ubuntu 20.04.02 LTS on Pine64.

abhinava@pine64-b:~$ uname -a
Linux pine64-b 5.10.21-sunxi64 #21.02.3 SMP Mon Mar 8 00:45:13 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
abhinava@pine64-b:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

Why doesn't dockerd create the docker0 Linux interface by default? My experience of docker on different Linux distributions have always been different :-|

I followed the installation guide mentioned here but doesn't work out of the box. I had to edit the /lib/systemd/system/docker.service to add the correct the -H, --host list local socket to unix:// - The default fd:// doesn't seem to work.

I also followed @ryanisn 's comment above I also created a /etc/docker/daemon.json file (which doesn't exist by default) to enable IPv6 and also restrict the docker0 bridge IPv4/IPv6 to a smaller subnet range:

{
    "experimental": true,
    "bip": "172.17.18.1/24",
    "fixed-cidr": "172.17.18.1/25",
    "debug": true,
    "ipv6": true,
    "fixed-cidr-v6": "fd00:dead:beef::/80"
}

NOTE: The above address ranges work for me - Others may try using this as-is, but if doesn't work (due to home LAN or VPN), you might need some trial-and-error.

Adding this comment to help others who may have the same issue.

And thanks to the great comments from @ToonSpinISAAC, @ccjjmartin, and @jhaprins - Perhaps Docker should force users to create the daemon.json configuration file rather than trying to "guess"? But this might put-off users without strong networking background. I think Docker must pick a smaller /24 (or even smaller) subnet for the bridge. Picking the 192.168.0.0/16 may not be good since lots of home routers use this range. Most "regular' Docker users may not need a large 172.16.0.0/12 range (in essence, most users may not need to run thousands of containers on their machine and hence, a smaller subnet would suffice).

[hankedang]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

Niubility! this worked for me

[niutouJust]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

good

[atorr0]

With openconnect VPN I have a 172.16.0.0/255.240.0.0 route for him, so I had to apply @kinglion811 magical recipe (#123 (comment)) with the next available address: 172.32.0.1/16

More info on http://jodies.de/ipcalc?host=172.17.0.0&mask1=255.240.0.0&mask2= (see HostMax field)

[pdolinic]

IT appears Docker currently is unable to fulltunnel in OpenVPN due to routes - as mentioned here several times, and this should be seriously worked upon? The solution is to get your admin to switch to splittunnel.

1. Log onto your OpenVPN Server
2. open /etc/openvpn/server/server.conf
3. Swap from fulltunnel to splittunnel via via ; this: ;push redirect-gateway def1 bypass-dhcp
4. systemctl restart openvpn-server@server.service
5. Whatever didn't work before should work now

[rulatir]

ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
can solve this issus

How and why? What do those command do, why does doing that thing solve this issue, how was it possible that a critically important part of system configuration just spontaneously "deconfigured" itself, and why isn't docker smart enough to reconfigure it for itself after finding it missing?

[NavarroMauro]

Sol

ip link add name docker0 type bridge ip addr add dev docker0 172.17.0.1/16 can solve this issus

This solution work in Ubuntu 22.04

[thachnv92]

ip link add name docker0 type bridge ip addr add dev docker0 172.17.0.1/16 can solve this issus

Many thanks.
It's work on Ubuntu 22.04
Before, let's run this command to show error.
sudo dockerd --debug # this command will show error

[thaJeztah]

I think moby/moby#43360 also addresses the situation with a VPN enabled (which is included on docker 20.10.15 and up)

[nishantvarma]

Thanks @thaJeztah , this really helped! I really had to switch to a different IP range -- due to office IP conflicts -- so this was a deal breaker for me! It was working in my machine, but not in my colleagues. I was worried if it was something at the OS level, but luckily that wasn't the case.