Docker run crashes with Error creating default "bridge" network: permission denied

[ivdorelian]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

sudo systemctl start docker should correctly start docker.

Actual behavior

It gives an error:

Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

journalctl -xe gives:

  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.039853271Z" level=debug msg="Firewalld passthrough: ipv4, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]"
  May 19 08:45:17 my.server.tld firewalld[128]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.049280804Z" level=debug msg="Firewalld passthrough: ipv4, [-t nat -D PREROUTING]"
  May 19 08:45:17 my.server.tld firewalld[128]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.058288942Z" level=debug msg="Firewalld passthrough: ipv4, [-t nat -D OUTPUT]"
  May 19 08:45:17 my.server.tld firewalld[128]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.068605847Z" level=debug msg="Firewalld passthrough: ipv4, [-t nat -F DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.078732710Z" level=debug msg="Firewalld passthrough: ipv4, [-t nat -X DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.088858154Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -F DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.099074239Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -X DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.109451286Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -F DOCKER-ISOLATION-STAGE-1]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.119789596Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -X DOCKER-ISOLATION-STAGE-1]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.129741289Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -F DOCKER-ISOLATION-STAGE-2]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.139899575Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -X DOCKER-ISOLATION-STAGE-2]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.149545851Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -F DOCKER-ISOLATION]"
  May 19 08:45:17 my.server.tld firewalld[128]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.161702471Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -X DOCKER-ISOLATION]"
  May 19 08:45:17 my.server.tld firewalld[128]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.189553198Z" level=debug msg="Firewalld passthrough: ipv4, [-t nat -n -L DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.201206779Z" level=debug msg="Firewalld passthrough: ipv4, [-t nat -N DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.211888849Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -n -L DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.221932805Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -N DOCKER]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.231964707Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -n -L DOCKER-ISOLATION-STAGE-1]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.241278619Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -N DOCKER-ISOLATION-STAGE-1]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.250686355Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -n -L DOCKER-ISOLATION-STAGE-2]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.261410608Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -N DOCKER-ISOLATION-STAGE-2]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.274901372Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.293487689Z" level=debug msg="Firewalld passthrough: ipv4, [-A DOCKER-ISOLATION-STAGE-1 -j RETURN]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.317872917Z" level=debug msg="Firewalld passthrough: ipv4, [-t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.337487620Z" level=debug msg="Firewalld passthrough: ipv4, [-A DOCKER-ISOLATION-STAGE-2 -j RETURN]"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.360083685Z" level=debug msg="Allocating IPv4 pools for network bridge (6d7ae465f646b2fd1d5ea39d36c9af111670c6e4f91c7b6a922c076050cf0a78)"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.360561939Z" level=debug msg="RequestPool(LocalDefault, 172.17.18.1/24, 172.17.18.0/25, map[], false)"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.361018354Z" level=debug msg="RequestAddress(LocalDefault/172.17.18.0/24/172.17.18.0/25, 172.17.18.1, map[RequestAddressType:com.docker.network.gateway])"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.361908437Z" level=debug msg="Request address PoolID:172.17.18.0/24 App: ipam/default/data, ID: LocalDefault/172.17.18.0/24, DBIndex: 0x0, Bits: 256, Unselected: 254, Sequence: (0x80000000, 1)->(0x0, 6)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:172.17.18.1
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.362397190Z" level=debug msg="Did not find any interface with name docker0: Link not found"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.362935554Z" level=debug msg="Failed to create bridge docker0 via netlink. Trying ioctl"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.368678322Z" level=debug msg="releasing IPv4 pools from network bridge (6d7ae465f646b2fd1d5ea39d36c9af111670c6e4f91c7b6a922c076050cf0a78)"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.369138101Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.18.0/24/172.17.18.0/25, 172.17.18.1)"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.369570818Z" level=debug msg="Released address PoolID:LocalDefault/172.17.18.0/24/172.17.18.0/25, Address:172.17.18.1 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.18.0/24, DBIndex: 0x0, Bits: 256, Unselected: 253, Sequence: (0xc0000000, 1)->(0x0, 6)
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.370108243Z" level=debug msg="ReleasePool(LocalDefault/172.17.18.0/24/172.17.18.0/25)"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.370541074Z" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.370957476Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.371334882Z" level=debug msg="Cleaning up old mountid : start."
  May 19 08:45:17 my.server.tld dockerd[10597]: time="2021-05-19T08:45:17.371525551Z" level=debug msg="Cleaning up old mountid : done."
  May 19 08:45:17 my.server.tld dockerd[10597]: Error starting daemon: Error initializing network controller: Error creating default "bridge" network: permission denied

Output of docker version:

[root@srv ~]# docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:01 2019
 OS/Arch:           linux/amd64
 Experimental:      false
error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.39/version: read unix @->/var/run/docker.sock: read: connection reset by peer

Output of docker info:

[root@srv ~]# docker info
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Additional environment details (AWS, VirtualBox, physical, etc.)

I'm on a CentOS 7.5 VPS:

[root@srv ~]# uname -a
Linux my.server.tld 3.10.0-1160.21.1.vz7.174.13 #1 SMP Thu Apr 22 16:18:59 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux

The only other things running are nginx with php, but I tried stopping those and no success.

I tried various solutions like deleting /var/lib/docker/network/files/local-kv.db or the entire network contents and no success. I tried a more recent version too.

I also tried the top upvoted solution here: #123

ip link add name docker0 type bridge 
ip addr add dev docker0 172.17.0.1/16

With and without sudo, the output is:

[root@srv ~]# ip link add name docker0 type bridge  
RTNETLINK answers: Permission denied

/etc/docker/daemon.json is (I also tried without this file at all):

{
    "experimental": false,   <- tried true as well
    "bip": "172.17.18.1/24", <- tried others like 192.168.x.y as well
    "fixed-cidr": "172.17.18.1/25",
    "debug": true,
    "ipv6": false, <- tried true as well
    "fixed-cidr-v6": "fd00:dead:beef::/80"
}

Output of ip addr:

[root@srv ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/void 
    inet 127.0.0.1/32 scope host venet0
       valid_lft forever preferred_lft forever
    inet MY_EXTERNAL_IP_REDACTED/32 brd MY_EXTERNAL_IP_REDACTED scope global venet0:0
       valid_lft forever preferred_lft forever

I'm at my wits' end here, anyone run into this before?

[ivdorelian]

This was solved for me by the VPS providers. Initially they said that I'm responsible for the server administration and they couldn't help, but because it works with no issue on a gcloud machine, they eventually took another look and made it work. I don't know what they did, all I know is that they run OpenVZ. I don't see any of the above outputs changed, other than docker running and docker0 being there of course.

So for anyone facing a similar issue, I suggest you pester your VPS providers if you're on a VPS.

I'm not sure if this should be closed or if the devs want to look into OpenVZ compatibility.

All the best!