Could not restore network resources for running containers

[dimitar-dimitrow]

Network could not be restored upon container-management start. Once the error pop up it persists even after container-management restart. As a result containers with bridge network could not be started.

time="2023-05-31T12:53:26.646123957Z" level=warning msg="[container-management][libnet_mgr_internal_common.go:177][pkg:github][func:com/eclipse-kanto/container-management/containerm/network]  host network already exists - will not create a new one"
time="2023-05-31T12:53:26.646141549Z" level=debug msg="[container-management][libnet_mgr.go:255][pkg:github][func:com/eclipse-kanto/container-management/containerm/network]  successfully created and initialized the new default network [host] from scratch "
time="2023-05-31T12:53:26.646151298Z" level=debug msg="[container-management][libnet_mgr.go:268][pkg:github][func:com/eclipse-kanto/container-management/containerm/network]  there are no active sandboxes - the default bridge network will be initialized"
time="2023-05-31T12:53:26.661309845Z" level=debug msg="Allocating IPv4 pools for network bridge (7ad3257b2622c81d2c14d87b8a4a2a1f501ec670a6019b319648b383673e3e72)"
time="2023-05-31T12:53:26.661325654Z" level=debug msg="RequestPool(LocalDefault, 172.18.0.0/16, , map[], false)"
time="2023-05-31T12:53:26.661352302Z" level=debug msg="RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[RequestAddressType:com.docker.network.gateway])"
time="2023-05-31T12:53:26.661370319Z" level=debug msg="Request address PoolID:172.18.0.0/16 App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:<nil> "
time="2023-05-31T12:53:26.661401542Z" level=debug msg="releasing IPv4 pools from network bridge (7ad3257b2622c81d2c14d87b8a4a2a1f501ec670a6019b319648b383673e3e72)"
time="2023-05-31T12:53:26.661409559Z" level=debug msg="ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.1)"
time="2023-05-31T12:53:26.661419441Z" level=debug msg="Released address PoolID:LocalDefault/172.18.0.0/16, Address:172.18.0.1 Sequence:App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:2"
time="2023-05-31T12:53:26.661428294Z" level=debug msg="ReleasePool(LocalDefault/172.18.0.0/16)"
time="2023-05-31T12:53:26.661448835Z" level=error msg="[container-management][mgr.go:153][pkg:github][func:com/eclipse-kanto/container-management/containerm/mgr]  could not restore network resources for running containers \n\t Error: cannot create network 7ad3257b2622c81d2c14d87b8a4a2a1f501ec670a6019b319648b383673e3e72 (kanto-cm0): conflicts with network 42be7d7b813c64d0bf0f4fb6dd7a8bcba93435eadcda9f3ab84c08b754250234 (kanto-cm0): networks have same bridge name "

[dimitar-dimitrow]

The problem was caused by the stored network setting, they were persisted in some intermediate state. The problem was fixed by deleting the network storage. Could not reproduce the issue and find the initial reason for it's occurrence.

[daniel-milchev]

I think this issue should stay open for further checks because I got the same problem.

time="2024-01-19T10:11:03.708653122+02:00" level=error msg="[container-management][ctrd_client_internal.go:389][pkg:github][func:com/eclipse-kanto/container-management/containerm/ctr]  failed to receive envelope: rpc error: code = Canceled desc = context canceled"
time="2024-01-19T10:11:03.944154848+02:00" level=warning msg="[container-management][daemon_config_util.go:106][pkg:main][func:extractThingsOptions]  DEPRECATED: Things connection settings are now deprecated and will be removed in future release. Use the global connection settings instead!"
time="2024-01-19T10:11:03.944352931+02:00" level=info msg="[container-management][daemon_internal_init.go:67][pkg:main][func:(*daemon)]  Containers Update Agent is disabled - no Update Agent Services will be registered. If you would like to enable Update Agent support, please, reconfigure ua-enable to true"
time="2024-01-19T10:11:03.944956784+02:00" level=warning msg="[container-management][libnet_mgr.go:176][pkg:github][func:com/eclipse-kanto/container-management/containerm/network]  no network settings are restored for container id = 5154df43-1633-4523-92ca-01feecbe7aff"
time="2024-01-19T10:11:03.94496961+02:00" level=warning msg="[container-management][libnet_mgr.go:176][pkg:github][func:com/eclipse-kanto/container-management/containerm/network]  no network settings are restored for container id = 8df6d4a9-4ff9-4afb-a8d1-ac7b0557c9c9"
time="2024-01-19T10:11:03.94497612+02:00" level=warning msg="[container-management][libnet_mgr.go:176][pkg:github][func:com/eclipse-kanto/container-management/containerm/network]  no network settings are restored for container id = e5ee081e-aa73-4d41-898a-86f8515f093e"
time="2024-01-19T10:11:04.64879172+02:00" level=info msg="Removing stale sandbox 9edb8d328c2543dc1bfa143ba18b693dc8acbb36829186f091260c97682b6093 (8df6d4a9-4ff9-4afb-a8d1-ac7b0557c9c9)"
time="2024-01-19T10:11:04.779869794+02:00" level=info msg="Removing stale sandbox d619a2aa36c8797f5b5ca2bd90cbaa7cf139b3fc4ab8a32cea8cb407e3beb984 (e5ee081e-aa73-4d41-898a-86f8515f093e)"
time="2024-01-19T10:11:04.848059847+02:00" level=info msg="Removing stale sandbox 317229f24ce460232b10149b8f5fe43169b7bb9159992013d169b19157fdede9 (5154df43-1633-4523-92ca-01feecbe7aff)"
time="2024-01-19T10:11:04.853559162+02:00" level=warning msg="[container-management][libnet_mgr_internal_common.go:150][pkg:github][func:com/eclipse-kanto/container-management/containerm/network]  host network already exists - will not create a new one"
time="2024-01-19T10:11:04.854243715+02:00" level=error msg="[container-management][mgr.go:153][pkg:github][func:com/eclipse-kanto/container-management/containerm/mgr]  could not restore network resources for running containers \n\t Error: cannot create network 7f66f853eb68547e3418193f7ba96c8913018db8a6a63a27ec5219ef1a939032 (kanto-cm0): conflicts with network 4380671a30539517096b7f019ad31345d836e84038ed1607f8a214532658fc89 (kanto-cm0): networks have same bridge name "