default route for container in multiple user defined networks

[thechile]

At the moment if you connect a container to multiple user defined networks then the default route used will be the subnet of the last connected network. Is there any way to avoid the order which the container is added to the networks determining which default route is used ? Something like docker network connect backend rose --no-default-route

In the below example, a way to avoid adding the backend network to the container and it automatically becoming the default gw. It would be nice to have the behavior user definable.

docker network create frontend
docker network create backend
docker run -d --name rose --net=frontend busybox top
docker exec rose ip r

default via 172.20.0.1 dev eth0 
172.20.0.0/16 dev eth0  src 172.20.0.2

docker network connect backend rose
docker exec rose ip r

default via 172.21.0.1 dev eth1 
172.20.0.0/16 dev eth0  src 172.20.0.2 
172.21.0.0/16 dev eth1  src 172.21.0.2 

[aboch]

As you said, the option is currently missing in the UI, but this can be done.

In fact libnetwork code already provides a way to set the priority of the endpoint, so that in case of container connected to multiple networks, the container's default gateway is determined by the highest priority endpoint.

[thechile]

ok thanks for confirming. Being able to specify a priority to control which user defined network associated with a container is chosen for default gw would be useful.

Hopefully this will be plumbed in soon.

[irsl]

+1 (similar request here: #20067)

[GordonTheTurtle]

USER POLL

The best way to get notified of updates is to use the Subscribe button on this page.

Please don't use "+1" or "I have this too" comments on issues. We automatically
collect those comments to keep the thread short.

The people listed below have upvoted this issue by leaving a +1 comment:

@JrCs

[aboch]

Regarding #20179 (comment), I forgot to mention: Any help is appreciated in adding the missing code.

[LK4D4]

@aboch is this still an issue? Do you think #20067 is duplicate?

[aboch]

@LK4D4 I think the two issues are not a duplicate. This one is about controlling the default route the other one is about a consistent interface order.

@thechile
Regarding

At the moment if you connect a container to multiple user defined networks then the default route used will be the subnet of the last connected network.

that is not the generic rule, the pickup logic is now clarified in the docs https://github.com/docker/docker/blob/master/docs/userguide/networking/index.md:

When a container is connected to multiple networks, its external connectivity is provided via the first non-internal network, in lexical order.

So even now if you play with the network names, you can deterministally impose the default route to choose.

[tarunlalwani]

Any update on this? What is the way to choose the default gateway in case of multiple networks?

[thaJeztah]

removed the "out of office" messages 👍

[logicethos]

I know it's been discussed before, but having --net=[ ] seems the logical way to do this. The first network in the list, becomes the default.

I just ran into this myself. Rebooted my server, and when the container came back up, eth0 and eth1 flipped causing the service inside the container to fail.

[thaJeztah]

@logicethos unfortunately, that won't resolve situations where you docker network connect or docker network disconnect networks.

Note that multiple --net / --network flags is not supported yet, but I opened a proposal to bring in the foundations for allowing this; see #31964, and the "related issues" section in that proposal.

The advanced syntax has been implemented for services, but not yet for docker run (see docker/cli#317)

[karthanistyr]

In docker/compose/issues/4645, support was added for setting a priority parameter on container endpoints ("networks") in the compose file.

It looks like there are still missing pieces according to @shin- : the priority parameter is still not passed to docker-py and therefore not to the docker engine.

compose/service.py @L652

self.client.connect_container_to_network(
                container.id, network,
                aliases=aliases,
                ipv4_address=netdefs.get('ipv4_address', None),
                ipv6_address=netdefs.get('ipv6_address', None),
                links=self._get_links(False),
                link_local_ips=netdefs.get('link_local_ips', None),
                # priority=netdefs.get('priority', 0),  # <= should be added
            )

It does look like libnetwork can, however, already accept a priority argument for ordering the container's interfaces as suggested by @aboch in comment#2:

libnetwork/endpoint.go @L1044

func JoinOptionPriority(ep Endpoint, prio int) EndpointOption {
	return func(ep *endpoint) {
		// ep lock already acquired
		c := ep.network.getController()
		c.Lock()
		sb, ok := c.sandboxes[ep.sandboxID]
		c.Unlock()
		if !ok {
			logrus.Errorf("Could not set endpoint priority value during Join to endpoint %s: No sandbox id present in endpoint", ep.id)
			return
		}
		sb.epPriority[ep.id] = prio
	}
}

There is a functional test in libnetwork/sandbox_test.go @L80

func TestSandboxAddMultiPrio(t *testing.T) {
...

@thaJeztah: please can you confirm these two entities are related in their intended function? If so, is there value to create PRs spanning 3 components (compose, docker-py, moby) to link these parameters to achieve deterministic interface ordering in containers according to priority set in the compose file, and possibly also deterministic assignment of the default route?

[atarlov]

The priority would define only the order or as well the default gw. The interface with the highest prio will set the default gw ?

e.g.
networks:
app_net_1:
priority: 1000 | default via x.x.x.x eth0
app_net_2:
app_net_3:
priority: 100

[Gramler]

Is there some "correct" way to solve this problem, since v3 now does not have support for priority any more?
https://docs.docker.com/compose/compose-file/compose-file-v2/#priority
not in v3... (See link above to that feature)

[jfellus]

@Gramler priority has been indeed removed from the docker-compose file format v3 docs, but looking to the code i don't see any difference between v2 and v3 about network priorities (see https://github.com/docker/compose/blob/b01601a53cc8839afbe332f00d0d1b8165012e7b/compose/network.py#L313). In fact, a rapid test shows that priorities are just ignored, would you use v2 or v3 : https://gist.github.com/jfellus/cfee9efc1e8e1baf9d15314f16a46eca
-> I would be cool to have the docs updated about that, this is a bit misleading, as the feature is not deprecated

@karthanistyr seems correct ! Please tell me If you need help on this

[thediveo]

Any update on this mess? And just blindly taking the highest priority might not be the best idea in case of internal networks. So it might be more like:

1. take network of highes prio that isn't internal
2. if 1. results in none, take the network with the highest prio.

[karthanistyr]

the pieces have moved significantly in two years since I analysed it, so my comment above is most probably obsolete. docker-compose has now become internalised as docker compose, therefore it may be a smaller change.

[thediveo]

afaik, https://github.com/docker/compose-cli isn't yet feature-complete with docker-compose.

[dereddy93]

I'm running into even more issues when using the new docker compose instead of the old Python implementation.
I tried to use the above mentioned solution of assigning priorities to the connecting networks. However, as @jfellus already mentioned, the values are simply being ignored. Both in compose files v2 and v3, as well as Go and Python docker-compose implementations.
This is really annoying tbh. I simply can't understand, why the feature of "not-messing-with-my-routes" is still not being implemented after almost 6 years!?
My current workaround is a bash script that deletes the default route and creates a new one with the gateway specified through an environment variable. Not the prettiest solution but it gets the job done.
I would really appreciate an update on this topic. If you do not plan to implement such a functionality, please, at least close this feature request and stop giving me hope...

[Vlad1mir-D]

Any updates?

[logopk]

I year after the last ping...

Still an issue. Alphabetically is bad if project name is added the the network name...

[UrusovVladimir]

+1

[ovizii]

This really, really su**s badly. I don't understand why this is not a bigger issue, how are other people getting around this limitation or is this such an edge case?

Or maybe I am rushing things, its barely been 8 years since this issue was created...

[json-m]

how are other people getting around this limitation or is this such an edge case?

try to solve it with a networking plugin or move on to alternatives such as podman

[thediveo]

how are other people getting around this limitation or is this such an edge case?

try to solve it with a networking plugin or move on to alternatives such as podman

1. this is not network plugin territory, or maybe you can detail how this should work then? (Note that I have already written a Docker networking plugin for a company, so I have at least a tiny bit of working knowledge on the API and responsibilities of plugins versus libnetwork.)
2. podman, good luck with its three different networking stacks; which one do you use and where does podman document that it supports the behavior discussed here? I'm interested to hear of a concrete link.

[rollicks-c]

This really, really su**s badly. I don't understand why this is not a bigger issue, how are other people getting around this limitation or is this such an edge case?

Or maybe I am rushing things, its barely been 8 years since this issue was created...

I'm aware that this might not be an option for you - but I migrated all my projects to kubernetes in order to no longer deal with limitations or flaws of this kind