Control interface name in docker multiple networking

[satyajitbm]

With the workaround suggested in #17796 for multiple network mode for docker containers, it is not possible to control the interface name with the docker network connect command. The order of the ethx interfaces after docker start is not consistent with the sequence of docker connect commands. Only the network added in the docker create command comes up as eth0 and is consistent.

My post boot script relies on the name of the interfaces and won't work if the order is changed and I require the interfaces to be up on boot. So, I can't use pipework script with the -iflag as well.

Output of docker version:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:23:11 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:23:11 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 24
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-253:0-877344-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.141 GB
 Data Space Total: 107.4 GB
 Data Space Available: 103.2 GB
 Metadata Space Used: 4.489 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null
Kernel Version: 3.10.0-229.20.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.24 GiB
Name: vspdemo1.mv.nuagenetworks.net
ID: JTCU:HSQR:NBHL:6T2O:JN32:JZHQ:IBIH:K3ZW:IHV5:LX2V:BILX:YM4R
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details: Running on a centos 7.1 bare metal.
uname -a
Linux "REDACTED" 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce the issue:

1. docker network create --driver bridge --subnet 10.100.1.0/24 net1
2. docker network create --driver bridge --subnet 10.100.2.0/24 net2
3. docker network create --driver bridge --subnet 10.100.3.0/24 net3
4. docker network create --driver bridge --subnet 10.100.4.0/24 net4
5. docker create -it --net=net1 --ip=10.100.1.3 --name=test centos:7
6. docker network connect --ip 10.100.2.3 net2 test
7. docker network connect --ip 10.100.3.3 net3 test
8. docker network connect --ip 10.100.4.3 net4 test
9. docker start test
10. docker exec -it test yum install -y net-tools
11. docker exec -it test ifconfig

Results received: eth2 was assigned 10.100.4.3 and eth3 was assigned 10.100.3.3

Results expected: eth2 must be assigned 10.100.3.3 and eth3 must be assigned 10.100.4.3 or there needs to be a way to specify the interface name in docker network connect command.

The issue happens occasionally and the probability of it occurring increases with the increase in the number of networks the container is connected to.

[cirocosta]

So, essentially, in your case you can work around it through subnet filtering, right? e.g., you know that a given subnet has been assigned to a given network, thus you can filter your iface list and find the one which is the one assigned to your desired network. Did you go through this path? Or did something different?

[thaJeztah]

With the "csv" style syntax for networks being worked on (see #31964, and PR's docker/cli#62 (for services), and docker/cli#156 for docker run), perhaps it would be possible to add an additional configuration option to specify the interface name for each network that the container is attached to.

ping @abhinandanpb @fcrisciani WDYT?

[abhi]

@thaJeztah I believe one of the goals of the network-opt initially was to ensure symmetry between the service create and docker run. But this interface option would only apply to container create / network connect right ?

[thaJeztah]

@abhinandanpb would it be different for services and "regular" containers? i.e.;

docker run \
  --network name=my-network,iface=my-eth \
  --network name=my-other-network,iface=my-other-eth \
  nginx

docker service create \
  --network name=my-network,iface=my-eth \
  --network name=my-other-network,iface=my-other-eth \
  nginx

For docker service create, docker would create an my-eth and my-other-eth in each task (container) backing the service.

[chrisjonez]

@thaJeztah I'm interested in being able to specify an explicit interface, so:

docker run \
  --network name=my-network,iface=my-eth \
  --network name=my-other-network,iface=my-other-eth \
  nginx

would be great...or something similar in docker network connect..., where the interface can be specified.

I also use docker-compose, so would also need to be able to specify the interface in the services block of a container.

Is there any likelihood that this will be implemented soon?

[purplesrl]

+1

This is pretty serious, meaning that if you have more networks, the interfaces will get mixed up.

Almost all network applications base themselves on the fact that the interface names are the same.

Why would they be randomly assigned it is beyond my understanding, when it would have been much easier to assigned them in order at least based on the network name or the order the networks were connected to the container.

[mikesir87]

I am interested in this as well. As a use case...

Have a service that is in two networks... one for reverse proxying (frontend) and one for its own backend (backend). In order to receive traffic, the app (a Java app) needs to listen to the interface for the frontend. Since we aren't using static subnets, it's difficult to discover/know which interface we should use.

The proposed solution above would work for us.

[sandersaares]

I would definitely be interested in having Docker network interfaces have predictable names. Most software dealing with networking assumes you have static interface names, so I cannot use any of that with Docker networks, which is a pain.

[ptman]

docker network create -o com.docker.network.bridge.name=br-extra br-extra

[multun]

@ptman that's not the point. it's about the interface name inside the container

[madpipeline]

docker run \
  --network name=my-network,iface=my-eth \
  --network name=my-other-network,iface=my-other-eth \
  nginx

@thaJeztah is that iface=my-eth functionality something that currently exists? Or is that just an implementation proposal? Because I couldn't find any docs on that.

If it exists, how would it be used from Docker Compose?

[uarmes]

@thaJeztah any updates? It's pretty strange that interface name isn't configurable. From my POV it's obvious that most of real network interfaces have their own names, very different from assumed ethX.

[jrobinson42]

Also would like to see this resolved.

[mdbraber]

Another +1 for getting this issue resolved (beyond having predictable interface order at least, for which @cziebuhr receives full credit)!

[jacekchalupka]

+1

Yes, this is something we need. Since dockers are used widely in automated environments it would be very nice to have.

[jansyren]

+1

This would be very useful for vrrp in combination with ha-proxy containers to make sure that the keepalived config is setting up the VIP on the right interface. I am using macvlan interfaces to geologically control where the access happens on a larger distributed docker swarm.

[laf0rge]

I'm sorry to sound sour, but for any but the most simplistic network use cases one obviously wants to have containers connected to multiple interfaces, and those interfaces of course have to have deterministic names. I'm surprised there is even a discsussion about it ?!?

[lump]

My Observium network monitor has literally thousands of randomly-named deleted docker interfaces. I would so love to be able to persistently name the interface myself in Swarm as part of networks: in the service configuration in docker-compose.yml. This veth thing is an obfuscation that discourages decent network resource monitoring.

[eyal0]

In my recent testing I found that the interfaces were assigned alphabetically. So which ever network was given the alphabetically first name, that's the one that got the alphabetically first device name.

Not sure how consistent that is but the rule held true for me 10 times in a row.

[aradmidias]

I can verify @eyal0 's observation.

It seems to result from this commit moby/libnetwork@c13db32 from 21. Mar 2018. It changed the way how interfaces are stored, from a heap to a list. There is also code for sorting the network interfaces, which resorts to comparing the interface's name, if their priority (sandbox_store.go#L28) is equal.

The above referenced pull requests (control network interface order for containers) try to add the functionality to allow setting the priority when connecting a container to a network via docker create and docker network connect.

Still I could not find a way to specify the network interface's name inside the container. The com.docker.network.bridge.name option only appears to change the host's network interface name.

[jfellus]

@eyal0 I can confirm to. See following gist to reproduce : https://gist.github.com/jfellus/cfee9efc1e8e1baf9d15314f16a46eca

[0x5ECF4ULT]

+1

This is something we need. I have an extra network for Traefik traffic for example and currently I have to do the alphabetical workaround to get the interfaces in the right order. Any updates on this?

[jelischer]

In our setup scripts we have the following lines:

  local PID=$(docker inspect --format='{{ .State.Pid }}' $CONTAINER)
  if [ -z "$PID" ]
  then
    echo "docker container $CONTAINER not present"
    exit 1
  fi

   # Set up network namespace identifier for ip netns to use (yech).
   # For some reason there is no direct way to specify the pid.
  sudo mkdir -p $NSDIR/
  sudo  ln -sfT /proc/$PID/ns/net $NSDIR/$CONTAINER

  sudo ip netns exec "$CONTAINER" ip link set "$HOST_IFACE" name "$CONT_IFACE_NAME"
}

it should be possible for docker connect to do something siimilar.

[corviv]

Any updates on this?

[tympanix]

One obstacle I see w.r.t a possible implementation is the inherent definition of the networking sandbox interface, which requires that a prefix is given for the network link name. This is i.m.h.o presumptuous and leads to loss of generality.

moby/libnetwork/osl/sandbox.go

Lines 26 to 31 in 0b39cc2

	// Add an existing Interface to this sandbox. The operation will rename
	// from the Interface SrcName to DstName as it moves, and reconfigure the
	// interface according to the specified settings. The caller is expected
	// to only provide a prefix for DstName. The AddInterface api will auto-generate
	// an appropriate suffix for the DstName to disambiguate.
	AddInterface(SrcName string, DstPrefix string, options ...IfaceOption) error

Changing the argument name dstPrefix to dstName (and thereby the meaning of the argument) does not change the interface definition. I assume that changing the meaning of the argument will break backwards compatibility though. Any suggestions on how to go about this?

[oponcea-dn]

We could make any changes we need inside a container before it starts if we could run hooks on certain events (like network connect) so, can we run some external hooks when container is created or after certain events (i.e. docker network attached)?
Docker reports these events but did not find anything related to hooks (we also need for the container to wait for the hook to finish running before continuing so that we can consistently change configuration inside the container).

[ericslandry]

Using latest Docker and Docker Compose, I run a container that is connected to two networks. This container needs to connect to an IPv6 link-local address (LLA). Such a connection needs to specify a zone suffix (e.g. interface name). Since my container is connected to multiple networks, the container has multiple network interfaces (e.g. eth0 and eth1). I need a deterministic way of telling the container's app how to select the right interface to use as the zone.

It is possible for an external script to query a MAC address from the Docker network for a particular container:

mac_addr=$(docker network inspect mytargetnetwork | \
    jq -r '.[].Containers[]|select(.Name=="mywork-mycontainer-1")|.MacAddress')

And then use that MAC address to determine the name of the associated network interface:

docker exec -it mywork-mycontainer-1 ip -o l | grep "$mac_addr"

But I wish there was a more Dockery way to do this.

I'm not quite familiar with the startup sequence, but could the container's internal interface name be externalized into an environment variable that I could pass to my container?

[epuronta]

I'm deploying containers that are supposed to form a cluster over a specific overlay network. All the containers are also connected to another network facilitating ingress traffic (Docker Swarm).

If not specified, the application (out of my control) picks the interface to cluster on randomly. This obviously does not work, as the containers cannot communicate over the ingress network.

I have the option to configure the application to listen on a specific IP, but as Swarm allocates the addresses dynamically as containers are created, I have a hard time providing the IP as a parameter during container startup.

I also have the option to bind to a specific, named interface. This forces me to rely on having the overlay network mapped to a specific network inside the container (say, eth0).

Being able to control network naming inside the container would make this significantly easier.

[aleek]

Ok, so with little help I'm willing to implement this feature. I see two scenarios I could implement:

1. Just use the network name provided when creating the network docker network create <name> as a name of the internal interface name. This theoretically might break all of thousands containers that use eth0 internally, so it might be safer to do scenario 2.
2. Add new parameter to docker network create to specify internal name, i.e. docker network create --internal-name macvlan0

Of course this feature implies implementing also support in compose files for compose and stack.

I just need some initial confirmation from staff (@thaJeztah ) that this will be merged, when ready.

To add my use case to the big pile of use cases:
We have applications that have multicast input and multicast output. Input is one physical interface and output is another. Interfaces are linked through macvlan. Currently, we hack it to get it to work predictable.

Alex

[thaJeztah]

I don't think this should be a property of the network, but when attaching a network to a container (or service); see #25181 (comment)

docker run \
  --network name=my-network,iface=my-eth \
  --network name=my-other-network,iface=my-other-eth \
  nginx

docker compose can then use the option using the advanced syntax (similar to how aliases are set in this example; https://docs.docker.com/compose/compose-file/#networks)

services:
  some-service:
    networks:
      some-network:
        iface: my-eth
      other-network:
        iface: my-eth2

[aleek]

Excellent!

See you in a bit.

[h4ck3rk3y]

ran into something similar at a project I am working on, seems like the interface is chosen based on whats alphabetically lowest

we have a bridge network, if we create a new network lets call it cat then eth0 will get bridge, eth1 will get cat

on the other hand if i create a new network named bat then eth0 will get bat and eth1 will get bridge. this caused some confusion as we had some networking rules being set for eth1; and the non determinism made those rules not work.

[acalatrava]

ran into something similar at a project I am working on, seems like the interface is chosen based on whats alphabetically lowest

we have a bridge network, if we create a new network lets call it cat then eth0 will get bridge, eth1 will get cat

on the other hand if i create a new network named bat then eth0 will get bat and eth1 will get bridge. this caused some confusion as we had some networking rules being set for eth1; and the non determinism made those rules not work.

Your comment saved my day! I can use this as a hack to establish the priority :)
Thanks!