IPv6 Support

[cvlc]

Uses an automatically generated ULA range for the bridge and guests.

No IP6Tables support (is it really necessary since IPv6 does not use NAT?) but I've tested the binary and confirmed that routing works just fine between the containers and host - do note that ULAs are not globally routable so IPv6 internet access is not possible as of yet.

There is some duplicated code for the IPv4-IPv6 decision trees - it seemed unavoidable, especially considering how strict Go seems to be about variable scope within if statements and the like.

All comments, criticism and ideas welcome; this is my first foray into Go, after all!

[cvlc]

Automatic public IPv6 address assignment (SLAAC) works as long as sysctl is configured to enable listening to IPv6 RAs and a router advertisment daemon such as radvd is pumping RAs through the bridge.

[pnasrat]

http://golang.org/pkg/net/#IP use net.IPv6zero

[cvlc]

Unfortunately this breaks things when dealing with individual bytes - []bytes seem directly convertible to IPs, but they are not as easily converted back. Keeping the IPs as []bytes seems to provide the most consistent results when doing low-level calculations and alterations. This applies to the other, similar annotations too!

[calavera]

@cvlc can you put an example of those broken things? for what it looks those objects are exactly the same than the constants that golang already provides. The IP type is just an array of bytes:

http://play.golang.org/p/JJDhH_oVjs
http://golang.org/src/pkg/net/ip.go?s=8421:8450#L35

[pnasrat]

Thanks for the pull request, I've made a basic first pass through review.

[tianon]

We'll also want to be careful with this that we don't suddenly put a container internet-facing now that the links feature is in and implies that only host connections or explicit links define valid paths into the container. (Don't accidentally want redis to be internet accessible. :D)

As long as we keep that in mind, I'm all for IPv6 support!

[cvlc]

Thanks, pnasrat, I'll heed the advice and make amendments as appropriate ASAP!

tianon: I don't think that's really a worry for this particular implementation - ULA addresses are totally unroutable on the internet. A user has to utilize radvd or a similar service to provide router advertisements so containers can reach outside of the private network and configure their kernel to accept such advertisements. Since there's no such thing as NAT in IPv6, this comes as close to a safe default 'local network configuration' as possible :)

[tianon]

@cvlc thanks for clarifying about ULA, that sounds great to me :)

[cvlc]

This syntax seems to be the best I can get - net.IPv6zero causes issues with the IP allocator (it returns duplicate pairs of addresses) that I've not had much success debugging. Is usage of net.IPv6zero strictly necessary?

[vieux]

LGTM

[cvlc]

Seeing as I've not had any further complaints, I'll merge and rebase this evening.

[cvlc]

Okay, that's the commit all squashed and ready to merge.

ping @crosbymichael, @jpetazzo, @shykes (I think that you guys might be interested)

[SvenDowideit]

@cvlc is there a chance that you could write about how it works in docs/source/use/networking.rst and assuming that it makes some changes to how the commandline is used, in the relevant sections of docs/source/commanline/cli.rst ?

I'm hoping that what you write will teach me something about ipv6 :)

[creack]

@cvlc Sorry for the delay. Could you rebase?

[vieux]

@cvlc network code changed a lot. Do you think you could rebase your PR ?

Thanks

[cvlc]

Hi there, sorry for the delayed response - a busy new year and a new job. I'll rebase when I find the time (and when the network code has settled a little), but until then any other dev can feel free to have a hack at this and throw me any questions they have.

[vieux]

@cvlc thanks

[ajwdev]

I have a rough, work-in-progress branch that has master as of a week ago merged in with @cvlc's branch. If my notes are correct the last things that need ported/written are requesting IPv6 addresses in networkdriver/ipallocator/allocator.go and creating the necessary plumbing for setting the IPv6 address on the LXC container.

Majority of @cvlc's code just needed moved to the 'networkdriver' package and have their tests updated. For new stuff I tried my best to add IPv6 support to existing methods without breaking the API. In cases where I couldn't (or didn't initially know how to) I made a new function that indicated it was for IPv6 just like this original PR did. The one exception is GetNameserversAsCIDR which I renamed to GetIPv4NameserversAsCIDR and then created GetIPv6NameserversAsCIDR.

I intentionally haven't squashed my commits yet in case someone wanted to cherry-pick any of them.

https://github.com/TheDude05/docker/commits/ipv6_support

[cvlc]

Excellent stuff! I had a cursory look and everything seems fine, though you may want to create your own pull request when you're done and ready to merge everything in. Be sure to link it here if you do!

[vieux]

@TheDude05 Do you think you'll be able to do a PR soon ?

[ajwdev]

@vieux No promises but hoping to have this knocked out by next week if time permits. Doing some testing with it now and finding/fixing bugs as I go along. After that I'll take a run through the code and make sure there are adequate tests in place around the new stuff, write some docs, and then create a pull request.

[vieux]

@TheDude05 do you have time to work on this ?

[ajwdev]

@vieux Hey sorry for the lack of updates. My life/work commitments took up more time than I expected. I should start having time to jump back into this again.

To bring everyone back up to speed, my branch has the basic IPv6 support outlined in the original branch working as far as I could tell. The part I was working on before I had to stop was getting ip6tables/IPv6 NAT support working on kernels that supported it. If that functionality is not a concern at the moment I could probably get something pushed up relatively quickly. Thoughts?

[ajwdev]

I found some time this weekend to start poking at this again since expanding the scope with the IPv6 NAT stuff. I need to implement IPv6 support with Docker Links, fix a portion of my code in pkg/iptables to not use the Docker core GetKernelVersion function, and then merge master again to grab the changes from 0.9 release.

[shykes]

Hi @TheDude05, we're going to close this for now since it seems unfinished. Feel free to re-open when you are ready for review, and of course to ask for help on irc.

Thanks!

[ajwdev]

@shykes Sounds good. Unfortunately I was trying to balance too many things at once when I started work on this. I'd like to try and pick it back up at some point.

@tmartinx I haven't tried it myself but I think what @cvlc outlined in comment #2 of this thread should work. If you ran something like "radvd" on the host and attached it to the docker0 bridge, your containers should get IPv6 addresses if they're configured to handle router advertisements. Again, I haven't tried this myself so it may not work like I expect.

And while I agree that we don't need IPv6 NAT due to address allocations running out, we still need NAT to perform the port mappings between the host and the container just like we do now with IPv4. The IPv6 addresses that are generated/assigned are Unique Local Addresses as defined in RFC4193 and therefore are not publicly routable. This means you also need masquerade support to reach the public IPv6 internet from those containers.

I've been going with the assumption that the IPv6 support should match that of IPv4, but maybe this is a good time to get clarification from the Docker team on what their vision for this feature is. What is the expected experience for users wanting to use IPv6 on their containers? For example should a user be able to map IPv6 ports the same way ports are currently mapped with IPv4? Will we ever want to give containers public IPv6 addresses or should we only be assigning private addresses? Etc.

[ghost]

Hey Andrew, tks for your answers!

Replying below...:

On 7 May 2014 22:07, Andrew Williams notifications@github.com wrote:

@shykes https://github.com/shykes Sounds good. Unfortunately I was
trying to balance too many things at once when I started work on this. I'd
like to try and pick it back up at some point.

@tmartinx https://github.com/tmartinx I haven't tried it myself but I
think what @cvlc https://github.com/cvlc outlined in comment #2https://github.com/dotcloud/docker/issues/2of this thread should work. If you ran something like "radvd" on the host
and attached it to the docker0 bridge, your containers should get IPv6
addresses if they're configured to handle router advertisements. Again, I
haven't tried this myself so it may not work like I expect.

Precisely! It works like this with radvd!

My radvd contains:

---

interface docker0
{

Enable RA on this gateway:

AdvSendAdvert on;

Disable clients from getting their IPs from DHCPv6 (no need for it):

AdvManagedFlag off;

Disable clients from getting other configs from DHCPv6:

AdvOtherConfigFlag off;

More options:

AdvLinkMTU 1500;
AdvDefaultPreference high;

Enable RA to the following subnet

prefix 2804:290:4:acbd::/64

{
AdvOnLink on;
# Allow clients to auto-generate their IPv6 address (SLAAC),
# DHCPv6 will not be used here.
AdvAutonomous on;
};

Ubuntu IPv6 clients requires the package rdnssd to deal with RDNSS (Not

required for Docker containers if the host have it under its
/etc/resolv.conf as usual):
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL yourdomain.com { };

};

All Docker containers have now IPv6 automatically... =D

---

docker.io run --privileged -t -i tmartinx/ubuntu-14.04-server-cloudimg

/bin/bash

But, lots of times I need a fixed IPv6 address based on MAC / SLAAC, so,
I'm running my Docker containers like this:

---

docker.io run --privileged
--lxc-conf="lxc.network.hwaddr=7a:72:33:68:4d:f8" -t -i

tmartinx/ubuntu-14.04-server-cloudimg /bin/bash

And while I agree that we don't need IPv6 NAT due to address allocations

running out, we still need NAT to perform the port mappings between the
host and the container just like we do now with IPv4. The IPv6 addresses
that are generated/assigned are Unique Local Addresses as defined in
RFC4193 and therefore are not publicly routable. This means you also need
masquerade support to reach the public IPv6 internet from those containers.

Well, I don't like to use workarounds... And that is precisely what NAT is.
:-P

I'm pretty sure I'll never use NAT66. IPv6 is already working smoothly, no
need to complicate it... All I need is "ip6tables -t filter"... I'm not
using port mappings and everything is running very smoothly...

BTW, I see that lots of suites/appliances provides NAT44 as a "solution" to
"fix" routing problems... Where you just need a route to a subnet, there is
a NAT to literally mask the routing problem (OpenVPN AS suggest that, for
example, Zentyal-OpenVPN too, it is so rooted)... From my point of view,
NAT66 is a bad thing and people should avoid using it.

The Internet should be routed, not NATed... I think that it works much
better when we have a network organized like a tree, not like "bubbles"...
And with NAT, the IPv4 Internet is a network of isolated "bubbles"... It
looks ugly... If you know what I mean... Just my two Bitcents...

But, nevertheless, people must have the option to enable / disable this
kind of features, I know that... Please, don't take me wrong... ^_^

I've been going with the assumption that the IPv6 support should match that

of IPv4, but maybe this is a good time to get clarification from the Docker
team on what their vision for this feature is.

NAT66 can wait... It is not needed, really... I don't see any use for if
(of course, this is only my own view)...

What is the expected experience for users wanting to use IPv6 on their

containers?

Just have native IPv6 support, that's it. So, we'll not need to "apt-get
install lxc" just to be able to use IPv6 with Docker. It must have native
support built-in.

For example should a user be able to map IPv6 ports the same way ports are

currently mapped with IPv4?

You mean, like port forward (NAT44 / DNAT / SNAT)? Well, no... I mean,
why?! :-P

lol

Will we ever want to give containers public IPv6 addresses or should we

only be assigning private addresses? Etc.

That's what I'm doing, each container have its own public IPv6...

I'll use private IPv6 address only, and only if, that subnet will remain
completely isolated and will never reach the Internet. If it needs to reach
the Internet in someway (apt-get update/upgrade ?), I'll use a proxy within
that subnet for that and, if the Internet needs to reach this private
subnet, well, it is not so private anymore (I think), so, I'll replace it
with a public one instead... NAT66? No, thanks...

So, I truly believe that NAT66 support in Docker can wait a bit more and
IPv6 is a huge requirement... For example, I'm building an OpenStack cloud,
with Docker and Heat, I'm planning to have millions of Docker instances...
I can not imagine a environment like this, with NAT, no way... Also, I have
no plans to use IPv4, IPv6-Only networks seems a "must to follow" "best
practice" these days...

Please, forgive me if I'm being rude, that is not my intention... E-mail is
complicated and English isn't my native language... It is hard to better
choose the words... :-)

Cheers!
Thiago

[LK4D4]

@TheDude05 Hi, you doing very important work. May be you should create new pull request with your changes, so we can link it here and to #3609?
+1 to @tmartinx that we need IPv6, not new shining NAT66.

[zargony]

+1 for IPv6 support (no NAT, with IPv6 you don't need NAT anymore)

I'd be looking forward to be able to route a static address into a container without workarounds like custom lxc configs or static mac addresses. Unfortunately I don't know Go, so I can't help with the code, but I'm willing to help with testing and stuff like that

[niclashoyer]

Any news on this? I'd like to vote, too (+1).
My use case covers cross host container networking. I would like to provide every container with an IPv6 address, so they can connect to each other directly.

[ghost]

I would like to vote +1000 for this! I'll not be able to use Docker until
it have native IPv6 support built in.

Also, a bit off-topic but, OpenvSwitch support is a must to have!

I'm planning to use Docker with OpenStack and, of course, with IPv6...
Millions of Dockers instances and each one with its own public IPv6, can't
wait for that! ^_^

Cheers!
Thiago

On 23 June 2014 19:13, niclashoyer notifications@github.com wrote:

Any news on this? I'd like to vote, too (+1).
My use case covers cross host container networking. I would like to
provide every container with a IPv6 address, so they can connect to each
other directly.

—
Reply to this email directly or view it on GitHub
#2974 (comment).

[LK4D4]

If I'll have time, I'll begin to implement this myself soon.

[cvlc]

Hi all,

To add to this discussion - it's my understanding that port forwarding/NAT66 is used with Docker only to conserve IP addresses, something totally unnecessary with IPv6. Then again, there are security implications for non-IPv6 aware users if all containers are given a publicly-routable IPv6 address without any firewall or other network-level controls.

As a happy middle ground, I'd suggest assigning each Docker container a unique IPv6 address. To provide the closest parallel to the expected IPv4 behaviour, I'd further suggest a ULA (unique local address) for each container provided by the Docker service in the same way that it provisions IPv4 local addresses. The alternative is to not assign addresses at all, but this will mean effectively nothing has changed (radvd or similar will need to be used).

Of course, users would still need to use radvd or a similar daemon if they want their Docker containers to become publicly routable - It'll be fairly easy to work out documentation for this. It's entirely possible to have multiple IPv6 and IPv4 addresses on a single interface simultaneously, this will permit local communication with other containers and the host through their ULA addresses.

All outgoing internet traffic will continue to go over IPv4/NAT and each container's network namespace can be firewalled via ip6tables to only allow traffic from other containers to specific ports.

I'd love to hear others' thoughts on this approach! I was considering picking this up again but I'm reluctant to invest the time without confirmation that I'm going the right way about it for Docker.

[niclashoyer]

I think there should be options for both and we should select a reasonable default, e.g.

The default way (no IPv6), just IPv4 port mapping:

docker run -d -p :6379 redis

IPv6 local (assign a link-local address to the docker container. Could be default, too):

docker run -d --ipv6 redis

IPv6 public (Assign a public IPv6 address if there is a public subnet available on the host. Shouldn't be default):

docker run -d --ipv6-public redis

And mixing the arguments shouldn't be a problem:

docker run -d -p :6379 --ipv6 --ipv6-public redis

edit: I don't like the radvd approach, because that requires containers running in privileged mode at the moment, which is quite heavy.

[cvlc]

IPv6 link-local addresses should be added regardless according to spec, I'd probably have --ipv6 allocate a ULA (good enough for inter-container communication on the same host or sub/net). I wouldn't add routable addresses via the Docker address provisioner since that's not done with IPv4. To keep the behaviour as similar as possible, I'd try to fix the issue with radvd or use DNSMasq as a SLAAC or DHCPv6 server!

[niclashoyer]

You're right that it is not done with IPv4, but the problem with IPv4 is that we simply don't have enough free addresses to add them as public routable. I'd also be happy if a simple radvd configuration could provide the same (if there is good documentation), but having to start every container with --privileged is a security issue.

[cvlc]

I've used radvd/dnsmasq in the past to provide DHCPv6 addresses via a dhcp client outside the container's environment (within it's network namespace), thereby avoiding --privileged altogether. You can see this method used in @jpetazzo's pipework (https://github.com/jpetazzo/pipework).

[niclashoyer]

great! If there is a way to automate the assignment of an IPv6 address after container startup without privileged mode I'd also be happy with that. Is there any documentation for that scenario?
Edit: one thing that still bothers me: If the IP is assigned outside from the container using radvd/dnsmasq, is there any way I could get the IP by a docker hash? At the moment I'm integrating docker containers using docker inspect, but if docker does not know the IPv6 address it is not available to the docker HTTP API.

[ajwdev]

For some background, the reason I was working on the NAT66 support was so that containers with IPv6 addresses could hit the public IPv6 internet just as containers with NAT'd IPv4 addresses could. Basically I was just trying to match the current behavior. Also, so far in my experience all of the cloud providers that offer public IPv6 connectivity do it by handing you a single IPv6 address, not a subnet for SLAAC.

I started looking back at this again last week and I don't think it'd be too difficult to rip out the IPtables stuff. It'll likely be more time consuming to merge with 1.0 code base. I'll also be at Velocity this week if anyone is here and wants to discuss in person.

[niclashoyer]

After all I think docker should support both or none. Or, maybe the best solution, develop NAT66 and subnet implementations as separate plugins, if that kind of plugins are possible.

[cvlc]

@TheDude05 I know that OVH and online provide /48s or /64s, SixxS also provide public-routable subnets for free if you have a good reason for one, though additional configuration will be required as they're tunnelled via IPv4.

NAT66 isn't ideal, but perhaps the reality is that it'd be more immediately useful to dual-stack users. I do understand why you chose that approach, @TheDude05 - somebody certainly needs to make the call! The problem is that it's not likely going to work very well for an IPv6 user's imagined use case and won't allow Docker it's full scaling capabilities, adding native IPv6 support is a Big Thing in terms of future potential, especially when scaled to thousands of microservices across a large platform. I do like your plugin idea, @niclashoyer!

[porjo]

@tmartinx wrote:
I'm pretty sure I'll never use NAT66. IPv6 is already working smoothly, no need to complicate it... All I need is "ip6tables -t filter"... I'm not using port mappings and everything is running very smoothly...

Agreed! Address allocation using autoconf with radvd (or similar) is the way to go. If you know what IPv6 prefix is being advertised to your containers and you know each container's MAC, then you have all the information you need for the IP. #4033 relates to this question.

@TheDude05 wrote:
cloud providers that offer public IPv6 connectivity do it by handing you a single IPv6 address, not a subnet for SLAAC.

This, or they provision a /64 'on link' which means you then have to bridge docker0 to your host's outside interface (assuming they don't also filter on MAC which many do!). This is a real problem but one that will hopefully change as demand for routed v6 increases.

[niclashoyer]

Thanks @porjo ! A working configuration for radvd + docker is here in a gist.

[ajwdev]

Was able to rip out all of the IPv6 IPtables/Portmapper code from my branch below. Note that this was based on master of 0.8 so I'll still need to do some work on making this work with master as of today

https://github.com/TheDude05/docker/compare/ipv6_no_nat

[root@centos-docker ~]# docker run -t -i centos ifconfig eth0
eth0      Link encap:Ethernet  HWaddr EE:4A:FB:54:66:9C  
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fc00:0:dac:b995::2/64 Scope:Global
          inet6 addr: fe80::ec4a:fbff:fe54:669c/64 Scope:Link
          UP BROADCAST  MTU:1500  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:90 (90.0 b)  TX bytes:110 (110.0 b)

[LK4D4]

@TheDude05 Ah, you are back! I will be happy to help you with rebase or something like this. Would you mind to create separate PR with your branch, so we all can see diff?

[tianon]

If you go to the link and click on "files changed", you can see the diff
already. ;)

[LK4D4]

@tianon From Nov 30th and from another contributor. Or I don't see something? :)

[tianon]

this was based on master of 0.8 so I'll still need to do some work on making this work with master as of today

So I think this is just a rebase, right @TheDude05 (and on a quite old master at that)? That would explain older commits. :)

[ajwdev]

I went ahead and squashed all of my commits and put up a new PR here: #6923

Note that if you want to see the original commit log pre-squash you can have a look at this branch: https://github.com/TheDude05/docker/compare/ipv6_no_nat_pre_merge

@tianon Yes it should just be a matter of rebasing/merging master but because of the extent of the changes that happened between 0.8 and 1.0 a simple rebase/merge wont suffice. Several files were moved/deleted and libcontainer was added since work was done on this branch. Also, we should determine how we want to deal with links if a user has disabled ICC and wants to talk over IPv6.

I signed my squashed commit with the Docker DCO so there shouldn't be any legal issues if someone wants to take my branch and continue work on it.

[RmsMit]

I have been looking into how to contribute code because I really think IPv6 is important and was quite disappointed it was not in from the beginning. to be a little extremist, Docker could have skipped IPv4 entirely by assigning IPv6 link-local addresses to each container (instead of 172.x.x.x.) and then doing to port mapping to the host as external eth to that IPv6 link-local address. If the user wanted IPv4 then the host external eth would be IPv4 and the port would be pipped through to the container as it currently is. ie
(host-192.168.0.11 port 80)-> (container - FF80::DOC0:1111:2222:3333 on port 8080)

This could be taken further and have the 1111:2222:3333 component of the IPv6 address be set to the containerID (the D0C0 part of the address could be used as an indicator of a docker address).

If the container was to be exposed to the world then all you would need to do is use the IPv6 Router Advertisement to set the first 64bits (as with normal IPv6 auto conf) and use the host forward IP routing to expose the container.

This would also make it easier for those wanting to go directly out a physical interface from the container which I have seen some people do and graph the performance advantage.

The expose ports settings could then be used to just set firewall open ports if security is a concern.

PS I am only a basic level programmer, have not programmed in GO before and don't know anything about how docker is coded but I would really like to help with this feature. I do pretty good documentation and could offer use-case testing.

[LK4D4]

@RmsMit Take a look at #6923. Choose some little part that can be rewritten without huge impact on daemon and try to adapt it to existing master code. I post #6975, but have a little time to do it now.