[Proposal] Use pion/ice for STUN/TURN NAT-traversal

[stv0g]

Hi @squat,

today I stumbled upon the wiretrustee project which enhances wireguard with some nice NAT-traversal techniques. Its largely based on the pion/ice package to handle endpoint discovery via STUN and even UDP proxying via TURN servers. Its bascially the same stack which is used by their WebRTC implementation. In case of a TURN connection a small proxy forwards packets from ICE to a local socket which gets used by the wirguard kernel module.
wiretrustee adds a signalling server which uses encrypted protobuf packages to exchange candidates.

All in all, I think their solution is the proper way of adding ICE NAT-traversal techniques to wiregard.
I am now considering using their design decisions to implement the same for kilo but replacing their signalling server with the K8S API server.

What do you think about it?

[colemickens]

This would be really amazing to see for cross-compatibility across wireguard clients. If the control servers have simple API surfaces and they exchange peer candidates similarly, it wouldn't be hard to imagine one client supporting more than just one control server.

[stv0g]

Hi @colemickens,

I started to work on an independent project which implements this: cunicu (formerly WICE).
Its still under active development and not ready for use or integration into kilo yet.

cunicu runs as a zero-configuration userspace daemon alongside the Wireguard kernel (or userspace) driver.
The configuration is done as usual directly against the Wireguard interface.

cunicu detects available Wireguard interfaces and changes in their peer configs and searches for valid endpoint addresses via the usual ICE procedure.

This means that cunicu can also be easily used without kilo in any other Wireguard setups.

cunicu also tries to stay out of datapath by using either NF/IPtables, or BPF to alter source/destination addresses and ports of the Wireguard UDP packets.

cunicu has pluggable backends. Which we use for exchanging signalling information (like SIP in VOIP).
I am currently working on three backends: Kubernetes APIserver, libp2p and a simple HTTP one.

Once a first release is ready, I am planning to submit a PR to kilo which adds cunicu as a new daemonset to the Kubernetes resources. Ideally this does not require any other change in the Kilo code.

[squat]

Hi @stv0g this sounds like an amazing project and in particular like something that would be really valuable for the Kilo ecosystem. Like you know, ever since closing #109, Kilo supports discovering WireGuard endpoints and as a result traversing certain NAT cones/configurations, but any additional capabilities in discovery and traversal would be extremely helpful.
I'd love to know more about the plans you have for WICE (I read the roadmap in the readme and it's impressive!) and about any ideas you have for integrations with Kilo. For example, would it make sense to deploy WICE as a core component of the Kilo stack and deprecate the endpoint discovery we've built into Kilo?

What would you all think about holding a community video call to talk about Kilo and WICE and discuss future roadmap for the project?

Take care!

[stv0g]

Hi @squat,

I am working on cunicu as part of my job as a researcher. Thats why there is a funding acknowledgement in the readme.

In the last days, progress on cunicu has slowed down a bit because I've been struggeling a bit to properly the code.
There are tons of different cases which need to be tested (direct connections, relayed via STUN, relayed via STUN and TCP, mixed variants, and more..).
Therefore, I spend some days in implementing a network testing tookit (Gont) which should allow me to test the NAT traversing capabilities of cunicu via unit tests.
Its very similar to Mininet but written and Go and be directly using within the unit tests.

For example, would it make sense to deploy cunicu as a core component of the Kilo stack and deprecate the endpoint discovery we've built into Kilo?

I already had the same idea. :)

In principle you could go even a step further. In the current state cunicu only focuses on endpoint discovery.
Peer discovery is also on the roadmap but not really required for my primary use case of Kilo + cunicu.
Peer discovery makes most sense in a setup where you don't have a central database like K8S.
For example, you can use libp2p's DHT to discover peers using a common rendevouz token. This simplifies the whole setup extremely.

[muvaf]

@stv0g I'm looking forward to use kilo in environments where I don't have the ability to have public IP. I do not have Wireguard-specific knowledge as much as you folks do but I've been coding in Go for the last couple of years. So, I can contribute to help you out finish the design objectives if you can give me some pointers 🙂

[stv0g]

Hi @muvaf,

Sure, I've got a bit stuck over the last weeks with other projects. But I plan to pick up the work pretty soon.
The main repo is WICE. Its a user space daemon which runs in the background and monitors Wireguard interfaces. Public keys of the Wireguard interfaces are exchanged via a pluggable Backend alongside with ICE candidates to establish an ICE handshake and hence UDP connectivity between two peers. This step is then repeated for each WG peer of each interface.

Most of the code is now written but lacks proper testing and validation.
For this purpose, I started to write a Mininet clone in Go named Gont.
Gont should allow us to test WICE using a larger set of Go unit tests to simulate a variety of different network setups (NAT, double-NAT, CGNAT, different firewalls configurations, etc..).

I could need some help with testing here. Any feedback is highly appreciated.

Currently, I plan to support three backends to exchange peer information:

• Kubernetes
• libp2p
• Simple HTTP REST or WebSocket

My testing has mostly been done with the HTTP backend. However, this one is still using a simple REST API.
I was planning to replace it with a WebSocket channel / relay between the peers because it would eliminate the busy polling and delay during the endpoint discovery.

Help here is also highly appreciated :)

[muvaf]

@stv0g My high level goal is to be able to create a Kubernetes cluster that any node in the public network can join, something like tailscale with proper CNI. And kilo is almost there except the NAT part, so I'm interested specifically in that part, which would be, from my understanding, endpoint discovery. Could you create more specific task issues in your repo that I can tackle? It's hard to know where to start as a newcomer with all these low-level networking stuff 🙂

[stv0g]

Hi @muvaf,

My current plan is to make cunicu and Kilo interoperable. Meaning that Kilo will handle peer discovery and cunicu will handle the endpoint discovery.

But adding Peer discovery also to WICE is a future goal. But its still far away. And I do not plan on replacing Kilo with cunicu.

[muvaf]

My current plan is to make WICE and Kilo interoperable.

@stv0g Yes, that's exactly what I'm interested. So, any pointers that I can start with to start contributing to make that happen would be appreciated.

[squat]

Hey @stv0g any update on cunicu? How can we get Kilo to interoperate with it and enable better automatic discovery? ❤️

[stv0g]

Hi @squat,

yes work on cunicu is progressing constantly but slowly, as I am doing this in my spare time.
The basic feature of endpoint discovery using ICE is implemented and could work alongside kilo as long as kilo does not touch the endpoints of the WireGuard peers.

cunicu has gained several other features similar to kilo as well:

• peer discovery
• synchronization of allowed-ips with kernel routes

However, the current code is still to experimental and unstable for giving it a first try.