Tracking issue for QUIC support

[kpp]

There are a couple of reasons to support QUIC network protocol.

The first, slightly minor, is that validators are maintaining more than 1000 TCP connections open and are running into file descriptor limits.
Second is we use Yamux for multiplexing multiple substreams into one, which is a rather poor protocol that does the best it can on top of TCP, but can't do more.
Because we can't know the TCP window size from user space, we can't actually properly allocate window sizes to individual substreams. If two or more substreams send a big volume of data, we'll run into head of line blocking issues (This is the same reason why HTTP2 is not so great and is being replaced with HTTP3, if you want to read more)
Third reason is we're suffering from a TCP slow start issue. A TCP connection is almost unused, and suddenly we send for example 2 MiB on it. Because of TCP slow start, it's going to take a rather long time to send these 2 MiBs even if the connection would be capable of sending them quickly.

First attempt was made in paritytech/substrate#6366.

So far there is https://crates.io/crates/libp2p-quic which should be reviewed.

The list of sub issues will be updated time to time:

• TLS 1.3 vs Noise. The golang version uses TLS (see https://github.com/libp2p/specs/blob/master/tls/tls.md#peer-authentication), also there is no nQUIC specs, so to be compatible with libp2p we have to use TLS+QUIC.
• Inspect whether we can replace x509-signature with picky. The answer is we can replace it with any crate we can including x509-parser, picky, etc...
• Implement x509 cert serializer, parser, verifier Add initial code handling certificate kpp/rust-libp2p#1
• Implement verification of all signature algorithms according to TLS 1.3 standard Support all verification algorithms kpp/rust-libp2p#4
• Implement rustls ServerCertVerifier & ClientCertVerifier Implement rustls ServerCertVerifier & ClientCertVerifier kpp/rust-libp2p#6
• Make libp2p-quic binary compatible with the go implementation (a standalone issue) Make tls cert binary compat with the go implementation ipfs-rust/libp2p-quic#6
• R&D: a basic QUIC server R&D: accept tls handshake and read a simple QUIC message kpp/rust-libp2p#8
• The existing crate libp2p-quic now supports TLS, but there still some issues left.
• Integrate libp2p-quic into rust-libp2p with the help of its maintainers Libp2p quic second attempt libp2p/rust-libp2p#2159 transports/quic: Add implementation based on quinn-proto libp2p/rust-libp2p#2289
• Integrate libp2p-quic into substrate Add experimental support for QUIC substrate#11514.

[burdges]

Yes. QUIC would improve our networking significantly. :)

We previously got hung up on keeping Noise IK via nQUIC = QUIC + Noise vs adopting QUIC with TLS 1.3 by defining some new grandpa TLS certificate for validators, crazier certificates for collators, and maybe others.

I'm completely happy outsourcing an initial spec for one or both of these. It's also fine if we do something fast and warn everyone to expect yet another network protocol migration. It's conversely also possible that @tomaka knows some wish-like of libp2p messes that would be good to bundle in with this big a migration.

I'd expect a "grandpa TLS certificate" for validators to consist of a Merkle proof identifying the Ed25519 grandpa session keys, and then a certificate by the session key on the transport key. We should first migrate storage from our radix 16 hashing, which bloats in a dense Merkle tree, to radix 2 hashing with hash fast forwarding (and radix 16 caching). We'll want crazy certificates for collators, like VRFs for parachains running Babe and worse for Sassafras in future.

I've no idea if we properly exploit Noise IK yet anyways, meaning whether nodes initiating connections identify themselves first via something vaguely like this "grandpa TLS certificate". It's annoying if the Merkle proof that shows your actually a validator does not fit into the MTU used for the first datagram, so that 4x space cost of our radix 16 hashing matters here too.

In brief, we need to decide how quickly we want this and what we want to try to rush to merge into such a migration. We'll surely punt on doing authentication perfectly optimal initially, but we're slowly creating the possibility for better authentication elsewhere so it'll circle back here one day. :)

[burdges]

I think the simplest would be QUIC with TLS 1.3 and no authentication, but authentication done late inside the stream. We simultaneously give someone reputable a grant for designing nQUIC properly. And push our storage folk to fix our radix 16 hashing issue.

[tomaka]

I'd strongly prefer for us to implement something that is already specified (i.e. TLS+QUIC) and not start creating parity-nquic that will get abandoned after 6 months.

[kpp]

So far I did a research and implemented:

1. a go tool to generate an x509 certificate with libp2p extension in DER format (the code is mostly a copypasta from https://github.com/libp2p/go-libp2p-tls);
2. a rust tool to parse, inspect and verify the signature according to libp2p docs.

The reason I did it was I wanted to be compatible with the go implementation of libp2p-quic. In the future I want to test our client with their server and vice versa.

https://gist.github.com/kpp/c9c84411e17f4b27dddf0d438b289862

The code is ugly but it's not the point. The point is that the rust tool is binary compatible.

The next step is to prettify the code, implement a certificate serializer and create a PR.

Also I found a tool in the Internets to read DER files which helped me a lot to inspect the certificate: https://lapo.it/asn1js.

[kpp]

So far I:

• updated tool to generate certs: https://gist.github.com/kpp/c9c84411e17f4b27dddf0d438b289862
• implemented verification of certs with rsa-pkc1, ecdsa, ed25519, rsa-pss : Support all verification algorithms kpp/rust-libp2p#4

[kpp]

These two weeks I:

• Worked on Implement rustls ServerCertVerifier & ClientCertVerifier kpp/rust-libp2p#6
• Tried to figure out how to work with quinn_proto: R&D: accept tls handshake and read a simple QUIC message kpp/rust-libp2p#8
• As a standalone issue applied a fix to the libp2p-quic crate: Make tls cert binary compat with the go implementation ipfs-rust/libp2p-quic#6
• Found an issue in the libp2p tls specs: Fix tls.md libp2p/specs#357

[kpp]

While I was playing with quinn_proto, David added TLS support into his libp2p-quic crate. We had a chat with Pierre and came to a conclusion the code is a good starting point. There are some issues left but so far it looks pretty good.

Since the last time I worked on multiple issues:

• Made a code review of libp2p-quic
• Fixed libp2p-quic: Make tls cert binary compat with the go implementation ipfs-rust/libp2p-quic#6
• Worked on specs: Fix tls docs according to specs kpp/rust-libp2p#10, Define precisely how to marshal SubjectPublicKeyInfo structure in tls.md libp2p/specs#358
• Found several issues in the go implementation: Support critical libp2p extension libp2p/go-libp2p-tls#87 Update go-libp2p-tls to 0.2.0 libp2p/go-libp2p-quic-transport#226
• Improved x509 parser: Add SubjectPublicKeyInfo::raw field rusticata/x509-parser#100
• Worked on extracting UDP code into a separate code (Extract udp crate quinn-rs/quinn#1180 quinn_udp: use async_io instead of tokio quinn-rs/quinn#1183) which will help libp2p-quic to reuse the code from quinn.

The current progress is: we are integrating libp2p-quic into rust-libp2p: libp2p/rust-libp2p#2159.

[kpp]

dvc94ch dropped, so I opened my own PR: libp2p/rust-libp2p#2289

• A lot of fixes being made:
• it was requested to remove noise support libp2p/rust-libp2p@25ee1fe
• generic crypto code was removed: libp2p/rust-libp2p@f35fde1
• we can dial without PK: libp2p/rust-libp2p@3b1ff34
• I replaced barebones-x509 to deal with X509 certs: libp2p/rust-libp2p@63e4696
• added some basic QUIC support to libp2p-perf: Actually run quic benches mxinden/libp2p-perf#56

Here are bench results:

Bench results

Local-Local TCP

# Start Rust and Golang servers.

# Rust -> Rust

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.00 s	5913 MBytes	4730.32 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.00 s	9203 MBytes	7362.40 MBit/s

# Rust -> Golang

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.00 s	5458 MBytes	4366.22 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.00 s	10284 MBytes	8227.13 MBit/s

# Golang -> Rust

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.00 s 	6880 MBytes	5502.57 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.00 s 	17534 MBytes	14026.69 MBit/s

# Golang -> Golang

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.00 s 	4881 MBytes	3904.79 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.00 s 	23115 MBytes	18489.50 MBit/s


Local-Local QUIC:

# Start Rust and Golang servers.

# Rust -> Rust

## Transport security noise
Local peer id: PeerId("12D3KooWAEgNvJB6tXtmpgjf2GDZjMhCneZvtUo2KTR4if4kkoH7")
Interval	Transfer	Bandwidth
0 s - 10.00 s	361 MBytes	288.72 MBit/s

## Transport security plaintext
Local peer id: PeerId("12D3KooWHiYCM8HrETLwqDwAwo3ovHWgAo1astGG2bLJ7VdydA95")
Interval	Transfer	Bandwidth
0 s - 10.00 s	381 MBytes	304.77 MBit/s

# Golang -> Golang

## Transport security noise
2021/12/09 14:35:18 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.00 s 	725 MBytes	579.98 MBit/s

## Transport security plaintext
2021/12/09 14:35:28 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.00 s 	724 MBytes	579.09 MBit/s


Local-Server TCP (mixed plaintext/noise):

# Rust -> Rust

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.02 s	126 MBytes	100.64 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.09 s	125 MBytes	99.08 MBit/s


# Rust -> Golang

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.21 s	111 MBytes	86.97 MBit/s

# Golang -> Rust

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.03 s 	129 MBytes	102.92 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.01 s 	125 MBytes	99.85 MBit/s

# Golang -> Golang

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.15 s 	89 MBytes	70.15 MBit/s



Local-Server QUIC:

# Rust -> Rust

## Transport security noise
Local peer id: PeerId("12D3KooWR1KRW9UoJd8XXwFLKWdvmC4yTEDgStvXJGUD3ZFnFZDw")
Interval	Transfer	Bandwidth
0 s - 10.01 s	9 MBytes	7.19 MBit/s

## Transport security plaintext
Local peer id: PeerId("12D3KooWA7vqQpUV3SxUNTWWeWHiM2N1WEebMPETwhLLtmD6QdnL")
Interval	Transfer	Bandwidth
0 s - 10.04 s	4 MBytes	3.19 MBit/s


# Golang -> Golang

## Transport security noise
2021/12/09 17:42:04 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.01 s 	109 MBytes	87.12 MBit/s

## Transport security plaintext
2021/12/09 17:42:14 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.01 s 	121 MBytes	96.71 MBit/s

Also we still cannot connect to go-libp2p-quic. I am working on it.

[burdges]

It's QUIC with QUIC's standard TLS 1.3 then? Cool we reinvent enough around here anyways. ;)

[kpp]

I discovered that I pollute Swarm threads with StreamMuxer::poll_event with:

let span = tracing::span!(tracing::Level::TRACE, "handle_event").entered();
while let Poll::Ready(event) = inner.endpoint.poll_channel_events(cx) {
    inner.connection.handle_event(event);
}
drop(span);

let span = tracing::span!(tracing::Level::TRACE, "send_transmit").entered();
let max_datagrams = inner.endpoint.max_datagrams();
while let Some(transmit) = inner.connection.poll_transmit(now, max_datagrams) {
    inner.endpoint.send_transmit(transmit);
}
drop(span);

And they are responsible for parsing packets and building packets respectively. I believe that's the main issue for poor performance.
tracing.folded.txt

[kpp]

The current state of this issue:

• We got a robust implementation in transports/quic: Add implementation based on quinn-proto libp2p/rust-libp2p#2289
• There is a tool to bench the performance Add support for QUIC mxinden/libp2p-perf#74
• I am working on integration: Add experimental support for QUIC substrate#11514

[kpp]

• TLS code is merged in transports/tls: Add libp2p-tls as per spec libp2p/rust-libp2p#2945
• The implementation is almost ready to merge: transports/quic: Add implementation based on quinn-proto libp2p/rust-libp2p#2289 thanks to the huge help from Protocol Labs.
• There is a new competitive implementation with less code: Quic attempt 4 libp2p/rust-libp2p#2801

[burdges]

Any idea what the performance looks like?

[kpp]

I am working on it.

[kpp]

libp2p/rust-libp2p#2289 is merged.

[kpp]

@burdges

localhost: QUIC with libp2p-perf

$ ./run.sh 
# Start Rust and Golang servers.
Local peer id: PeerId("Qmcqq9TFaYbb94uwdER1BXyGfCFY4Bb1gKozxNyVvLvTSw")
about to listen on "/ip4/127.0.0.1/udp/9992/quic"
Listening on "/ip4/127.0.0.1/udp/9992/quic".

# Rust -> Rust

Local peer id: PeerId("12D3KooWH1oPSwuvp5v5AUqvGT3HR64YZRC7VZmhVZkewiLcPDpa")
IncomingConnection { local_addr: "/ip4/127.0.0.1/udp/9992/quic", send_back_addr: "/ip4/127.0.0.1/udp/54502/quic" }
ConnectionEstablished { peer_id: PeerId("12D3KooWH1oPSwuvp5v5AUqvGT3HR64YZRC7VZmhVZkewiLcPDpa"), endpoint: Listener { local_addr: "/ip4/127.0.0.1/udp/9992/quic", send_back_addr: "/ip4/127.0.0.1/udp/54502/quic" }, num_established: 1, concurrent_dial_errors: None }
Behaviour(PerfRunDone(10.001692342s, 4396669396))
Interval	Transfer	Bandwidth
0 s - 10.00 s	4396 MBytes	3516.68 MBit/s
ConnectionClosed { peer_id: PeerId("Qmcqq9TFaYbb94uwdER1BXyGfCFY4Bb1gKozxNyVvLvTSw"), endpoint: Dialer { address: "/ip4/127.0.0.1/udp/9992/quic", role_override: Dialer }, num_established: 0, cause: None }
ConnectionClosed { peer_id: PeerId("12D3KooWH1oPSwuvp5v5AUqvGT3HR64YZRC7VZmhVZkewiLcPDpa"), endpoint: Listener { local_addr: "/ip4/127.0.0.1/udp/9992/quic", send_back_addr: "/ip4/127.0.0.1/udp/54502/quic" }, num_established: 0, cause: Some(IO(Custom { kind: Other, error: ApplicationClosed(ApplicationClose { error_code: 0, reason: b"" }) })) }

# Rust -> Golang

Local peer id: PeerId("12D3KooWLy9RkY4uW26ryp3seZ6QXuiJbmBywVE359MnQopnYEj2")
Interval	Transfer	Bandwidth
0 s - 10.00 s	3062 MBytes	2449.60 MBit/s
ConnectionClosed { peer_id: PeerId("12D3KooWL3XJ9EMCyZvmmGXL2LMiVBtrVa2BuESsJiXkSj7333Jw"), endpoint: Dialer { address: "/ip4/127.0.0.1/udp/9993/quic", role_override: Dialer }, num_established: 0, cause: None }


# Golang -> Rust

2022/08/05 15:10:43 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
IncomingConnection { local_addr: "/ip4/127.0.0.1/udp/9992/quic", send_back_addr: "/ip4/127.0.0.1/udp/54359/quic" }
ConnectionEstablished { peer_id: PeerId("12D3KooWKGLyqMFxSXgqv8n1mwX3Ljxg18ENFTtuiHTRYE8DHxrv"), endpoint: Listener { local_addr: "/ip4/127.0.0.1/udp/9992/quic", send_back_addr: "/ip4/127.0.0.1/udp/54359/quic" }, num_established: 1, concurrent_dial_errors: None }
Interval 	Transfer	Bandwidth
0s - 10.00 s 	2160 MBytes	1727.97 MBit/s
Behaviour(PerfRunDone(9.999945043s, 2160640000))
ConnectionClosed { peer_id: PeerId("12D3KooWKGLyqMFxSXgqv8n1mwX3Ljxg18ENFTtuiHTRYE8DHxrv"), endpoint: Listener { local_addr: "/ip4/127.0.0.1/udp/9992/quic", send_back_addr: "/ip4/127.0.0.1/udp/54359/quic" }, num_established: 0, cause: Some(IO(Custom { kind: Other, error: ApplicationClosed(ApplicationClose { error_code: 0, reason: b"" }) })) }

# Golang -> Golang

2022/08/05 15:10:53 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.00 s 	2085 MBytes	1667.91 MBit/s

localhost: TCP with libp2p-perf

   Compiling libp2p-core v0.32.1
   Compiling libp2p-noise v0.35.0
   Compiling libp2p-plaintext v0.32.0
   Compiling netlink-proto v0.10.0
   Compiling rtnetlink v0.10.1
   Compiling if-watch v1.1.1
   Compiling ed25519-dalek v1.0.1
   Compiling toml v0.5.9
   Compiling proc-macro-crate v1.1.3
   Compiling multihash-derive v0.8.0
   Compiling multihash v0.16.2
   Compiling multiaddr v0.14.0
   Compiling libp2p-swarm v0.35.0
   Compiling libp2p-dns v0.32.1
   Compiling libp2p-tcp v0.32.0
   Compiling libp2p-yamux v0.36.0
   Compiling libp2p v0.44.0
   Compiling libp2p-perf v0.1.0 (/home/kpp/parity/libp2p-perf/rust)
    Finished release [optimized + debuginfo] target(s) in 37.42s
kpp@xps:~/parity/libp2p-perf$ ./run.sh 
# Start Rust and Golang servers.

# Rust -> Rust

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.00 s	8201 MBytes	6560.56 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.00 s	14598 MBytes	11678.23 MBit/s

# Rust -> Golang

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.00 s	6832 MBytes	5465.37 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.00 s	12462 MBytes	9969.54 MBit/s

# Golang -> Rust

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.00 s 	9636 MBytes	7707.60 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.00 s 	20705 MBytes	16563.89 MBit/s

# Golang -> Golang

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.00 s 	5741 MBytes	4592.47 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.00 s 	21973 MBytes	17578.36 MBit/s

a realworld bench on 2 VMs

megamachines:

TCP plaintext:

Interval	Transfer	Bandwidth
0 s - 10.00 s	8397 MBytes	6716.62 MBit/s

Interval	Transfer	Bandwidth
0 s - 10.00 s	8418 MBytes	6734.40 MBit/s

Interval	Transfer	Bandwidth
0 s - 10.00 s	8414 MBytes	6731.19 MBit/s

TCP noise:

Interval	Transfer	Bandwidth
0 s - 10.00 s	6398 MBytes	5118.11 MBit/s

Interval	Transfer	Bandwidth
0 s - 10.00 s	6136 MBytes	4908.64 MBit/s

Interval	Transfer	Bandwidth
0 s - 10.00 s	6365 MBytes	5091.84 MBit/s

QUIC:

Interval	Transfer	Bandwidth
0 s - 10.02 s	709 MBytes	566.07 MBit/s

Interval	Transfer	Bandwidth
0 s - 10.01 s	728 MBytes	581.88 MBit/s

Interval	Transfer	Bandwidth
0 s - 10.01 s	701 MBytes	560.16 MBit/s

[elenaf9]

@kpp are those benchmarks for the now merged implementation (libp2p/rust-libp2p#2289) or for libp2p/rust-libp2p#2801?

[burdges]

Interesting, so quite a dramatic loss of performance for now.

[Ralith]

Much faster than the go stack on localhost, though.

How much latency is there between the two VMs, and what was the instantaneous throughput like at the end of the test? The discrepancy between localhost and "real world" is interesting, and hints at significant room for improvement in our congestion controller, which is known to have a few significant TODOs still (e.g. HyStart for Cubic). Which congestion controller did you use? Did you try the experimental BBR impl?

[burdges]

I've heard claims the congestion controller winds up being some application specific black magic that Google never explains, but maybe you guys understand better the underlying concerns there?

[Ralith]

I'm not sure what you mean. Quinn implements three different congestion controllers (New Reno, Cubic, and BBRv1), all of which are fairly well documented in the literature. The underlying logic is very similar to that used in the Linux kernel TCP implementation, but there's a few optimizations to Cubic that we haven't implemented yet.

[kpp]

@elenaf9 these are for libp2p/rust-libp2p#2801. I will re-do it for the master branch too.

[kpp]

libp2p/rust-libp2p#3454 is merged

[kpp]

QUIC is added to rust-libp2p: libp2p/rust-libp2p#2883 (comment)

[burdges]

I'd assume the benchmarks remain pretty lackluster?

[mxinden]

I'd assume the benchmarks remain pretty lackluster?

@burdges the latest libp2p-quic offers a ~10x throughput improvement compared to libp2p-tcp on a single connection single stream benchmark.

Validated via https://github.com/libp2p/test-plans/tree/master/perf

Expect more metrics to come. But at this point, I don't see a reason why to be pessimistic about libp2p-quic's performance characteristics.

[Ralith]

I believe they're thinking of the numbers at #536 (comment), which suggest overly pessimistic congestion controller behavior when there's significant latency, which tends to not be reflected by loopback tests. The details I asked for in #536 (comment) could help drive further investigation.

Of course, those older numbers were also worse than TCP, so maybe they're just obsolete.