transports/quic: Add implementation based on `quinn-proto` #2289

kpp · 2021-10-14T12:05:12Z

Inital PR description

A continuation of https://github.com//pull/2159.

Adds support for tls, removes the UnboundedSenders from the endpoint. The endpoint still has UnboundedReceivers, but that's not different than having a VecDeque. The transport dial api for example doesn't allow applying backpressure.

Closes paritytech/substrate#1334

Todo: transfer unresolved comments from the previous PR

Updated PR description:

Description

Add QUIC as transport to rust-libp2p, using the quinn-proto QUIC implementation.
Initially a continuation of paritytech/substrate#2159, later switched to use the code from paritytech/substrate#1334 (see 4a317df).

Links to any relevant issues

Turn storage items into parameters paritytech/substrate#2883

Open TODOs

Fix docs
Review / document how backpressure is enforced
Add more tests

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

Commit message body

Add experimental QUIC implementation based on quinn-proto

Co-authored-by: Demi Marie Obenour demiobenour@gmail.com
Co-authored-by: Pierre Krieger pierre.krieger1708@gmail.com
Co-authored-by: Thomas Eizinger thomas@eizinger.io
Co-authored-by: Max Inden mail@max-inden.de
Co-authored-by: Marco Munizaga marco@marcopolo.io
Co-authored-by: Elena Frank elena.frank@protonmail.com

* protocols/gossipsub: Fix inconsistency in mesh peer tracking (libp2p#2189) Co-authored-by: Age Manning <Age@AgeManning.com> * misc/metrics: Add auxiliary crate to record events as OpenMetrics (libp2p#2063) This commit adds an auxiliary crate recording protocol and Swarm events and exposing them as metrics in the OpenMetrics format. * README: Mention security@ipfs.io * examples/: Add file sharing example (libp2p#2186) Basic file sharing application with peers either providing or locating and getting files by name. While obviously showcasing how to build a basic file sharing application, the actual goal of this example is **to show how to integrate rust-libp2p into a larger application**. Architectural properties - Clean clonable async/await interface ([`Client`]) to interact with the network layer. - Single task driving the network layer, no locks required. * examples/README: Give an overview over the many examples (libp2p#2194) * protocols/kad: Enable filtering of (provider) records (libp2p#2163) Introduce `KademliaStoreInserts` option, which allows to filter records. Co-authored-by: Max Inden <mail@max-inden.de> * swarm/src/protocols_handler: Impl ProtocolsHandler on either::Either (libp2p#2192) Implement ProtocolsHandler on either::Either representing either of two ProtocolsHandler implementations. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * *: Make libp2p-core default features optional (libp2p#2181) Co-authored-by: Max Inden <mail@max-inden.de> * core/: Remove DisconnectedPeer::set_connected and Pool::add (libp2p#2195) This logic seems to be a leftover of libp2p#889 and unused today. * protocols/gossipsub: Use ed25519 in tests (libp2p#2197) With f2905c0 the secp256k1 feature is disabled by default. Instead of enabling it in the dev-dependency, simply use ed25519. * build(deps): Update minicbor requirement from 0.10 to 0.11 (libp2p#2200) Updates the requirements on [minicbor](https://gitlab.com/twittner/minicbor) to permit the latest version. - [Release notes](https://gitlab.com/twittner/minicbor/tags) - [Changelog](https://gitlab.com/twittner/minicbor/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/twittner/minicbor/compare/minicbor-v0.10.0...minicbor-v0.11.0) --- updated-dependencies: - dependency-name: minicbor dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Update salsa20 requirement from 0.8 to 0.9 (libp2p#2206) * build(deps): Update salsa20 requirement from 0.8 to 0.9 Updates the requirements on [salsa20](https://github.com/RustCrypto/stream-ciphers) to permit the latest version. - [Release notes](https://github.com/RustCrypto/stream-ciphers/releases) - [Commits](RustCrypto/stream-ciphers@ctr-v0.8.0...salsa20-v0.9.0) --- updated-dependencies: - dependency-name: salsa20 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * *: Bump pnet to v0.22 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> * *: Dial with handler and return handler on error and closed (libp2p#2191) Require `NetworkBehaviourAction::{DialPeer,DialAddress}` to contain a `ProtocolsHandler`. This allows a behaviour to attach custom state to its handler. The behaviour would no longer need to track this state separately during connection establishment, thus reducing state required in a behaviour. E.g. in the case of `libp2p-kad` the behaviour can include a `GetRecord` request in its handler, or e.g. in the case of `libp2p-request-response` the behaviour can include the first request in the handler. Return `ProtocolsHandler` on connection error and close. This allows a behaviour to extract its custom state previously included in the handler on connection failure and connection closing. E.g. in the case of `libp2p-kad` the behaviour could extract the attached `GetRecord` from the handler of the failed connection and then start another connection attempt with a new handler with the same `GetRecord` or bubble up an error to the user. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * core/: Remove deprecated read/write functions (libp2p#2213) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/ping: Revise naming of symbols (libp2p#2215) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/rendezvous: Implement protocol (libp2p#2107) Implement the libp2p rendezvous protocol. > A lightweight mechanism for generalized peer discovery. It can be used for bootstrap purposes, real time peer discovery, application specific routing, and so on. Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> * core/src/network/event.rs: Fix typo (libp2p#2218) * protocols/mdns: Do not fire all timers at the same time. (libp2p#2212) Co-authored-by: Max Inden <mail@max-inden.de> * misc/metrics/src/kad: Set query_duration lowest bucket to 0.1 sec (libp2p#2219) Probability for a Kademlia query to return in less than 100 milliseconds is low, thus increasing the lower bucket to improve accuracy within the higher ranges. * misc/metrics/src/swarm: Expose role on connections_closed (libp2p#2220) Expose whether closed connection was a Dialer or Listener. * .github/workflows/ci.yml: Use clang 11 (libp2p#2233) * protocols/rendezvous: Update prost (libp2p#2226) Co-authored-by: Max Inden <mail@max-inden.de> * *: Fix clippy warnings (libp2p#2227) * swarm-derive/: Make event_process = false the default (libp2p#2214) Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: Ruben De Smet <ruben.de.smet@rubdos.be> Co-authored-by: Thomas Eizinger <thomas@eizinger.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> Co-authored-by: David Craven <david@craven.ch>

Co-authored-by: Max Inden <mail@max-inden.de>

mxinden · 2021-10-14T12:14:19Z

Thanks @kpp for driving this forward. I will take a look.

kpp

Linked discussions from the previous PR

transports/quic/src/endpoint.rs

transports/quic/src/crypto.rs

transports/quic/src/endpoint.rs

transports/quic/src/muxer.rs

transports/quic/src/tls/verifier.rs

transports/quic/tests/smoke.rs

mxinden

Short update: I am currently testing this pull request with https://github.com/mxinden/libp2p-perf. Still need a bit more time before I can share any results.

Small comment below.

transports/quic/Cargo.toml

transports/quic/src/transport.rs

transports/quic/src/endpoint.rs

transports/quic/src/tls/certificate.rs

transports/quic/src/tls/verifier.rs

transports/quic/tests/smoke.rs

dvc94ch

What's the point of this? @marten-seemann is not a contributor or even user of rust-libp2p, so don't know why his opinion is relevant here.

Test will be removed completely from the test harness in libp2p#3108.

Make `IfWatcher` a `Provider` item. Use `if_watch::smol::IfWatcher` in `async-std::Provider` and `if_watch::tokio::IfWatcher` in `tokio::Provider`.

Depending on what the remote sent in the initial packet, a new inbound connection may directly be marked as closed.

Block on sending `pending_to_endpoint` before polling `from_endpoint`. If the endpoint is blocked on writing to the udp socket it will backpressure the connection, which will in return then stop polling for events from the endpoint. The endpoint will drop new inbound packets if the channel to the connection is full, thus eventually backpressuring remote peers.

elenaf9 · 2022-11-12T18:12:32Z

Only missing pieces from my end:
...

Co-author lines on squash merge commit

I added co-author lines to the "# Commit message body" section of this PR.
If I understand it correctly this will add them to the squash merge commit @thomaseizinger? I might still merge manually though, but in that case those would be the co-authors that I include. Since @kpp is the author of the PR I think I don't have to add an extra line for him, right?
Let me know in case I missed anybody!

thomaseizinger · 2022-11-12T23:36:56Z

Only missing pieces from my end:
...

Co-author lines on squash merge commit

I added co-author lines to the "# Commit message body" section of this PR.
If I understand it correctly this will add them to the squash merge commit @thomaseizinger?

Correct. See #3106 for a recent example and c32f03c for the resulting commit!

mergify · 2022-11-13T00:00:41Z

This pull request has merge conflicts. Could you please resolve them @kpp? 🙏

mxinden · 2022-11-13T10:58:45Z

For the record multiformats/rust-multiaddr#64 has been released with multiaddr v0.16.0. See multiformats/rust-multiaddr#62.

mxinden · 2022-11-13T19:01:56Z

Only missing pieces from my end:
...

Co-author lines on squash merge commit

I added co-author lines to the "# Commit message body" section of this PR.
If I understand it correctly this will add them to the squash merge commit @thomaseizinger?

Correct. See paritytech/substrate#3106 for a recent example and c32f03c for the resulting commit!

Note that this is no longer the case with #3082 merged.

thomaseizinger · 2022-11-13T21:48:41Z

Only missing pieces from my end:
...

Co-author lines on squash merge commit

I added co-author lines to the "# Commit message body" section of this PR.
If I understand it correctly this will add them to the squash merge commit @thomaseizinger?

Correct. See paritytech/substrate#3106 for a recent example and c32f03c for the resulting commit!

Note that this is no longer the case with #3082 merged.

Yes, needs to be at the end of ## Description now.

With multiformats/multiaddr#145 the `quic` codepoint should be interpreted as QUIC draft-29. For QUIC v1 (RFC9000) the new codepoint `quic-v1` should be used. Quinn supports both, draft-29 and v1 as server, for clients however the version has to be set when dialing. Right now we use the default, v1. Proper support for `Protocol::QUIC` / draft-29 will be added in a follow-up PR.

elenaf9 · 2022-11-13T23:18:59Z

transports/quic/src/transport.rs

+
+    matches!(first, Ip4(_) | Ip6(_) | Dns(_) | Dns4(_) | Dns6(_))
+        && matches!(second, Udp(_))
+        && matches!(third, QuicV1)


Right now we only support the new quic-v1 codepoint. Support for quic/ draft-29 will be added in a follow-up PR. See multiformats/multiaddr#145 (comment) for more info. cc @marten-seemann

elenaf9 · 2022-11-13T23:22:48Z

Unless there are any objections I will merge this PR tomorrow 🎉

elenaf9 · 2022-11-14T16:57:58Z

🎉 We finally have a QUIC transport.
Huge thanks to @kpp, @DemiMarie, @thomaseizinger, and everyone else that contributed to this!

DemiMarie · 2022-11-14T21:40:08Z

Glad that I could help @elenaf9!!!

- Implementation based on `quinn-proto` was added in libp2p#2289 - Add new entry to evaluate using `quinn` directly

- Implementation based on `quinn-proto` was added in #2289 - Add new entry to evaluate using `quinn` directly

kpp and others added 10 commits September 8, 2021 13:40

Use std HashMap instead of fnv's

f760494

Remove ToLibp2p to_keypair

9a88cf2

Move iovs init into a separate function

35ac0fb

Use mem::replace to unbind recv_buf

d665212

Update transports/quic/src/endpoint.rs

ee7a7f5

Merge branch 'master' into libp2p-quic

314e1c5

Update transports/quic/src/transport.rs

9b36bc1

Co-authored-by: Max Inden <mail@max-inden.de>

Add David's credentials to quic/src headers

7c33cdc

Merge branch 'master' into libp2p-quic

da60d1f

kpp mentioned this pull request Aug 24, 2023

Tracking issue for QUIC support paritytech/polkadot-sdk#536

Open

10 tasks

kpp commented Oct 14, 2021

View reviewed changes

Merge branch 'master' into libp2p-quic

4e94add

kpp marked this pull request as ready for review October 14, 2021 14:38

This was referenced Oct 15, 2021

*: Update to libp2p v0.40.0-rc.1 mxinden/libp2p-perf#50

Merged

rust/: Add support for QUIC mxinden/libp2p-perf#51

Open

mxinden reviewed Oct 15, 2021

View reviewed changes

transports/quic/Cargo.toml Outdated Show resolved Hide resolved

mxinden reviewed Oct 19, 2021

View reviewed changes

transports/quic/src/transport.rs Outdated Show resolved Hide resolved

transports/quic/src/endpoint.rs Outdated Show resolved Hide resolved

marten-seemann reviewed Oct 21, 2021

View reviewed changes

transports/quic/src/tls/certificate.rs Outdated Show resolved Hide resolved

transports/quic/src/tls/verifier.rs Outdated Show resolved Hide resolved

transports/quic/tests/smoke.rs Outdated Show resolved Hide resolved

kpp added 9 commits October 21, 2021 15:05

Merge branch 'master' into libp2p-quic

d889fb1

Mirror quic features in the root Cargo.toml

af73d98

Fix tests after merging master

1e0078c

Fix TLS 1.2 error msg

b30ae08

Expect substreams when accessing by id in the inner struct

9f11c30

Comment out _max_datagrams

9425cc5

Cut off noise crypto

25ee1fe

Remove generic crypto code

f35fde1

Merge branch 'master' into libp2p-quic

8844e7c

dvc94ch reviewed Nov 9, 2021

View reviewed changes

elenaf9 added 8 commits November 12, 2022 17:21

quic: clean dependencies

20c3fce

quic: make clippy happy

bea79a8

quic/tests: remove dialer_can_receive test

0ab672a

Test will be removed completely from the test harness in libp2p#3108.

quic: Update to if-watchv3.0.0

4b042b2

Make `IfWatcher` a `Provider` item. Use `if_watch::smol::IfWatcher` in `async-std::Provider` and `if_watch::tokio::IfWatcher` in `tokio::Provider`.

quic: rm debug_assert(!is_closed) on new connection

7770943

Depending on what the remote sent in the initial packet, a new inbound connection may directly be marked as closed.

quic: Add initial release entry to CHANGELOG

56e39b8

Merge branch 'master' of github.com:libp2p/rust-libp2p into libp2p-quic

1b5741e

quic: Add entry to libp2p CHANGELOG

cd7f38f

elenaf9 mentioned this pull request Nov 13, 2022

deps: Update to multiaddr v0.16.0 #3117

Merged

elenaf9 added 2 commits November 13, 2022 23:42

Merge branch 'master' of github.com:libp2p/rust-libp2p into libp2p-quic

588b76e

elenaf9 reviewed Nov 13, 2022

View reviewed changes

Merge branch 'master' into libp2p-quic

5ac7e54

elenaf9 merged commit 0f5c491 into libp2p:master Nov 14, 2022

kpp deleted the libp2p-quic branch November 14, 2022 21:54

mxinden added a commit to mxinden/rust-libp2p that referenced this pull request Jan 20, 2023

docs(roadmap): Mark alpha QUIC as done and add quinn work

771fed5

- Implementation based on `quinn-proto` was added in libp2p#2289 - Add new entry to evaluate using `quinn` directly

mxinden mentioned this pull request Jan 20, 2023

docs(roadmap): Mark alpha QUIC as done and add quinn work #3356

Merged

mergify bot pushed a commit that referenced this pull request Jan 23, 2023

docs(roadmap): Mark alpha QUIC as done and add quinn work (#3356)

8d6a2fc

- Implementation based on `quinn-proto` was added in #2289 - Add new entry to evaluate using `quinn` directly

goodboy mentioned this pull request Mar 24, 2023

QUIC transport research & planning goodboy/tractor#353

Open

mxinden mentioned this pull request Nov 20, 2023

fix(quic): fix address translation #4896

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

transports/quic: Add implementation based on `quinn-proto` #2289

transports/quic: Add implementation based on `quinn-proto` #2289

kpp commented Oct 14, 2021 •

edited by elenaf9

mxinden commented Oct 14, 2021

kpp left a comment

mxinden left a comment

dvc94ch left a comment

elenaf9 commented Nov 12, 2022 •

edited

thomaseizinger commented Nov 12, 2022 •

edited

mergify bot commented Nov 13, 2022

mxinden commented Nov 13, 2022

mxinden commented Nov 13, 2022

thomaseizinger commented Nov 13, 2022

elenaf9 Nov 13, 2022

elenaf9 commented Nov 13, 2022

elenaf9 commented Nov 14, 2022

DemiMarie commented Nov 14, 2022

transports/quic: Add implementation based on quinn-proto #2289

transports/quic: Add implementation based on quinn-proto #2289

Conversation

kpp commented Oct 14, 2021 • edited by elenaf9

Description

Links to any relevant issues

Open TODOs

Change checklist

Commit message body

mxinden commented Oct 14, 2021

kpp left a comment

Choose a reason for hiding this comment

mxinden left a comment

Choose a reason for hiding this comment

dvc94ch left a comment

Choose a reason for hiding this comment

elenaf9 commented Nov 12, 2022 • edited

thomaseizinger commented Nov 12, 2022 • edited

mergify bot commented Nov 13, 2022

mxinden commented Nov 13, 2022

mxinden commented Nov 13, 2022

thomaseizinger commented Nov 13, 2022

elenaf9 Nov 13, 2022

Choose a reason for hiding this comment

elenaf9 commented Nov 13, 2022

elenaf9 commented Nov 14, 2022

DemiMarie commented Nov 14, 2022

transports/quic: Add implementation based on `quinn-proto` #2289

transports/quic: Add implementation based on `quinn-proto` #2289

kpp commented Oct 14, 2021 •

edited by elenaf9

elenaf9 commented Nov 12, 2022 •

edited

thomaseizinger commented Nov 12, 2022 •

edited