feat: add WebRTC transport #12529

melekes · 2022-10-19T12:42:20Z

Refs paritytech/smoldot#1712
Impl libp2p/rust-libp2p#2622
polkadot companion: paritytech/polkadot#6529
cumulus companion: paritytech/cumulus#2076
Closes paritytech/polkadot-sdk#547

In dev mode the node will advertise and accept /webrtc connections;
The initial plan was to enable it for non-validators as well, but ultimately we've decided to postpone this until the majority of nodes & downstream users (parachains) upgrade to libp2p-identify 0.41.1 (see feat: add WebRTC transport #12529 (comment));
The transport will generate and store the certificate, which is required for WebRTC identity, in
base dir. In the future, when a new version of ring library is released, the certificate will be
deterministically derived from the node's peer ID.

client/cli/src/params/network_params.rs

client/network/src/transport.rs

by analogue with `id_keys`. See paritytech/substrate#12529 (comment) Both peer's identity and certificate will need to be stored somewhere (disk) and reused across restarts. Since this is a dominant use-case, I've switched `Config` API to accept a certificate and opted out of generating one completely.

client/network/src/config.rs

paritytech#12529 (comment)

Cargo.toml

tomaka

It's still unclear to me what to do w.r.t. the bandwidth
Everything would be way easier if webrtc-rs simply kept track of how many bytes have been sent and received in total, which would let us calculate the bandwidth, it really doesn't seem like a difficult change to me

client/network/src/config.rs

client/network/src/service.rs

client/network/src/discovery.rs

client/cli/src/params/webrtc_certificate_params.rs

melekes · 2022-11-22T04:23:06Z

sorry had to force push to do #12529 (comment)

paritytech#12529 (review)

tomaka · 2022-11-22T08:03:03Z

sorry had to force push to do #12529 (comment)

It doesn't matter now, but you didn't have to do that. We squash commits anyway, so whether you're based on top of another branch doesn't matter. The point was just to merge this in two times: first the libp2p changes, then the WebRTC changes. This way, we can easily bisect in case a bug is introduced.

melekes · 2022-11-22T08:28:20Z

so whether you're based on top of another branch doesn't matter.

well it matters for me because the amount of conflicts reduces dramatically when you rebase 1 commit vs 22 😄

seunlanlege · 2023-07-21T00:56:43Z

Bump

Polkadot-Forum · 2023-07-25T14:42:58Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/infrastructure-treasury-funding/3507/3

paritytech-cicd-pr · 2023-07-25T15:00:53Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3257797

melekes · 2023-07-26T09:04:29Z

zombienet test failed again, but with different error this time:

2023-07-25 16:48:35.169  INFO tokio-runtime-worker libp2p_kad::handler: Remote 12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm no longer supports our kademlia protocol

2023-07-25 16:48:35.169 TRACE tokio-runtime-worker sub-libp2p: Ignoring self-reported address /ip4/10.24.122.238/udp/30333/webrtc-direct/certhash/uEiAvF2IuPZJuKkdb5E-gSM_AmumLi3P5ZR_fvZGPE3vJEA from 12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm as remote node is not part of the Kademlia DHT supported by the local node.

2023-07-25 16:48:37.548 TRACE tokio-runtime-worker sub-libp2p: Libp2p => Query for 0020cc78cfbc884fd5668ebb8f2948752f2dbc3a4c665b1b678f6190ec7208cbff72 yielded 0 results    
2023-07-25 16:48:37.548 DEBUG tokio-runtime-worker sub-libp2p: Libp2p => Random Kademlia query has yielded empty results

not yet sure what is happening, but it looks like Kademlia does not provide dave with addresses to connect to...

``` [2023-07-26 09:13:50] error: failed to select a version for `regex`. [2023-07-26 09:13:50] ... required by package `webrtc v0.8.0` [2023-07-26 09:13:50] ... which satisfies dependency `webrtc = "^0.8.0"` of package `libp2p-webrtc v0.6.0-alpha` [2023-07-26 09:13:50] ... which satisfies dependency `libp2p-webrtc = "^0.6.0-alpha"` of package `sc-network v0.10.0-dev (/builds/parity/mirrors/substrate/client/network)` [2023-07-26 09:13:50] ... which satisfies git dependency `sc-network` (locked to 0.10.0-dev) of package `polkadot-availability-distribution v0.9.43 (/builds/parity/mirrors/substrate/companions/polkadot/node/network/availability-distribution)` [2023-07-26 09:13:50] ... which satisfies path dependency `polkadot-availability-distribution` (locked to 0.9.43) of package `polkadot-service v0.9.43 (/builds/parity/mirrors/substrate/companions/polkadot/node/service)` [2023-07-26 09:13:50] ... which satisfies path dependency `service` (locked to 0.9.43) of package `polkadot-cli v0.9.43 (/builds/parity/mirrors/substrate/companions/polkadot/cli)` [2023-07-26 09:13:50] ... which satisfies path dependency `polkadot-cli` (locked to 0.9.43) of package `polkadot v0.9.43 (/builds/parity/mirrors/substrate/companions/polkadot)` [2023-07-26 09:13:50] versions that meet the requirements `^1.7.1` are: 1.9.1, 1.9.0, 1.8.4, 1.8.3, 1.8.2, 1.8.1, 1.8.0, 1.7.3, 1.7.2, 1.7.1 [2023-07-26 09:13:50] [2023-07-26 09:13:50] all possible versions conflict with previously selected packages. [2023-07-26 09:13:50] [2023-07-26 09:13:50] previously selected package `regex v1.6.0` [2023-07-26 09:13:50] ... which satisfies dependency `regex = "^1.1"` (locked to 1.6.0) of package `Inflector v0.11.4` [2023-07-26 09:13:50] ... which satisfies dependency `Inflector = "^0.11.4"` (locked to 0.11.4) of package `frame-benchmarking-cli v4.0.0-dev (/builds/parity/mirrors/substrate/utils/frame/benchmarking-cli)` [2023-07-26 09:13:50] ... which satisfies git dependency `frame-benchmarking-cli` (locked to 4.0.0-dev) of package `polkadot-cli v0.9.43 (/builds/parity/mirrors/substrate/companions/polkadot/cli)` [2023-07-26 09:13:50] ... which satisfies path dependency `polkadot-cli` (locked to 0.9.43) of package `polkadot v0.9.43 (/builds/parity/mirrors/substrate/companions/polkadot)` [2023-07-26 09:13:50] [2023-07-26 09:13:50] failed to select a version for `regex` which could resolve this conflict ```

mxinden · 2023-07-26T11:59:43Z

zombienet test failed again, but with different error this time:

2023-07-25 16:48:35.169  INFO tokio-runtime-worker libp2p_kad::handler: Remote 12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm no longer supports our kademlia protocol

2023-07-25 16:48:35.169 TRACE tokio-runtime-worker sub-libp2p: Ignoring self-reported address /ip4/10.24.122.238/udp/30333/webrtc-direct/certhash/uEiAvF2IuPZJuKkdb5E-gSM_AmumLi3P5ZR_fvZGPE3vJEA from 12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm as remote node is not part of the Kademlia DHT supported by the local node.

2023-07-25 16:48:37.548 TRACE tokio-runtime-worker sub-libp2p: Libp2p => Query for 0020cc78cfbc884fd5668ebb8f2948752f2dbc3a4c665b1b678f6190ec7208cbff72 yielded 0 results    
2023-07-25 16:48:37.548 DEBUG tokio-runtime-worker sub-libp2p: Libp2p => Random Kademlia query has yielded empty results

not yet sure what is happening, but it looks like Kademlia does not provide dave with addresses to connect to...

Just an idea: are you either calling Kademlia::set_mode(Mode::Server) or are you providing an external address @melekes?

melekes · 2023-07-26T12:16:27Z

Just an idea: are you either calling Kademlia::set_mode(Mode::Server) or are you providing an external address @melekes?

@mxinden neither? I was hoping that the node will discover it's external address via bootnode and switch to server mode automatically.

refs paritytech#12529

This reverts commit 142b884.

This reverts commit 7d4e8e2.

melekes · 2023-08-02T08:45:47Z

2023-07-31 09:04:31 Substream 10786 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10788 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10790 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10792 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10794 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10796 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10798 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10800 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10802 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10804 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10806 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10808 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10810 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10812 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10814 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10816 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10818 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10820 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10822 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10824 dropped without graceful close, sending Reset
2023-07-31 09:04:31 Substream 10826 dropped without graceful close, sending Reset

Not sure why Charlie needs 10000 substreams unless there's a bug somewhere.

Alice & Bob writing to non-existing streams:

2023-07-31 09:05:15.895 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.895 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.896 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)
2023-07-31 09:05:15.897 ERROR tokio-runtime-worker webrtc_sctp::association::association_internal: [] stream 104 not found)

Dave fails to open ~2k substreams due to gone receiver:

rg "Can't send data channel 0: send failed because receiver is gone" | wc -l
    2387

altonen · 2023-08-02T10:11:28Z

@melekes are these problems reproducible with just libp2p or is Substrate somehow causing these issues?

melekes added A3-in_progress Pull request is in progress. No review needed at this stage. C3-medium PR touches the given topic and has a medium impact on builders. labels Oct 19, 2022

melekes self-assigned this Oct 19, 2022

tomaka reviewed Oct 19, 2022

View reviewed changes

client/cli/src/params/network_params.rs Outdated Show resolved Hide resolved

client/cli/src/params/network_params.rs Outdated Show resolved Hide resolved

tomaka reviewed Oct 19, 2022

View reviewed changes

client/network/src/transport.rs Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

tomaka added B5-clientnoteworthy and removed C3-medium PR touches the given topic and has a medium impact on builders. labels Oct 19, 2022

melekes force-pushed the anton/webrtc-transport branch from 391f122 to ca39f65 Compare November 4, 2022 07:46

tomaka reviewed Nov 17, 2022

View reviewed changes

client/network/src/config.rs Outdated Show resolved Hide resolved

melekes added a commit to melekes/substrate that referenced this pull request Nov 17, 2022

rename enum option

3f417fe

paritytech#12529 (comment)

melekes commented Nov 17, 2022

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

tomaka suggested changes Nov 17, 2022

View reviewed changes

client/network/src/config.rs Outdated Show resolved Hide resolved

client/network/src/config.rs Outdated Show resolved Hide resolved

client/network/src/service.rs Outdated Show resolved Hide resolved

client/network/src/discovery.rs Outdated Show resolved Hide resolved

tomaka reviewed Nov 18, 2022

View reviewed changes

client/cli/src/params/webrtc_certificate_params.rs Outdated Show resolved Hide resolved

tomaka reviewed Nov 18, 2022

View reviewed changes

client/cli/src/params/webrtc_certificate_params.rs Outdated Show resolved Hide resolved

upgrade libp2p to 0.50.0

81bdbdf

melekes mentioned this pull request Nov 18, 2022

upgrade libp2p to 0.50.0 #12734

Merged

melekes added 6 commits November 21, 2022 11:25

on_swarm_event and on_connection_handler_event

21cf90c

replace Swarm::new with Swarm::with_threadpool_executor

8a227e4

on_swarm_event and on_connection_handler_event part 2

8a97a52

on_swarm_event and on_connection_handler_event part 3

e9b731b

on_swarm_event and on_connection_handler_event part 4

d27d73a

update libp2p

672cac2

melekes force-pushed the anton/webrtc-transport branch from 33c615a to c4b3dcc Compare November 22, 2022 04:22

melekes force-pushed the anton/webrtc-transport branch from c4b3dcc to 43ed99e Compare November 22, 2022 04:31

melekes added a commit to melekes/substrate that referenced this pull request Nov 22, 2022

put webrtc certificate within base, not network config directory

0f49005

paritytech#12529 (review)

stale bot removed the A3-stale label Jul 21, 2023

Merge branch 'master' into anton/webrtc-transport

baf5b8f

fixes after merge

424e319

melekes requested review from a team July 25, 2023 14:51

fix protocol name

e24baa9

melekes changed the title ~~[client/network] Add WebRTC transport~~ feat: add WebRTC transport Jul 25, 2023

melekes added 3 commits July 25, 2023 19:16

use libp2p-webrtc dep in CLI

6025ea4

Merge branch 'master' into anton/webrtc-transport

09efa75

remove webrtc_sctp trace from dave's log

f0f7075

melekes added 2 commits July 26, 2023 13:30

more logging

01ba6d2

melekes requested a review from koute as a code owner July 26, 2023 11:26

melekes added 7 commits July 26, 2023 16:23

set mode to server by default

7d4e8e2

refs paritytech#12529

Revert "bump regex to resolve polkadot companion CI job"

ff958f8

This reverts commit 142b884.

Revert "set mode to server by default"

27ef6c9

This reverts commit 7d4e8e2.

Merge branch 'master' into anton/webrtc-transport

634ed48

reduce logging

2647ae6

Merge branch 'master' into anton/webrtc-transport

fb2caf8

Merge branch 'master' into anton/webrtc-transport

c650ae8

Merge branch 'master' into anton/webrtc-transport

20d33ad

more logging for dave

7d8623a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add WebRTC transport #12529

feat: add WebRTC transport #12529

melekes commented Oct 19, 2022 •

edited

This comment was marked as resolved.

tomaka left a comment

melekes commented Nov 22, 2022

tomaka commented Nov 22, 2022 •

edited

melekes commented Nov 22, 2022

seunlanlege commented Jul 21, 2023

Polkadot-Forum commented Jul 25, 2023

paritytech-cicd-pr commented Jul 25, 2023

melekes commented Jul 26, 2023

mxinden commented Jul 26, 2023

melekes commented Jul 26, 2023

melekes commented Aug 2, 2023 •

edited

altonen commented Aug 2, 2023

feat: add WebRTC transport #12529

Are you sure you want to change the base?

feat: add WebRTC transport #12529

Conversation

melekes commented Oct 19, 2022 • edited

This comment was marked as resolved.

tomaka left a comment

Choose a reason for hiding this comment

melekes commented Nov 22, 2022

tomaka commented Nov 22, 2022 • edited

melekes commented Nov 22, 2022

seunlanlege commented Jul 21, 2023

Polkadot-Forum commented Jul 25, 2023

paritytech-cicd-pr commented Jul 25, 2023

melekes commented Jul 26, 2023

mxinden commented Jul 26, 2023

melekes commented Jul 26, 2023

melekes commented Aug 2, 2023 • edited

altonen commented Aug 2, 2023

melekes commented Oct 19, 2022 •

edited

tomaka commented Nov 22, 2022 •

edited

melekes commented Aug 2, 2023 •

edited