p2p-circuit v2 #125

vyzo · 2021-02-04T23:06:31Z

v2 of the protocol and Limited Relays.

Depends on:

TBD:

An Overview of the p2p-circuit/v2 Protocol

The relay protocol has been split into two protcols: the Hop protocol and the Stop protocol.

the Hop protocol governs communication with the relay for reservations and circuit connection initiation.
the Stop protocol governs termination of circuit connections.

Establishing a relayed connection

Let's suppose that a peer A wants to connect to peer B through relay peer R.

Reservations

Peer B must first establish a reservation by connecting (and maintaining a connection) to R.

This is done using the Hop protocol and sending a Reserve message. The relay responds with OK if the reservation is accepted and has a specific TTL, after which the reservation will dissipate.

Peer B is responsible for refreshing its reservation when it reaches the end of the TTL.
In the relay logic, reservations are governed by strict resource limits and ACLs.

Connection Establishment

Peer A connects to Relay R, opens a Hop protocol stream and sends a CONNECT message.

Relay R must have a reservation and connection to peer B. Active relay has been deprecated in the code, but the protocol allows for implementation without any change in the wire protocol.

Once R receives the CONNECT message, it opens a Stop protocol stream to B and sends a CONNECT message.
Peer B responds with OK; the relay R responds with OK to peer A and bridges the two streams in order to establish the relayed connection.

Connection Limits

Relays are allowed (and encouraged! the relay implementation sets strict limits by default) to limit the relayed connection, by time and bytes relayed in each direction.
The limits are conveyed to both sides of a relayed connection; relayed connections are marked as transient by the transport.

Implementation Details

Implementation lives under v2.

The Stop protocol implementation lives in v2/client and provides a Transport suitable for attachment to a host. The implementation mounts the v1 protocol and provides backwards compatibility in dialing v1 peers and accepting relayed connections from v1 peers through v1 relays.

The Hop protocol implementation lives in v2/relay and provides a Relay object that can be instantiated on deman. When instantiated, the Relay registers a stream handler from the v2 hop protocol.

The two are completely independent. The client package implements the transport and is the point of integration with go-libp2p. The relay package is the actual relay and can be used in dedicated relays and (as planned for Project Flare) public DHT servers. The default resource limits are very much constrained which makes it safe for public nodes.

vyzo · 2021-02-05T15:41:43Z

Summoning @Stebalien @raulk @yusefnapora @aarshkshah1992 -- this is ready for (initial) review!

vyzo · 2021-02-12T17:38:01Z

Added tests and godocs.

aarshkshah1992

@vyzo

Did a first pass on the Relay code. Will move to the client now.

Awesome stuff man !

v2/pb/circuit.proto

v2/util/io.go

aarshkshah1992 · 2021-02-15T04:40:34Z

v2/util/io.go

@@ -0,0 +1,67 @@
+package util
+


Need to write unit tests for this file.

meh.... if I have nothing better to do.

v2/relay/acl.go

v2/relay/relay.go

v2/client/client.go

aarshkshah1992

@vyzo Finished first pass of client too.

Will give all this one more in-depth look tomorrow. But looking amazing so far.

v2/client/conn.go

v2/relay/relay.go

aarshkshah1992 · 2021-02-15T08:48:18Z

v2/client/transport.go

+}
+
+func (c *Client) Listen(addr ma.Multiaddr) (transport.Listener, error) {
+	// TODO connect to the relay and reserve slot if specified


This should happen in AutoRelay, right ?

that's a different topic -- it is for explicit relay address listening, when/if we ever support it.

v2/client/reservation.go

aarshkshah1992 · 2021-02-15T08:56:10Z

v2/client/conn.go

+}
+
+// TODO: is it okay to cast c.Conn().RemotePeer() into a multiaddr? might be "user input"
+func (c *Conn) RemoteMultiaddr() ma.Multiaddr {


Please can we add a unit test for this one function ?

ugh.... maybe, I am not inclined to do it right now.

v2/client/dial.go

v2/client/handlers.go

v2/client/dial.go

aarshkshah1992

Some more changes.

Need to do one last round of an intense review including all tests and I'll be done.

v2/relay/acl.go

v2/relay/resources.go

v2/util/io.go

v2/util/pbconv.go

aarshkshah1992

Finished last round of review.

Will take a look at the tests tomorrow but looks great otherwise.

aarshkshah1992 · 2021-02-16T13:47:58Z

v2/relay/relay.go

+		}
+	}
+
+	if r.rc.Limit != nil {


We should definitely add reference counting for all go-routines we are spawning here and ensure they all finish before we return from Close to ensure proper cleanup for the caller.

that's just not worth it and also the goroutines may never finish because it's an unlimited relay.
So, no.

I am not sure this is the right approach to take. We can easily stop relaying on unlimited relays by closing connections to all peers we have opened circuits for.

When Close returns, the caller expects that the subsystem has cleaned-up after it itself. Not doing so can leak go-routines if the caller dosen't know this implementation detail.

cc @Stebalien Isn't this in-line with what Close does for other sub-systems in libp2p.

If you insist on closing the connections it is much easier to tag the Hop/Stop streams and go and reset them; there is no tracking needed and it is much easier to implement (and doesn't suffer from the problem of unlimited relay conns which will never exit).

Waiting for the goroutines is irrelevant.

Actually, we don't even have to tag streams, which could be a little problematic with the current APIs; we can just look for incoming Hop streams and outgoing Stop streams and reset them.

v2/relay/relay.go

v2/client/dial.go

aarshkshah1992 · 2021-02-17T12:48:48Z

v2/relay/ipcs.go

+var (
+	ErrNoIP              = errors.New("no IP address associated with peer")
+	ErrTooManyPeersInIP  = errors.New("too many peers in IP address")
+	ErrTooManyPeersInASN = errors.New("too many peers in ASN")


we probably don't need to export any of these

Yeah, the only reason I exported them is so that the test could could live together with the other tests in the test directory.
It felt odd to have just one test in here.

v2/client/reservation.go

mxinden · 2021-03-04T13:22:20Z

v2/pb/circuit.proto

+
+enum Status {
+  OK                      = 100;
+  RESERVATION_REFUSED     = 200;


How about splitting Status into a HopMessage specific Status and a StopMessage specific Status like done with Type? The split would enforce protocol semantics at the datastructure level. E.g. it encodes the semantic that one could not receive a RESERVATION_REFUSED when sending a StopMessage.

We could I guess, but there would be a lot of duplication; I don't feel strong about it nonetheless.

Stebalien · 2021-03-19T00:22:41Z

Relay R must have a reservation and connection to peer B. Active relay has been deprecated in the code, but the protocol allows for implementation without any change in the wire protocol.

Can we include a flag in the reservation response that specifies whether or not the relay supports active relays? That way the peer can disconnect if it sees that active relay is supported.

We can also just punt this and do it later as it should be backwards compatible.

Stebalien

Attacker connects.
Attacker reserves.
Attacker uses entire budget.
Attacker disconnects.
Attacker re-rolls their peer ID and starts at 1.

I think we need:

A bit of local history (i.e., a per-peer/ip range cooldown period).
A global rate-limit.
Ideally, some form of "uses less than X bits/s" that can prevent large spikes.

v2/proto/voucher.go

Stebalien · 2021-03-19T00:36:15Z

v2/relay/ipcs.go

+	asn, _ := asnutil.Store.AsnForIPv6(ip)
+	peersInAsn = ipcs.asns[asn]
+	if len(peersInAsn) >= ipcs.asnlimit {
+		return ErrTooManyPeersInASN


👍 (but I can still just repeatedly reconnect)

Stebalien · 2021-03-19T00:38:59Z

v2/relay/relay.go

+	buf := pool.Get(r.rc.BufferSize)
+	defer pool.Put(buf)
+
+	limitedSrc := io.LimitReader(src, limit)


It would be nice if we had some form of rate-limit as well instead of just an absolute limit.

agreed, although not exactly clear how to do that.

Discussed offline. The current limit is 128KiB (or something like that). That should be low enough as it is.

Stebalien · 2021-03-19T00:39:58Z

v2/relay/relay.go

+		dest.CloseWrite()
+		if count == limit {
+			// we've reached the limit, discard further input
+			src.CloseRead()


We should reset both streams if there's an error here, not close them.

limit reached is not an error, but yes, we should check if there is one.

It's not an error, but we should still reset. If we send a close, it'll look like a clean close. We should only send an EOF if we receive an EOF.

mxinden · 2021-04-19T16:38:46Z

v2/client/reservation.go

+	result := &Reservation{}
+	result.Expiration = time.Unix(rsvp.GetExpire(), 0)
+
+	for _, ab := range rsvp.GetAddrs() {


How about requiring at least one address? As far as I can tell a reservation without a single address of the relay node is useless as one could not announce its relayed address to anyone else, correct?

v2/relay/relay.go

…time instead of TTL

…yed connections

marten-seemann · 2021-07-06T18:49:06Z

Changed the merge target from master to circuit-v2. Further development will happen in this feature branch.

vyzo requested review from Stebalien, yusefnapora, raulk and aarshkshah1992 February 4, 2021 23:06

vyzo removed the request for review from yusefnapora February 12, 2021 17:39

aarshkshah1992 mentioned this pull request Feb 14, 2021

Project Flare(decentralised Hole Punching) Phase1 Meta Issue libp2p/go-libp2p#1039

Closed

23 tasks

aarshkshah1992 reviewed Feb 15, 2021

View reviewed changes

aarshkshah1992 mentioned this pull request Feb 15, 2021

Integrate Limited Relay Reservations into AutoRelay with static config libp2p/go-libp2p#1053

Closed

aarshkshah1992 reviewed Feb 15, 2021

View reviewed changes

libp2p deleted a comment from vyzo Feb 16, 2021

aarshkshah1992 reviewed Feb 16, 2021

View reviewed changes

aarshkshah1992 mentioned this pull request Feb 17, 2021

Telemetry and metrics in Limited Relays libp2p/go-libp2p#1054

Closed

aarshkshah1992 reviewed Feb 17, 2021

View reviewed changes

aarshkshah1992 reviewed Feb 21, 2021

View reviewed changes

v2/client/reservation.go Show resolved Hide resolved

mxinden reviewed Mar 4, 2021

View reviewed changes

mxinden mentioned this pull request Mar 6, 2021

protocols/relay: Implement circuit relay specification libp2p/rust-libp2p#1838

Merged

Stebalien reviewed Mar 19, 2021

View reviewed changes

This was referenced Mar 26, 2021

Do we care about address length vs message length in circuit-relay? libp2p/specs#29

Open

Decentralized NAT traversal using nodes in the network libp2p/specs#307

Open

This was referenced Apr 6, 2021

Project Flare aka. decentralised hole punching protocol/web3-dev-team#21

Open

NAT traversal tracking issue libp2p/specs#312

Open

mxinden mentioned this pull request Apr 16, 2021

Add spec fo circuit relay v2 protocol libp2p/specs#314

Closed

mxinden reviewed Apr 19, 2021

View reviewed changes

v2/relay/relay.go Outdated Show resolved Hide resolved

mxinden mentioned this pull request May 5, 2021

p2p-circuit v2 spec libp2p/specs#325

Merged

vyzo and others added 24 commits July 4, 2021 20:47

untag peers with reservations when closing the relay

7e8bef0

gomod: get go-libp2p-asn-util

49e9649

add IP/ASN reservation constraints

5b7a615

gomod: update deps

6389453

fix e2e test

148d307

increase default limit duration to 2min

68d07f5

update protocol for vouched relay addrs; provide absolute expiration …

526fc92

…time instead of TTL

update for reservation changes

6d34d5b

add voucher to the reservation pb

16d5616

TODO about reservation vouchers

1cec70e

deduplicate protocol ID definitions between relay and client

4ec7298

add reservation vouchers

c7b3913

emit and consume reservation vouchers

533abb2

improve limit data test

a3d765a

deduplicate concurrent relay dials to the samke peer

75b4777

improve dialer deduplication

470c088

add a short timeout to dialing the relay in order to aid deduplication

2333151

gomod: fix go1.16 madness

84c6610

spec compliance: don't include p2p-circuit in reservation addrs

30b71a3

spec compliance: refuse reservation and connection attempts over rela…

ffa309e

…yed connections

test shim: add empty file in test directory

12eefbc

spec compliance: update protobuf

b4d0d1a

spec compliance: use libp2p envelopes for reservation vouchers

e1bbd83

fix staticcheck

97ec1d6

marten-seemann force-pushed the v2 branch from 0a6ad22 to 97ec1d6 Compare July 5, 2021 03:47

marten-seemann changed the base branch from master to circuit-v2 July 6, 2021 18:44

marten-seemann changed the title ~~[WIP] p2p-circuit v2~~ p2p-circuit v2 Jul 6, 2021

marten-seemann merged commit 9da0b93 into circuit-v2 Jul 6, 2021

vyzo deleted the v2 branch August 5, 2021 19:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

p2p-circuit v2 #125

p2p-circuit v2 #125

vyzo commented Feb 4, 2021 •

edited by raulk

vyzo commented Feb 5, 2021

vyzo commented Feb 12, 2021

aarshkshah1992 left a comment

aarshkshah1992 Feb 15, 2021

vyzo Feb 15, 2021

aarshkshah1992 left a comment

aarshkshah1992 Feb 15, 2021

vyzo Feb 15, 2021

aarshkshah1992 Feb 15, 2021

vyzo Feb 15, 2021

aarshkshah1992 left a comment •

edited

aarshkshah1992 left a comment

aarshkshah1992 Feb 16, 2021

vyzo Feb 16, 2021 •

edited

aarshkshah1992 Feb 16, 2021 •

edited

vyzo Feb 16, 2021

vyzo Feb 16, 2021

aarshkshah1992 Feb 17, 2021

vyzo Feb 17, 2021

mxinden Mar 4, 2021

vyzo Mar 4, 2021

Stebalien commented Mar 19, 2021

Stebalien left a comment

Stebalien Mar 19, 2021

Stebalien Mar 19, 2021

vyzo Mar 19, 2021

Stebalien Mar 25, 2021

Stebalien Mar 19, 2021

vyzo Mar 19, 2021

Stebalien Mar 25, 2021

mxinden Apr 19, 2021

marten-seemann commented Jul 6, 2021

p2p-circuit v2 #125

p2p-circuit v2 #125

Conversation

vyzo commented Feb 4, 2021 • edited by raulk

An Overview of the p2p-circuit/v2 Protocol

Establishing a relayed connection

Reservations

Connection Establishment

Connection Limits

Implementation Details

vyzo commented Feb 5, 2021

vyzo commented Feb 12, 2021

aarshkshah1992 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aarshkshah1992 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aarshkshah1992 left a comment • edited

Choose a reason for hiding this comment

aarshkshah1992 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo Feb 16, 2021 • edited

Choose a reason for hiding this comment

aarshkshah1992 Feb 16, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Mar 19, 2021

Stebalien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marten-seemann commented Jul 6, 2021

vyzo commented Feb 4, 2021 •

edited by raulk

aarshkshah1992 left a comment •

edited

vyzo Feb 16, 2021 •

edited

aarshkshah1992 Feb 16, 2021 •

edited