RFC6062: Implement server side for TCP allocations #315

AndyLc · 2023-03-22T03:46:27Z

Description

Builds off of #148

Currently the PR includes a commit for the client side too, but my plan is to rebase off #311 once it is merged. I am opening this PR for any initial feedback/review.

Main changes include:

CreateAllocation now takes in a Protocol, either TCP or UDP, which describes the allocation <-> peer connection`.
allocation.NewManager now takes in a Protocol, either UDP/TCP/TLS which describes the client <-> turn connection type, not the allocation.
Add AllocateListener, (I don't think we need AllocateConn anymore?), connectionHandler, connection maps, etc to handle TCP-connection based allocation logic.
Move stun_conn.go to utils/stun_conn.go and add .Conn() method to get underlying net.Conn. This is because in handleConnectionBindRequest, after we receive the request and convert the TURN <-> Peer connection to a data connection, we need to retrieve the original TCP connection to io.Copy to. STUNConn's ReadFrom does not seem appropriate.

Usage

Start turn server (turn/examples/turn-server/tcp)

./tcp --public-ip 127.0.0.1 --port 3478 --users andy=secret

Run tcp client, (turn/examples/turn-client/tcp-alloc)

./tcp-alloc --host 127.0.0.1 --user andy=secret -signaling

Run another tcp client, which gets the relay IP from the other client and sends its own.

./tcp-alloc --host 127.0.0.1 --user andy=secret

Both clients should read and write a single message.

Reference issue

#143, #118

rg0now · 2023-03-25T20:21:08Z

Thanks a lot, this is a super-useful work!

One suggestion: is it possible to rebase this PR to on top of #311? At least for me it'd be easier to see the client and the server side changes separately (I'm much more familiar with the server-side code).

Add AllocateListener, (I don't think we need AllocateConn anymore?), connectionHandler, connection maps, etc to handle TCP-connection based allocation logic.

You mean the RelayAddressGenerator interface? Currently Allocate has the following signature:

AllocateConn(network string, requestedPort int) (net.Conn, net.Addr, error)

You suggest to rename this to AllocateListener, no?

Now, I think we again face the same problem as on the client side: the thingie returned from AllocateListener should both be able to make new connections (implement our transport.Dialer interface) and accept new connections (implement net.Listener). So supposing that we create the new Relay interface that's both a Dialer and a Listener in internal/transport the new signature for AllocateListener would be something like the below:

AllocateListener(network string, requestedPort int) (transport.Relay, net.Addr, error)

But then we could also call this AllocateRelay no? Wdyt?

codecov · 2023-05-19T23:06:41Z

Codecov Report

Patch coverage: 51.42% and project coverage change: -2.01 ⚠️

Comparison is base (521e5ad) 69.21% compared to head (e1e061f) 67.20%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #315      +/-   ##
==========================================
- Coverage   69.21%   67.20%   -2.01%     
==========================================
  Files          42       42              
  Lines        2823     3193     +370     
==========================================
+ Hits         1954     2146     +192     
- Misses        701      858     +157     
- Partials      168      189      +21

Flag	Coverage Δ
go	`67.20% <51.42%> (-2.01%)`	⬇️
wasm	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
internal/allocation/five_tuple.go	`100.00% <ø> (ø)`
internal/server/server.go	`59.01% <0.00%> (-4.15%)`	⬇️
relay_address_generator_none.go	`21.87% <0.00%> (-13.13%)`	⬇️
relay_address_generator_range.go	`0.00% <0.00%> (ø)`
server_config.go	`41.66% <ø> (ø)`
utils/stun_conn.go	`62.50% <0.00%> (ø)`
internal/server/turn.go	`51.39% <35.33%> (-7.86%)`	⬇️
internal/allocation/allocation_manager.go	`57.57% <53.84%> (ø)`
relay_address_generator_static.go	`54.34% <66.66%> (+7.91%)`	⬆️
server.go	`69.69% <70.00%> (-0.94%)`	⬇️
... and 1 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

AndyLc · 2023-05-19T23:18:29Z

Hi @stv0g and @rg0now,

I think this PR should now be ready for review now that the client side is merged.

I believe we want the API to support a custom Dial (when an allocation receives a Connect request and creates a connection to peer via Dial), and a custom Listen (when an Allocation is created and listens to create data connection).

The custom listen is already possible through specifying a RelayAddressGenerator's AllocateListener that I added. What do you guys think is suitable for the custom Dial?

Perhaps a transport.Dialer can be specified in turn.ServerConfig?

stv0g

Hi @AndyLc,

great work :-) I am a big fan of your PR like I was already for the client-side 👍

I started reviewing it. But, will need to do this in several passed as its to big to review it at once..

Maybe @rg0now also has some capacity to support the reviewing process?

Thanks already!

stv0g · 2023-05-20T11:03:25Z

client_test.go

@@ -227,7 +228,7 @@ func TestTCPClient(t *testing.T) {
 	require.NoError(t, err)

 	client, err := NewClient(&ClientConfig{
-		Conn:           NewSTUNConn(conn),


Isn't this breaking the API by removing NewSTUNConn?

I agree that isnt a nice API. And I would much like to see it moved to pion/stun (see #308 for a related issue).

I think we currently have a lot STUN related code in pion/turn. E.g. PerformTransaction which would be better located in pion/stun. However, this is a different issue which we should not address in this PR.

stv0g · 2023-05-20T11:06:45Z

server_config.go

@@ -67,7 +70,7 @@ func (c *PacketConnConfig) validate() error {
 // ListenerConfig is a single net.Listener to accept connections on. This will be used for TCP, TLS and DTLS listeners
 type ListenerConfig struct {
 	Listener net.Listener
-
+	Protocol string // TCP, or TLS


Please indicate the expected string constants. E.g. "tcp", "tls" in the comment.

stv0g · 2023-05-20T11:08:47Z

internal/server/turn_test.go

@@ -111,3 +112,67 @@ func TestAllocationLifeTime(t *testing.T) {
 		assert.Nil(t, r.AllocationManager.GetAllocation(fiveTuple))
 	})
 }
+
+func TestTCPAllocation(t *testing.T) {
+	t.Run("TCPAlloc", func(t *testing.T) {


Do we really need a sub test here if there is only one in TestTCPAllocation?

stv0g · 2023-05-20T11:09:31Z

internal/server/errors.go

@@ -19,7 +19,14 @@ var (
 	errFailedToCreateChannelData              = errors.New("failed to create channel data from packet")
 	errRelayAlreadyAllocatedForFiveTuple      = errors.New("relay already allocated for 5-TUPLE")
 	errUnsupportedTransportProtocol           = errors.New("RequestedTransport must be UDP or TCP")
+	errUnsupportedClientProtocol              = errors.New("cannot allocate tcp if client connection transport is not TCP/TLS")


Please change lowercase tcp to TCP

Also, I prefer to use the following wording for errors: failed to .... E.g. in this instance failed to allocate TCP...

stv0g · 2023-05-20T11:13:06Z

examples/turn-server/tls/main.go

@@ -77,6 +77,7 @@ func main() {
 					RelayAddress: net.ParseIP(*publicIP),
 					Address:      "0.0.0.0",
 				},
+				Protocol: "tls",


Do we break this for existing users if they dont set the protocol?

stv0g · 2023-05-20T11:14:54Z

internal/allocation/allocation_manager.go

+		a.RelayListener = listener
+		a.RelayAddr = relayAddr
+	} else {
+		conn, relayAddr, err := m.allocatePacketConn("udp4", requestedPort)


Not a change you introduced, but I am wondering why we are listening on IPv4 only..
Seems like we are not supporting REQUESTED-ADDRESS-FAMILY at all..

Another topic for another PR :D

stv0g · 2023-05-20T11:16:13Z

internal/allocation/allocation_manager_test.go

+		AllocateListener: func(network string, requestedPort int) (net.Listener, net.Addr, error) {
+			config := &net.ListenConfig{Control: func(network, address string, c syscall.RawConn) error {
+				var err error
+				c.Control(func(fd uintptr) {


Check error of c.Control()

stv0g · 2023-05-20T11:18:13Z

relay_address_generator_none.go

+	config := &net.ListenConfig{Control: func(network, address string, c syscall.RawConn) error {
+		var err error
+		c.Control(func(fd uintptr) {
+			err = syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, unix.SO_REUSEADDR|unix.SO_REUSEPORT, 1)


Can you add a comment explaining the reasoning behind SO_REUSEADDR/PORT?

stv0g · 2023-05-20T11:29:35Z

relay_address_generator_range.go

@@ -102,6 +106,49 @@ func (r *RelayAddressGeneratorPortRange) AllocatePacketConn(network string, requ
 	return nil, nil, errMaxRetriesExceeded
 }

+// AllocatePacketConn generates a new PacketConn to receive traffic on and the IP/Port to populate the allocation response with


The comment does not match the function name

rg0now · 2023-05-30T17:44:10Z

First of all: thanks for this big chunk of code, the effort is truly appreciated!

I've made a first pass and I concluded that I will need at least another round...:-)

My major concern right now is that the TCP dialer in the allocation handler is hard-coded to net.Dialer as per the below:

func (a *Allocation) newConnection(cid proto.ConnectionID, dst string) error {
        ...
        
	dialer := &net.Dialer{ ... }

	conn, err := dialer.Dial("tcp", dst)

        ...
}

This is problematic because the caller cannot pass in their own Dialer to implement custom dialers, e.g., to report new connections to Prometheus.

I think a better solution would be to let the user to (finally!) really implement RelayAddressGenerator.AllocateConn and call manager.allocateConn every time we would create a new connection in allocation.newConnection. This unfortunately would force the user to play the SO_REUSEPORT/SO_REUSEADDR trick inside the AllocateConn implementation they provide for us (btw I think SO_REUSEPORT is enough here but adding also SO_REUSEADDR should make no harm) but that's why we would provide the prefab relay-address-generators that would cover the straightforward use cases (i.e., generate TCP connections).

An alternative would be to let the user to specify a transport.Dialer in the RelayAddressGenerator and call the Dial function on this Dialer ourselves from allocation.newConnection, or we could even unify the AllocateListener and AllocateConn functionality by passing only a transport.Net to the allocation manager, but I guess this would be an overkill (and would break the API).

stv0g mentioned this pull request Apr 1, 2023

RFC6062 implementation #143

Closed

stv0g mentioned this pull request Apr 19, 2023

RFC6062: Implement client side for TCP allocations #311

Merged

This was referenced May 10, 2023

Support active TCP candidates (RFC6544) pion/ice#565

Merged

Add a callback to handle allocation requests #325

Draft

Implement server side for TCP allocations

e1e061f

AndyLc force-pushed the server branch from c7aff31 to e1e061f Compare May 19, 2023 23:01

AndyLc marked this pull request as ready for review May 19, 2023 23:18

stv0g self-requested a review May 20, 2023 10:58

stv0g requested changes May 20, 2023

View reviewed changes

rg0now mentioned this pull request Apr 20, 2024

TURN-client tcp-alloc example #391

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC6062: Implement server side for TCP allocations #315

RFC6062: Implement server side for TCP allocations #315

AndyLc commented Mar 22, 2023 •

edited by stv0g

rg0now commented Mar 25, 2023

codecov bot commented May 19, 2023

AndyLc commented May 19, 2023

stv0g left a comment

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

stv0g May 20, 2023

rg0now commented May 30, 2023

RFC6062: Implement server side for TCP allocations #315

Are you sure you want to change the base?

RFC6062: Implement server side for TCP allocations #315

Conversation

AndyLc commented Mar 22, 2023 • edited by stv0g

Description

Usage

Reference issue

rg0now commented Mar 25, 2023

codecov bot commented May 19, 2023

Codecov Report

AndyLc commented May 19, 2023

stv0g left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rg0now commented May 30, 2023

AndyLc commented Mar 22, 2023 •

edited by stv0g