Hijacked connection from HTTP/3 unable to open/accept new QUIC stream

[RithvikChuppala]

I've been working on MASQUE and using HTTP/3 to tunnel QUIC to a remote destination server (sending a CONNECT-UDP request to an HTTP Masque server in order to proxy QUIC to a remote server via the initial HTTP3/QUIC connection). I've described my overall approach in a follow-up here: #3370 (tldr: I was able to accomplish tunneling by hijacking the quic.Connection and quic.Stream used in the initial HTTP/3 Connect-UDP exchange. After the masque server has opened a UDP socket to the destination server, we can then use that hijacked Stream, along with some net.Conn stubbing and io.Copying, as the substrate upon which the tunneled packets are delivered).

My issue is as follows: I'm able to use the hijacked stream to read and write data to/from the client and masque server (which is indeed how the tunneling happens); I am also able to use the hijacked connection to read and write datagrams between the two as well. What I'm not able to do is open a new stream on that hijacked connection and have the masque server accept a new stream on the other side of the hijacked connection. The client side opens the stream and starts writing but the other side doesn't proceed past the "AcceptStream" call.

Here's a snippet:
Client side:

outer_stream2, err := outer_conn.OpenStreamSync(context.Background())
if err != nil {
panic(err)
}
outer_stream2.Write([]byte("f"))
fmt.Println("sent another F")
outer_stream2.Write(buf)
outer_stream2.Close()
fmt.Println("wrote buf and closed")
readBuf2, err := io.ReadAll(outer_stream2)
fmt.Println("readBuf size: ", len(readBuf2))

Here, everything seems fine until readBuf is empty (it shouldn't be because the server side should echo whatever was sent)

Server side:

fmt.Println("waiting for stream")
stream, err := conn.AcceptStream(context.Background())
fmt.Println("accepted stream")
if err != nil {
	panic(err)
}

Here, the "accepted stream" print statement is never reached and the client blocks waiting for the stream.

It's strange since it seems all the other functionalities of the Connection are working apart from this (writing/reading from the original stream, sending/receiving datagrams, ConnectionStates match up, etc).

This is a roadblock since MASQUE uses outer connection streams to forward packets to multiple remote destination servers (ie Outer Stream <-> Inner Connection). The timing of this issue works out since it seems Masque is starting to be prioritized (#4392; #4393) so this may also help future development. If anyone had some insight as to why this might be or why this could be intended behavior and how to navigate around it, or would also like some code snippets, please let me know.

Thanks in advance!

[marten-seemann]

Can you point me to the section in the specification where CONNECT-UDP requires hijacking of streams?

[RithvikChuppala]

Does it not require you to hijack the connection? It seems most provisional attempts (including the one linked from a few days ago) involve connection hijacking (instead of creating a new connection). Regardless of whether or not Masque itself requires it, it does seem like a roadblock for anything that might want to tunnel Quic over Quic in the style that I mentioned since I'm unable to open new streams on the connection (I've read a few papers that describe some advantages of stream-over-stream as opposed to stream-over-datagram, specifically over lossy links, so this might not be entirely unrealistic).

[marten-seemann]

I don't get it. The specification doesn't require stream hijacking, so why would it be a roadblock? Hijacking works perfectly fine by the way, webtransport-go is using it all the time.

[RithvikChuppala]

Not sure if my terminology is proper but there must at least be a connection hijack, correct (or else we would just creating a separate connection instead of re-using the HTTP/3 connection that we opened up to send the CONNECT-UDP); that's what I meant by "hijacking", please correct me if I'm misusing the term.

The issue isn't in hijacking the connection itself but rather creating new streams from the hijacked connection. In order to make new e2e remote server connections, we need to open up new streams in our connection with the Proxy which I'm unable to do (hence the roadblock); I'm able to perform connection hijacking properly (and am also able to do everything else with that connection like sending/receiving datagrams except open new streams). I can also give more code samples if it helps but the gist of the issue is in the code sample above.

The current way I'm working around that issue is by creating a separate connection to the Proxy and using that to create new streams for e2e connections but this workaround doesn't seem as fitting as directly using the QUIC connection in the CONNECT-UDP HTTP request.

[marten-seemann]

Sounds like you're building your own proxying protocol, not CONNECT-UDP. This is only supported to the extent that WebTransport needs it. Note that WebTransport is able to create and accept new streams, so I'm very confident that this API does work as intended.