Hijacking in Rack 3

[ayushn21]

Hey folks,

I'm a bit confused about socket hijacking in Rack 3. I should mention I'm fairly new to working with Rack directly.

The spec says:

Full Hijack¶ ↑

Full hijack is used to completely take over an HTTP/1 connection. It occurs before any headers are written and causes the request to ignores any response generated by the application.

It is intended to be used when applications need access to raw HTTP/1 connection.

If rack.hijack is present in env, it must respond to call and return an IO instance which can be used to read and write to the underlying connection using HTTP/1 semantics and formatting.

Partial Hijack¶ ↑

Partial hijack is used for bi-directional streaming of the request and response body. It occurs after the status and headers are written by the server and causes the server to ignore the Body of the response.

It is intended to be used when applications need bi-directional streaming.

If rack.hijack? is present in env and truthy, an application may set the special response header rack.hijack to an object that responds to call, accepting a stream argument.

After the response status and headers have been sent, this hijack callback will be invoked with a stream argument which follows the same interface as outlined in “Streaming Body”. Servers must ignore the body part of the response tuple when the rack.hijack response header is present. Using an empty Array instance is recommended.

The special response header rack.hijack must only be set if the request env has a truthy rack.hijack?.

My understanding is that the usage of these headers has changed in Rack 3, as the upgrade guide says:

rack.hijack? now only applies to partial hijack

Previously, both full and partial hijiack were controlled by the presence and value of rack.hijack?. Now, it only applies to partial hijack (which now can be replaced by streaming bodies).

rack.hijack alone indicates that you can execute a full hijack

Previously, rack.hijack? had to be truthy, as well as having rack.hijack present in the request environment. Now, the presence of the rack.hijack callback is enough.

My first question is: does the spec need to be updated?

Secondly, my confusion lies in the behaviour of the two kinds of hijacking. Am I right in saying that partial hijacking writes HTTP response headers to the socket, whereas full hijacking does not?

If my above statement is correct, then should the following code write HTTP response headers since it's a "full hijack" as per the Rack 3 spec?

    body = proc do |stream|
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [200, { "content-type" => "text/plain", "rack.hijack" => body }, []]

I tested that on Puma and Falcon and both servers wrote HTTP headers back to the client before streaming the body.

I'd appreciate any clarification :). Thanks!

[ioquatix]

My first question is: does the spec need to be updated?

What do you think needs to be updated?

Am I right in saying that partial hijacking writes HTTP response headers to the socket, whereas full hijacking does not?

That's correct. The example you give is a partial hijack. However, it's better to write it like this now:

[200, {"content-type" => "text/plain"}, body]

Full hijack is very difficult to use correctly and impossible for HTTP/2+, I advise you don't use it. To use full hijack, you'd write code like this:

if io = env['rack.hijack']&.call
  io.write("connection: close\r\n\r\nHello World\r\n")
  io.close
end

[ayushn21]

What do you think needs to be updated?

Sorry I should've been clearer. I think these two lines in the spec need updating?

If rack.hijack? is present in env and truthy, an application may set the special response header rack.hijack to an object that responds to call, accepting a stream argument.

The special response header rack.hijack must only be set if the request env has a truthy rack.hijack?.

The upgrade guide says:

Previously, both full and partial hijiack were controlled by the presence and value of rack.hijack?. Now, it only applies to partial hijack (which now can be replaced by streaming bodies).

And the changelog says:

rack.hijack? (partial hijack) and rack.hijack (full hijack) are now independently optional.

The language in the spec makes me think rack.hijack? and rack.hijack are related to each other, whereas the upgrade guide and changelog say they're not. Or am I a bit confused and interpreting this incorrectly?

The example you give is a partial hijack.

Ahh right, thanks! Yeah I'm aware of streaming bodies, and also that full hijacking is a bad idea. My curiosity here is largely academic to try and understand what's what. Your talk at RubyKaigi 2022 sent me down this rabbit hole! (Fantastic talk btw!)

Thanks for the help :)

[ioquatix]

Let's step back a little bit.

In Rack 2, there was no way to advertise support for only partial hijack or only full hijack. The presence and truthiness of env['rack.hijack?'] implied:

1. You could respond with a partial hijack [200, {'rack.hijack' => body}, nil].
2. env['rack.hijack'] is available for full hijack.

This is enforced by Rack::Lint.

Let's clarify what's actually required:

• A partial hijack goes via the response headers - support for this has to be advertised, and that can be done by env['rack.hijack?'].
• A full hijack must invoke env['rack.hijack']. The mere presence of that field is sufficient to indicate that full hijack is possible.

Full and partial hijack are distinct operations and there is no real dependency between them. More specifically, all HTTP versions can support partial hijack, but not all versions can (easily) support full hijack - the expectations of full hijack is that you'd write response headers using the HTTP/1 protocol, but HTTP/2 isn't a stream based protocol (trying to call io.puts("header: value\r\n") simply doesn't make sense in HTTP/2). If you wanted to support full hijack in HTTP/2+, you'd need to actually implement an HTTP/1 <-> HTTP/x proxy between the full hijack IO and the actual response.

In Rack 3, streaming response body is always supported and a requirement of Rack 3, which is [200, {}, body]. It's equivalent to partial hijack, which is [200, {'rack.hijack' => body}, nil].

Because full hijack and partial hijack are actually distinct operations, and because Rails depends on full hijack, I did need to support it in falcon.

So, from the changelog:

rack.hijack? (partial hijack) and rack.hijack (full hijack) are now independently optional.

Yes, this is true, the cases are:

1. env['rack.hijack?'] is truthy implies that you can respond with {'rack.hijack' => body}.
2. env['rack.hijack'] being a callable proc implies that you can call it to do a full hijack.

Both of these are independent of each other. But in Rack 2, that was not the case (as per Rack::Lint linked above).

The upgrade guide states this clearly and also tries to guide users to wards streaming response bodies (because they are slightly more ergonomic from an interface POV):

Previously, both full and partial hijiack were controlled by the presence and value of rack.hijack?. Now, it only applies to partial hijack (which now can be replaced by streaming bodies).

This is stating that previously env['rack.hijack?'] being truthy implies that both (1) and (2) above are required, but that is no longer the case - env['rack.hijack?'] now only implies (1).

This is outlined in the Rack specification (derived from the Rack 3 Rack::Lint):

If rack.hijack? is present in env and truthy, an application may set the special response header rack.hijack to an object that responds to call, accepting a stream argument.

The above statement refers to the case (1).

The special response header rack.hijack must only be set if the request env has a truthy rack.hijack?.

This implies that applications should NOT set the response header rack.hijack UNLESS case (1) is advertised.

Does this clarify things for you or am I mis-understanding something? Let me make one more clarification, (1) talks about a response header called rack.hijack while (2) talks about an env key/value called rack.hijack. They are different.

[ayushn21]

That makes perfect sense, I finally get it now! Thanks so much for such a detailed writeup, I really appreciate it :D.