Receiving chunk extensions in HTTP/1.1 chunked encoding

[BGR360]

Requested functionality

When writing an HTTP server with hyper, I want to be able to receive and read the chunk extensions that clients send when using HTTP/1.1 chunked encoding.

Currently hyper simply discards these chunk extensions.

Motivating use case

Implementing an S3-compatible server using hyper would require this functionality. Amazon S3 Signature Version 4 includes a "streaming mode" where the SHA256 signature of each chunk is included as a chunk extension:

Example Header:

PUT /examplebucket/chunkObject.txt HTTP/1.1\r\n
<Other Headers>
x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD\r\n
Content-Encoding: aws-chunked\r\n
x-amz-decoded-content-length: 66560\r\n
Transfer-Encoding: chunked\r\n
\r\n
<Payload>

Example chunk from payload:

10000;chunk-signature=ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648\r\n
<65536-bytes>

The server is required to validate the signature of each chunk as it is sent, so it needs to be able to extract the chunk signatures from the HTTP chunk extensions.

Idea 1: New frame type

Define a new Frame variant for chunk extensions.

impl Frame<T> {
    pub fn chunk_extensions(extensions: HeaderMap) -> Self { ... }
    pub fn is_chunk_extensions(&self) -> bool { ... }
    pub fn into_chunk_extensions(self) -> Result<HeaderMap, Self> { ... }
    pub fn chunk_extensions_ref(&self) -> Option<&HeaderMap> { ... }
    pub fn chunk_extensions_mut(&mut self) -> Option<&mut HeaderMap> { ... }
}

Then the Incoming body can yield a Frame::chunk_extensions(..) before each Frame::data(..) (or rather, before the first frame in each chunk?).

This seems like the least invasive option, but might not really solve the problem sufficiently (see Limitations below).

Limitations

Ordering

In order to implement the S3 chunked signature, the server has to be able to associate each set of chunk extensions with the chunk that it precedes.

However, the Body trait provides no guarantee about the order of different types of Frames that are yielded. So programmers would have to trust that any Body is going to yield chunk extensions before each chunk. But theoretically somebody could implement Body in a way that subverts that expectation.

Perhaps this is not truly an issue, but rather an "understood quirk" of the Body trait. I say this because a similar problem exists with the chunked trailers. The Body trait does not document any guarantee that you'll receive at most one Frame::trailers(..), nor that that frame will be the last frame yielded. But I'd wager that users of a Body might make one or more of those assumptions.

Sending

It's not clear how the hyper client code should interpret any Frame::chunk_extensions(..) that are yielded by the request Body 🤔 🤔 🤔 .

Idea 2: Callbacks

Maybe the hyper server code could allow callers to specify some sort of callback that the Dispatch will call whenever it encounters chunk extensions. Haven't thought this idea through very much.

Alternatives considered

I can't think of any. The processing of the chunk extensions is completely hidden away in hyper's guts and cannot be customized by end users.

Any thoughts on other ways to expose this functionality?

[seanmonstar]

Thanks for the write up! Yea, chunked extensions are a part of HTTP/1.1 that sounded good in the spec, but few implementations bothered to support. And since it's allowed to ignore them, any additional hops (proxies, gateways, load balancers) will likely strip them too.

We could try to figure out how to support them, but I genuinely think that part of S3 should be ignored, and a better supported mechanism should be used to checksum the payload.

[BGR360]

additional hops (proxies, gateways, load balancers) will likely strip them too

A fair point that I hadn't considered.

I genuinely think that part of S3 should be ignored

I wish our customers thought the same 🙃