Message head to large on 22 kb of headers #3189
-
The issueI am having issues with header parsing while integrating with a 3rd party API. I get the error: These must be the response headers since I only have 2 pretty small & standard headers in my requests. This surprises me since the response headers are around 24 kb which from what I read should be ok using the default client builder. I have tried out the API using a rest client and there everything looks fine. I have googled around and taken a look at some of the source code and I came up with setting the max buf size: let https = HttpsConnector::new()
let client = Client::builder()
.http1_max_buf_size(500 * 1024)
.build::<_, Body>(https);That however didn't work. I am a bit stuck, any insights would be much appreciated. CodeHere is the code that makes the request, pretty standard but I thought I would include it: let request = Request::builder()
.uri(&url)
.method(Method::POST)
.header("content-type", "application/json")
.header("accept-api-version", "resource=2.0,protocol=1.0")
.body(Body::from(body))
.map_err(anyhow::Error::new)?;
let res = self
.client
.request(request)
.await
.map_err(anyhow::Error::new)?;
let response_body = hyper::body::aggregate(res)
.await
.map_err(anyhow::Error::new)?;
let res: Step3Response =
serde_json::from_reader(response_body.reader()).map_err(anyhow::Error::new)?;Trace[2023-03-30T16:01:24Z TRACE hyper::client::pool] take? ("https", tjwlj.audkenni.is:443): expiration = Some(90s) |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 3 replies
-
|
What part of the response is too big? It's not the buffer size, you'd see a debug log about that. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Sean, I believe it's related to the headers, but exactly what causes it eludes me. This is a polling endpoint that provides cookies via set-cookie headers once a user has authenticated. When I was testing the endpoint I noticed that the total size of response headers went from 16 kb to 24 kb once the user had authenticated, which fits pretty well with the debug message. I haven't been able to figure out the exact cause of the error, but I assume it's related to this. If it's not the buffer size, are there any other likely culprits that cause hyper to throw this error? |
Beta Was this translation helpful? Give feedback.
-
|
@seanmonstar I did some more digging and I think I've figured this out. The 3rd party API increases the number of response headers when I get the error. The number of headers surpasses 100. I looked through the Hyper source code and saw that the TooManyHeaders error from httparse maps to the Parse::TooLarge error: impl From<httparse::Error> for Parse {
fn from(err: httparse::Error) -> Parse {
match err {
httparse::Error::HeaderName
| httparse::Error::HeaderValue
| httparse::Error::NewLine
| httparse::Error::Token => Parse::Header(Header::Token),
httparse::Error::Status => Parse::Status,
httparse::Error::TooManyHeaders => Parse::TooLarge,
httparse::Error::Version => Parse::Version,
}
}
}In src/proto/h1/role.rs we have this constant which as I understand it controls the size of the header array passed to httparse: const MAX_HEADERS: usize = 100;I am wondering, can you think of any workaround that is currently available in hyper? |
Beta Was this translation helpful? Give feedback.
-
|
Oh, thanks for digging into this! I'd say there's a couple actions to do then:
|
Beta Was this translation helpful? Give feedback.
-
|
Yeah, 100+ headers is a bit strange, I was shocked to see it myself 😅. I have a pretty big deadline at the end of the month. I'll write up an issue and try for a PR as well once I have finished that. |
Beta Was this translation helpful? Give feedback.
Oh, thanks for digging into this! I'd say there's a couple actions to do then:
role.rswhen converting from anhttparse::Error, to help debugging in the future.