HttpParser.parsedHeader consume too much memory

[leonchen83]

Jetty version(s)

10.0.5

Java version/vendor (use: java -version)

openjdk version "11.0.2"

OS type/version

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

Description

HttpParser.parsedHeader consume too much memory than jetty 9.4

How to reproduce?

jetty 10.0.5

more details see attachment
admin-20210702-01.zip

[gregw]

@leoluecken83 Can you give us the exact stack trace of the areas you circled in red?
Also some examples of the headers you are sending would be useful

[leonchen83]

@gregw is this related issue 5562?
you can download admin-20210702-01.zip in attachment and open svg in chrome. then click red areas you can see stack trace in red areas

[leonchen83]

this profile generated by async-profiler.

[gregw]

@leonchen83 OK I can see the full stack traces now, so it is indeed effort spent building the header cache.
So firstly you can disable the header cache if it is not suitable for you load profile.
It's kind of related to #5562 as its the same memory usage. But the fix to that issue has been applied, so the cache should only be created on second requests on a connection.

But again, I can't tell you much more unless you can give me examples of the headers/requests being sent.

[leonchen83]

@gregw

curl "https://dev-erebor.nextop.cn/rest/market/market" ^
  -H "authority: dev-erebor.nextop.cn" ^
  -H "sec-ch-ua: ^\^" Not;A Brand^\^";v=^\^"99^\^", ^\^"Google Chrome^\^";v=^\^"91^\^", ^\^"Chromium^\^";v=^\^"91^\^"" ^
  -H "x-version: 643a4b7d" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" ^
  -H "accept: */*" ^
  -H "sec-fetch-site: same-origin" ^
  -H "sec-fetch-mode: cors" ^
  -H "sec-fetch-dest: empty" ^
  -H "referer: https://dev-erebor.nextop.cn/" ^
  -H "accept-language: zh-CN,zh;q=0.9,en;q=0.8" ^
  -H "cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IkltWmpOemswTVRrNUxUTm1Zamt0TkRGaE1pMDVNV05rTFRkbVl6TXlPVFV4TmpaaVlpST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ^%^3D^%^3D--742a49809da7daf9bff80f8f4e9bd9d977112127; _ga=GA1.2.1221233173.1600842719; lang=1; JSESSIONID.admin=admin9^@b2yjdk84qhop.1tu12pmkvk45.1pz8m3jokxsb2.admin" ^
  --compressed

curl "wss://dev-erebor.nextop.cn/rest/realtime" ^
  -H "Pragma: no-cache" ^
  -H "Origin: https://dev-erebor.nextop.cn" ^
  -H "Accept-Language: zh-CN,zh;q=0.9,en;q=0.8" ^
  -H "Sec-WebSocket-Key: G3j1oZyDeepRq+zZ6eCbww==" ^
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" ^
  -H "Upgrade: websocket" ^
  -H "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits" ^
  -H "Cache-Control: no-cache" ^
  -H "Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IkltWmpOemswTVRrNUxUTm1Zamt0TkRGaE1pMDVNV05rTFRkbVl6TXlPVFV4TmpaaVlpST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ^%^3D^%^3D--742a49809da7daf9bff80f8f4e9bd9d977112127; _ga=GA1.2.1221233173.1600842719; lang=1; JSESSIONID.admin=admin9^@b2yjdk84qhop.1tu12pmkvk45.1pz8m3jokxsb2.admin" ^
  -H "Connection: Upgrade" ^
  -H "Sec-WebSocket-Version: 13" ^
  --compressed

curl "https://dev-erebor.nextop.cn/rest/market/market/refresh" ^
  -H "authority: dev-erebor.nextop.cn" ^
  -H "sec-ch-ua: ^\^" Not;A Brand^\^";v=^\^"99^\^", ^\^"Google Chrome^\^";v=^\^"91^\^", ^\^"Chromium^\^";v=^\^"91^\^"" ^
  -H "x-version: 643a4b7d" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" ^
  -H "accept: */*" ^
  -H "sec-fetch-site: same-origin" ^
  -H "sec-fetch-mode: cors" ^
  -H "sec-fetch-dest: empty" ^
  -H "referer: https://dev-erebor.nextop.cn/" ^
  -H "accept-language: zh-CN,zh;q=0.9,en;q=0.8" ^
  -H "cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IkltWmpOemswTVRrNUxUTm1Zamt0TkRGaE1pMDVNV05rTFRkbVl6TXlPVFV4TmpaaVlpST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ^%^3D^%^3D--742a49809da7daf9bff80f8f4e9bd9d977112127; _ga=GA1.2.1221233173.1600842719; lang=1; JSESSIONID.admin=admin9^@b2yjdk84qhop.1tu12pmkvk45.1pz8m3jokxsb2.admin" ^
  --compressed

curl "https://dev-erebor.nextop.cn/rest/utility/bootstrap" ^
  -X "POST" ^
  -H "authority: dev-erebor.nextop.cn" ^
  -H "content-length: 0" ^
  -H "sec-ch-ua: ^\^" Not;A Brand^\^";v=^\^"99^\^", ^\^"Google Chrome^\^";v=^\^"91^\^", ^\^"Chromium^\^";v=^\^"91^\^"" ^
  -H "x-version: 643a4b7d" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" ^
  -H "content-type: application/x-www-form-urlencoded" ^
  -H "accept: */*" ^
  -H "origin: https://dev-erebor.nextop.cn" ^
  -H "sec-fetch-site: same-origin" ^
  -H "sec-fetch-mode: cors" ^
  -H "sec-fetch-dest: empty" ^
  -H "referer: https://dev-erebor.nextop.cn/" ^
  -H "accept-language: zh-CN,zh;q=0.9,en;q=0.8" ^
  -H "cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IkltWmpOemswTVRrNUxUTm1Zamt0TkRGaE1pMDVNV05rTFRkbVl6TXlPVFV4TmpaaVlpST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ^%^3D^%^3D--742a49809da7daf9bff80f8f4e9bd9d977112127; _ga=GA1.2.1221233173.1600842719; lang=1; JSESSIONID.admin=admin9^@b2yjdk84qhop.1tu12pmkvk45.1pz8m3jokxsb2.admin" ^
  --compressed

we refresh chrome and execute above requests many times, then we got the profiler

[gregw]

Ahhh!!! The proposed fix for #5562 of only creating the expensive header cache on a second request was taken out of the fix for #5562.
So we still always create the expensive header cache, even if it is not used.

@leonchen83 it is really important for us to see the headers of your requests so that we can improve our cache handling. It may just be that you have non-persistent HTTP requests, so the cache is never used.

@sbordet Can you re-evaluate your feedback on #5562 - as I think only creating the cache on a second request may well be helpful here.

[gregw]

@leonchen83 our comments passed! Thanks for that. You are using curl to send individal requests with lots of custom headers. So each connection only ever has a single request, thus the header cache is of zero use to you (unless your real load actually does use persistent connections). If you only ever have a single request per connection, then set the header cache size to 0.

@sbordet I really REALLY think we should reconsider the original fix to #5562 and only create the expensive ArrayTrie if we have a second request!