File upload gives net:ERR_CONNECTION_RESET

[rijkr]

Jetty version
9.4.39.v20210325

Java version

OpenJDK 64-Bit Server VM - 1.8.0_282

OS type/version

Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-140-generic x86_64)

Description

Our Jetty Web Server application is deployed on various servers and can be accessed over the Internet via router port forwarding or Apache reverse proxy. After upgrading Jetty 9.4.38.v20210224 to 9.4.39.v20210325, one server gives a "(failed) net:ERR_CONNECTION_RESET" error for large 2MB< file uploads via https://fail.example.com : upload of smaller files, such as 60 KB, will work fine. This finding cannot be reproduced on our reference site https://okay.example.org : all types of file uploads work fine there. (URLs are obfuscated for privacy. The error is observed in Chromium Browser > Development tools > Network.)

So what is the difference between the two sites? After numerous checks and tests the only difference we see is the kind of SSL cert used:

1. https://fail.example.com uses a wild card certificate having two "Not Critical" Subject Alternative DNS Names: *.example.com and example.com (in that order).
2. https://okay.example.org uses a plain (no wild card) Let's Encrypt certificate with explicit Subject Alternative DNS Name okay.example.org (DNS names for other subdomains are also present).

The upload errors for https://fail.example.com disappear when we revert to Jetty 9.4.38.v20210224.

Could this behavior be related to #6034 ?

[joakime]

Can you capture DEBUG logs for the failing example and attach them here?

[rijkr]

@joakime That's a bit hard to do as it's a production environment for a user organisation. Yesterday I spent a lot of time figuring out what was going on, and I finally solved the problem by creating a new installer with the older Jetty version. If absolutely necessary I have to ask the user to set up a debugging session when web activity is low. If you can somehow get clues from the code itself, that would be preferable.
P.S. The log4j log file show no warnings or errors.

[gregw]

This may be fixed by #6142
We introduced an issue in 9.4.39 with SSL buffering, that can sometimes result in an underflow state when the buffer could be compacted. I'm not exactly sure how the different certificates could effect that, but then it is at least plausible. Can you test that branch in production (or do a build with just that change applied?)

Either way, we are likely to make a new release early next week.

[rijkr]

@gregw Thanks for your quick reply! I'd rather wait for the next 9.4.40 (?) release with the #6142 fix. I can then set DEBUG right away during installation. What log4j DEBUG scope should I use? I assume log4j.logger.org.eclipse.jetty=DEBUG will probably create too much log entries?

[gregw]

Rijk, assuming my guess about the cause is correct, then logging in org.eclipse.jetty.io.ssl would be appropriate.... but then if my guess is correct, there will be nothing to see in that logging. If my guess is wrong, that debugging might help, but it probably won't. So perhaps just deploy it with no extra logging for a start.

…

On Sat, 10 Apr 2021 at 23:25, Rijk Ravestein ***@***.***> wrote: @gregw <https://github.com/gregw> Thanks for your quick reply! I'd rather wait for the next 9.4.40 (?) release with the #6142 <#6142> fix. I can then set DEBUG right away during installation. What log4j DEBUG scope should I use? I assume log4j.logger.org.eclipse.jetty=DEBUG will probably create too much log entries? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6152 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARJLL4DNHOBFZY2TGTREDTIBGWLANCNFSM42WOIYXQ> .

-- Greg Wilkins ***@***.***> CTO http://webtide.com

[rijkr]

@gregw When we install the new release we will have enough time to test and set log levels as needed. Thanks for your support. I'll keep you posted about the results.

[rijkr]

@gregw We installed 9.4.40.v20210413 and first tests show the issue is solved. We deployed to production and will remain alert to user reports for the next few days.

[sbordet]

@rijkr closing this as solved. Please reopen if you still have problems.