Improper use of Servlet InputStream Async I/O results in 100% CPU #5153
Assignees
Labels
Third Party
Issues with third party libraries or projects
Comments
jwalker-highspot
changed the title
That version of Eclipse Jetty is subject to some security reports. https://www.eclipse.org/jetty/security-reports.html Eclipse Jetty 9.4.31.v20200723 is available has many fixes, some even in websocket.
That stacktrace shows code that is doing a Servlet spec Async I/O read of HTTP Servlet Request bytes.
Issue #5090 is unique to the UnixSocket server connector. And is unrelated to your described issue. This has to do with Winstone behaviors in Jenkins. The current status of this one is about a SelectorProvider bug on the JVM / OS combo. Two pieces of information will help us narrow things down for you.
Those logs will hopefully tell us both where the problem is occurring, and what the nature of the problem is. Example: those DEBUG logs will tell us if you have the OS networking bug (or not). This will be logged at DEBUG level on the https://github.com/eclipse/jetty.project/blob/jetty-9.4.31.v20200723/jetty-io/src/main/java/org/eclipse/jetty/io/ManagedSelector.java#L160 |
Can you describe the steps you take to reproduce the problem with this project? |
|
Here are 10 thread dumps taken with jstack -l -e taken 10 seconds apart. I've clipped everything except one of the hung threads. It looks like it's reading data from the stack trace but other things suggest it might not be. For example, in the netstat -n -e -t output, no established tcp connections on the server's port are listed. (This server has been removed from load balancer and is not accepting new incoming connections.) In this case, you can see the loop it's stuck in involves SockJS.
I'll get you more info here ASAP (along with debug logs), but very vaguely speaking: |
|
... has a few bugs with how it's using Servlet Async I/O. Servlet Async I/O has many gotchas, and your code for Async I/O read definitely has a few. Read this https://www.slideshare.net/SimoneBordet/servlet-31-async-io (read is covered starting at slide 42)
|
joakime
added
Third Party
joakime
changed the title
|
Thanks for the quick and thorough response! We will look at patching sockjs-servlet or moving to a different sockjs implementation. |
|
Closing this issue, as it seems it's been addressed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jetty version
9.4.28.v20200408
Java version
openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment (build 11.0.5+10-post-Ubuntu-2ubuntu116.04)
OpenJDK 64-Bit Server VM (build 11.0.5+10-post-Ubuntu-2ubuntu116.04, mixed mode, sharing)
OS type/version
Ubuntu 16.04.6 LTS
Description
I have a jetty server listening for web socket requests ultimately handled by an implementation of sockjs as a servlet:
https://github.com/projectodd/sockjs-servlet
Over time, typically immediately after finishing with a request, more and more threads begin to hang. While hung, they use 100% of a single CPU core, and appear to just be spinning around in a loop within Jetty. Eventually, the zombie threads monopolize all cores.
The thread keeps going and CPU usage remains elevated even if I take the server out of the load balancer.
There seem to be many tickets describing vaguely similar issues, including:
#5090
#4537
#4717
jstack output varies slightly from case to case, but here's one example:
Strace of one of the stuck threads (it just loops through this same basic set of operations):
To me the segfault seems significant, but I don't know what to make of it.
Thanks!
The text was updated successfully, but these errors were encountered: