Memory Leak in Jetty HttpConnections with SSE in Spring Boot 3.2.4 Using Jetty 12.0.7 #32629
Assignees
Labels
in: web
Issues in web modules (web, webmvc, webflux, websocket)
status: waiting-for-triage
An issue we've not yet triaged or decided on
Comments
spring-projects-issues
added
the
status: waiting-for-triage
snicoll
added
the
in: web
|
Thanks for the sample @vakiovale - the fact that only changing the Jetty version pointed to a bug in Jetty, or a bug in our Jetty support. I confirmed that:
This is a perfectly valid use case. There are some runtime differences, but we do support this use case. DebuggingI've tried running the latest 12.0.9-SNAPSHOT version of Jetty, because a connection leak was recently fixed. <properties>
<java.version>17</java.version>
<jetty.version>12.0.9-SNAPSHOT</jetty.version>
</properties>
<repositories>
<repository>
<id>jetty-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/jetty-snapshots/</url>
<snapshots><enabled>true</enabled></snapshots>
</repository>
</repositories>When the connection is closed from the client side, we're seeing the following: Nothing looks suspicious to me, as the
Later, connections are reclaimed when the idle timeouts are triggered: In short, I don't have a clear understanding of the behavior difference right now. Maybe we should try to create a repro case that doesn't involve the Spring MVC stack. |
|
I managed to track down the behavior difference between Jetty 12.0.3 - 12.0.4+ with Spring MVC and SSE events. When Spring MVC produces SSE events asynchronously, events are written to the Servlet response and flushed. With Jetty 12.0.3, this would be caught here and the With Jetty 12.0.4+, Unfortunately, I could not reproduce this reliably with a minimal Jetty sample and I'm not familiar enough with Jetty's connection lifecycle. I'm sorry to ping you here @gregw, but do you have any insight or advice here that could help me craft a reproducer for Jetty? Or is this a bug in Spring MVC in your opinion? I did debug the repro app with Jetty 12.0.3 and the stacktrace was the following up to the line that aborted the servlet channel : |
|
Hey, thanks @bclozel for your help with this issue.
We also observed this same behavior.
This observation made me question whether I had made an error, as I did not notice any cleanup activity from the timeout handler. Below, I can provide examples of what I observed. It seems that the timeout handler continuously loops (every 30 seconds for each client cancellation), yet the HttpConnection objects are not being cleared from memory. Test runsMy test scenario for versions 12.0.3, 12.0.7, and 12.0.9-SNAPSHOT involved the following steps:
Test with Jetty 12.0.3After cancelling request: Taking a heap dump with version 12.0.3, you will notice that the Test with Jetty 12.0.7You can see the Checking the logs 1.5 minutes later, we can see that the timeout handler loops every 30 seconds: Screenshot of the memory dump shows that Test with Jetty 12.0.9-SNAPSHOTThis looks pretty much the same as with 12.0.7, but I'll show the logs and screenshots here as well:
And the same problem occurs after timeouts as well. Extra test with 12.0.9-SNAPSHOT: I canceled 12 requests consecutively, and the timeout loop never ends. Even over 30 minutes later, we still observe the same pattern in the logs and in the memory dump.
|
|
@vakiovale I think the timeout handler issue is completely separate and should be raised to the Jetty team separately. I had a discussion with @rstoyanchev about this issue and we think that we might commit a fix in Spring Framework for this. Also, we are wondering if Jetty's behavior is consistent in this case. In a typical asynchronous Servlet write scenario, an We are relying on the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Our Spring Boot 3.2.2 application experiences a memory leak leading to an
OutOfMemoryError, with persistentorg.eclipse.jetty.server.internal.HttpConnectionobjects found in the heap dump. This occurs when using Server-Sent Events (SSE) with Flux endpoints and is particularly evident after anEofExceptionis thrown by Jetty after the client disconnects but the server attempts to continue emitting events. The issue has been replicated in a minimal project to demonstrate the behavior under specific configurations and versions.Link to the example project on GitHub
The example application uses Spring Boot 3.2.4 (and Jetty 12.0.7) and has a
Flux<ServerSentEvent>endpoint emitting heartbeats for a duration of 10 seconds.Steps to Reproduce
curlto connect to the SSE endpoint atlocalhost:9898/bug/huntand then terminate the connection manually using Ctrl+C.HttpConnectioninstances remain in memory, especially afterEofExceptionoccurs due to attempted heartbeat emissions.Example
Doing this twice:
Taking a heap dump from the application (using IntelliJ IDEA's memory snapshot) and observing two instances of
HttpConnection:Expected Behavior
HttpConnection instances should be cleared from memory when the client disconnects, preventing any memory leaks.
Possible Solution
spring-boot-starter-webor setting the application toWebApplicationType.REACTIVEalso mitigates the issueAdditional Context
spring-boot-starter-webtogether with reactive streams (like Flux) for SSE might be inherently problematic. We have simultaneously usedspring-boot-starter-webandspring-boot-starter-webfluxin our project, utilizingFlux<ServerSentEvent>endpoints without any issues. However, after upgrading our production environment from Spring Boot 3.0.5 to 3.2.2, we encountered this problem.Thank you for your attention to this matter. I am more than happy to provide further details or additional information if needed.
The text was updated successfully, but these errors were encountered: