LEAK: ByteBuf.release() was not called before it's garbage-collected. #1130
Labels
for/springframework
This belongs to the Spring Framework project
Comments
shikouchen
added
status/need-triage
|
@shikouchen There were fixes in Reactor Core (https://github.com/reactor/reactor-core/releases/tag/v3.3.5.RELEASE) related to leaks. Can you upgrade to Spring Boot 2.2.7 and test again? |
violetagg
added
for/user-attention
|
@violetagg I have the same issue. And I tried upgraded to Spring Boot 2.2.7 with openjdk11. The error still occurs and our JVM metrics show the direct buffer keeps growing over time. |
|
@lesongjia Will you be able to provide a reproducible example? |
|
@violetagg We see this in our testing env and prod. With Spring Boot 2.2.7 and also with advanced leak detection option. We have find more useful logs: This log aline with the problem could happen on the |
|
@lesongjia Can you also enable Reactor Netty logs? either |
|
@violetagg After I turn on the debug log for reactor netty, I saw the logs on one thread and one channel when this issue happened. The logs shows within 10 min, there were multiple request handled and I do think our 1 min timeout was triggered within this. And what is interested to me is the last few logs before the leak log says Also notice, there is a 4 mins gap between the Our WebClient config |
|
@lesongjia Is it possible to share also how you use |
|
@violetagg Sure, we have 2 usages, they are similar. |
|
We do use |
@lesongjia did you just discover this or was it working okay fine up to some previous version? It would help to have either something to run or to narrow down if possible when the issue might have appeared (if it was a regression). |
|
@rstoyanchev We just discovered this with an older version of Meanwhile, we also discovered another problem on our end with A quick question here that we want to get explained, what is the different on the |
When you use |
|
Thanks for the explanation @violetagg. Our fix looks good in non-prod env. Once all clear in our production env. I will follow up here with a summary. |
|
After we fixed One point to help others with the same issue, improper use of operators could cause system resource leak esp. when canceling subscriptions. So be careful with such operators. One suggestion here (not sure if it is possible) is to alert when improper use of an operator that actually may cause a leak or unexpected cancel of a subscription. Thanks for the help @violetagg! |
|
Will you be able to test with the recommended components here spring-projects/spring-framework#25216 (comment) |
|
Additional fix in Spring Framework was made spring-projects/spring-framework@21d0696 I'm closing this one for now, if the issue appears even with the new Spring Framework version we can reopen it. |
violetagg
added
for/springframework
|
Hi ... we saw the same "LEAK: ByteBuf.release() was not called before it's garbage-collected" issue...can you guys tell me to fix this....we are using same reactor netty and making webclient calls |
|
@vmadd03 Please create a new issue, specify the components versions and if possible provide a reproducible example. |
|
Issue #1254 was created for the comment above |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
The problem occurred when I used
Webclientto retrive data from the remote server by http requests.I checked memory usage of the system, it showed the memory usage was around 50%.
The remote server which received requests from other clients showed fine as well. So it might not be problems on the remote server side.
I use same spring-boot and netty version for two microservices. But only one side with much more requests has this error. Therefore, it might be related to the number of requests?
Actual Behavior
Steps to Reproduce
the example code written in Kotlin I used is as follows, But I can not reproduce the error. It just occasionally occurred.
Possible Solution
Your Environment
The text was updated successfully, but these errors were encountered: