New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak on Micronaut HTTP server #10677
Comments
I don't know if it is of any help but I notice on an heap dump that it appears that in the |
cc @yawkat |
Just raw information, our whole application is broken due to this memory leak and customers and users are complaining, we try multiple workaround with no success 😭 |
@tchiotludo please give us some way to reproduce this. The form/multipart code is very complex and I don't see a starting point for debugging here |
@yawkat it's very problematic as I didn't succeed in reproducing the problem. That's why I added as much information as I could; users seem to not using form/multipart that much, and the memory leak points to I can ask if I can share the dump if you want, but as a memory dump can contain sensitive data, I need to check first with the user and share it privately. I can ask our users to provide more information but creating a reproducer seems to be very complex. |
you could try setting |
Thanks @yawkat we will test it, meanwhile I'll try my best to make a reproducer |
Hello. |
@yawkat we cannot use
|
@katoquro to check if it's the same issue you can try the following command to see if the same objects are accumulating:
|
@loicmathieu only works if you lower your max-request-size to something that fits in memory. |
@yawkat on user confirm that using the following configuration fixes the issue (or works around it): configuration:
micronaut:
server:
max-request-size: 1GB
netty:
server-type: full_content |
@yawkat with this configuration, file of more than 1GB lead to a request that seems to be "blocked forever" without an exception. So it's a workaround for some of our users but not a long term solution. Do you still need a reproducer (I'm working on it but still didn't make it reproduce the issue)? |
yes i still need a reproducer, either from you or from @katoquro . full_content buffers the full request and bypasses most places that use DelayedExecutionFlow. but it's not recommended for permanent use. |
@loicmathieu
|
@katoquro remove the grep and look at the most present objects in the histogram: And if it's a different leak, better to open a new issue ;) |
Can you analyze the memory dump and see what is being leaked? You can try https://eclipse.dev/mat/ |
@dstepanov @loicmathieu it's something out of heap, non-heap, etc... 🤔 |
@loicmathieu any luck on a reproducer? |
@graemerocher unfortunately, no, that's why I added as much information as I could have. |
@loicmathieu is there a way to run a Kestra locally to reproduce? |
@graemerocher yes, you can either run it from its repository or its docker image. But what very annoy me is that I cannot reproduce it myself, some users report the issue, I tried to setup Kestra locally with the same configuration and use it with the same scenario but didn't succeed in triggering the issue. I'll try to take some time this week to try to reproduce the issues I opened lately. |
ok thanks |
Expected Behavior
No memory leak.
Actual Behaviour
Heap histograms show a potential memory leak.
The following part of the heap histograms are relevant:
There are more than 50 millions of
io.micronaut.core.execution.DelayedExecutionFlowImpl$Map
in memory! And this heap histogram is on an application with very few request (so there cannot be 50 millions of file currently uploading).I think it may be related to this endpoint that bind one part with
@Part Publisher<StreamingFileUpload> files
then use a rawHttpRequest<?> inputs
as we have parts both as files and String attributes.The memory leak is new in Micronaut 4, in Micronaut 3 we bind multiple times the body, once in a part as today, and once in an
@Body Map<String, Object> inputMap
which is no more possible in Micronaut 4.Reference GitHUb discussion: https://github.com/micronaut-projects/micronaut-core/discussions/10662GitHub
Steps To Reproduce
No response
Environment Information
Example Application
No response
Version
4.3.4
The text was updated successfully, but these errors were encountered: