What is io.micronaut.core.execution.DelayedExecutionFlowImpl for

[loicmathieu]

Hi,
We have heap usage issue in our application and making a heap histogram we can see a lot of instances of io.micronaut.core.execution.DelayedExecutionFlowImpl inner classes Map/OnErrorResume/FlatMap/OnComplete.

   1:      57108507     1370604168  io.micronaut.core.execution.DelayedExecutionFlowImpl$Map
   2:      57108505     1370604120  io.micronaut.core.execution.DelayedExecutionFlowImpl$OnErrorResume
   3:      38072339      913736136  io.micronaut.core.execution.DelayedExecutionFlowImpl$FlatMap
   4:      19036169      456868056  io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete

We didn't directly use these classes so I wonder from where they come as you can see, there are tens of millions of them in the heap.

A heap histogram from some minutes before didn't show any of them, but after the histogram with all those class the application goes into OOM.

[sdelamo]

ping @dstepanov @yawkat

[loicmathieu]

I opened a memory dump using Eclipse MAT and it detect it as a memory leak retaining 90% of the heap for 24 instances.

The stacktrace is:

io-executor-thread-43
  at io.micronaut.core.execution.DelayedExecutionFlowImpl.map(Ljava/util/function/Function;)Lio/micronaut/core/execution/ExecutionFlow; (DelayedExecutionFlowImpl.java:98)
  at io.micronaut.core.execution.DelayedExecutionFlowImpl$Map.apply(Lio/micronaut/core/execution/ExecutionFlow;)Lio/micronaut/core/execution/ExecutionFlow; (DelayedExecutionFlowImpl.java:262)
  at io.micronaut.core.execution.DelayedExecutionFlowImpl.work(Lio/micronaut/core/execution/DelayedExecutionFlowImpl$Step;Lio/micronaut/core/execution/ExecutionFlow;)V (DelayedExecutionFlowImpl.java:51)
  at io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(Lio/micronaut/core/execution/ExecutionFlow;)V (DelayedExecutionFlowImpl.java:64)
  at io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(Ljava/lang/Object;)V (DelayedExecutionFlowImpl.java:70)
  at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(Lio/micronaut/core/execution/DelayedExecutionFlow;Ljava/lang/Object;Ljava/lang/Throwable;)V (ExecutionFlow.java:94)
  at io.micronaut.core.execution.ExecutionFlow$$Lambda$2103+0x00007f5a059f88c0.accept(Ljava/lang/Object;Ljava/lang/Object;)V ()
  at io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(Ljava/util/function/BiConsumer;)V (ImperativeExecutionFlowImpl.java:132)
  at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(Ljava/util/function/Supplier;Lio/micronaut/core/execution/DelayedExecutionFlow;)V (ExecutionFlow.java:87)
  at io.micronaut.core.execution.ExecutionFlow$$Lambda$2076+0x00007f5a059e8ea8.run()V ()
  at io.micrometer.core.instrument.composite.CompositeTimer.record(Ljava/lang/Runnable;)V (CompositeTimer.java:141)
  at io.micrometer.core.instrument.Timer.lambda$wrap$0(Ljava/lang/Runnable;)V (Timer.java:193)
  at io.micrometer.core.instrument.Timer$$Lambda$928+0x00007f5a05385bc8.run()V ()
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V ()
  at java.util.concurrent.ThreadPoolExecutor$Worker.run()V ()
  at java.lang.Thread.run()V ()

If I understood correctly the memory dump, there is a single reference of io.micronaut.core.execution.DelayedExecutionFlowImpl$FlatMap which recursively contains all the other (tens of millions?) io.micronaut.core.execution.DelayedExecutionFlowImpl$FlatMap recursivel in the next field.

This definitely looks like a memory leak.

[yawkat]

it's mostly for form stuff. can you build a reproducer and report a bug

[loicmathieu]

I checked and we use multipart form data in a few places, including one that seems suspicious, as the reports of OOM we have are around this endpoint.
This is a very specific endpoint where we bind the body using @Part and inject the HttpRequest because we have some parts that are files and others that are plain attributes.

@ExecuteOn(TaskExecutors.IO)
    @Post(uri = "/{namespace}/{id}", consumes = MediaType.MULTIPART_FORM_DATA)
    public Execution create(
        @Parameter(description = "The inputs") HttpRequest<?> inputs, 
        @Parameter(description = "The inputs of type file") @Nullable @Part Publisher<StreamingFileUpload> files
    ) throws IOException {
        Map<String, Object> inputMap = (Map<String, Object>) inputs.getBody(Map.class).orElse(null);
        // do something with the files ..
    }

In Micronaut 3, we bound the body as an HashMap, but it's no longer possible due to #9694.

I already noticed that it causes issues as now, when we get the body as a Map, it also includes the file, which was not the case before as Micronaut 3 only binds in the body what was not bound elsewhere.

I try to use MultipartBody as we have parts that are files or attributes and it seems to be done for that purpose but failed as it is limited to 1GB, see #10666

So if the memory leak is potentially done due to the fact that we bind the request in a Part for file upload and in an HttpRequest object, fixing #10666 will allow us to switch to a more robust solution.

[loicmathieu]

I created #10677 to report it as an issue

[loicmathieu]

@yawkat I have the feeling that it is not linked to multipart formdata but with access log.
When I disable them, I cannot see any io.micronaut.core.execution.DelayedExecutionFlowImpl but when I enable them the class starts to be visible in the heap.

[loicmathieu]

We ask our user to disabled access log and it still happening so even if access log uses this class, it didn't leak it.

[loicmathieu]

I think I found the source of the leak, see #10677 (comment)