Memory leak in ktlintFormat task

[BraisGabin]

Probably related with #447

I'm running git rebase -i develop --exec "./gradlew ktlintFormat" on a branch with 40 commits and I'm getting this error on my local machine (macOS):

> A failure occurred while executing org.jlleitschuh.gradle.ktlint.worker.KtLintWorkAction
   > Java heap space

It seems like a leak to me. The first 5 coomits ok, then the next failed. I reran the task manually on that commit and it worked. But then the 8th commit failed. and the pattern continued until the commit 15th. I can't run the task any more. It fails every single time with the error I pasted above. As soon as I killed the daemon ./gradlew --stop it worked.

I'm using v10.1.0

[Tapchicoma]

Yes, for some unknown reason (Kotlin update) Ktlint 0.41.0 uses more memory then before...

[Tapchicoma]

@BraisGabin do you see similar behaviour with KtLint 0.42.1?

[BraisGabin]

Yes, same behaviour. I'm using this to reproduce it:

for run in {1..20}; do ./gradlew ktlintFormat --rerun-tasks; done

And after some runs I get this type of outputs:

Expiring Daemon because JVM heap space is exhausted
Daemon will be stopped at the end of the build after running out of JVM memory
Expiring Daemon because JVM heap space is exhausted
...
Expiring Daemon because JVM heap space is exhausted
Expiring Daemon because JVM heap space is exhausted
Expiring Daemon because JVM heap space is exhausted
Expiring Daemon because JVM heap space is exhausted

[gabrielittner]

I'm also still seeing this

[svinouze]

I know it doesn't solve the issue, but you can still increase the heap size as described in the README since the default heap is 256 mb.

tasks.withType<org.jlleitschuh.gradle.ktlint.tasks.BaseKtLintCheckTask> {
    workerMaxHeapSize.set("512m")
}

[jeremymailen]

I believe there's a memory leak in the underlying ktlint library and then the gradle daemon's memory is eventually exhausted.
You notice it more/faster in the format task because it runs over all files rather than incrementally only on changed files like the lint task does.
I was trying to spot where it could be based on commits in ktlint. It's hard to tell. The kotlinPsiFileFactory is a resource that requires careful management though, so it could be something to look at.

[davidburstromspotify]

I've just started investigating that same issue. For every invocation, there are a number of org.jetbrains.kotlin.com.intellij.mock.MockProjects added to strongly referenced memory, as well as com.pinterest.ktlint.core.internal.KotlinPsiFileFactory$FormatPomModels.

[BraisGabin]

After updating to Gradle 7 I see this issue more often, now I can reproduc it in CI too (randomly). I assume that this increase is related with #457 because now more tasks are runned.

Is there something where I could help? I'm a bit lost here.

[Tapchicoma]

@BraisGabin probably sharing heap dump may help. I still suspect problem in KtLint -> Internal Kotlin version

[davidburstromspotify]

@Tapchicoma I cannot share the whole heap dump, but did you find anything based off of my screenshot?

[jeremymailen]

I had been scouting for possible commits in ktlint between 0.40.0 and 0.41.0 that may have been the causes.
There are two I suspect might be relevant, but I can't be certain:
https://github.com/pinterest/ktlint/pull/1088/files
pinterest/ktlint@1bf0d2e

Possibly a closure approach can make sure the factory is always released (with a finally clause wrapped around the block to execute:

withPsiFileFactory { psiFileFactory ->
  // some linting code
}

The other possibility is that the reference counting approach isn't accounting for all situations?

[gabrielle-cozart]

I have run into the same issue as well. Even changing the following to 4096m is still failing. I had to remove my ktlint pre-push hook to be able to push to my project.

tasks.withType<org.jlleitschuh.gradle.ktlint.tasks.BaseKtLintCheckTask> {
    workerMaxHeapSize.set("512m")
}

[breispeloton]

Just an FYI for anyone seeing this that this may have been fixed in ktlint 0.43.0. See pinterest/ktlint#1216 and the linked kotlinter-gradle issue.

[gabrielle-cozart]

@breispeloton Thanks! I've updated our plugin and hopefully it will be resolved.

[BraisGabin]

I can't reproduce this issue anymore. @gabrielittner if you can confirm it too I can close the issue. @breispeloton thanks for the heads up!

[flgubler-ergon]

Can also confirm: Explicitly setting the ktlint-version to 0.43.0 seems to have solved the problem for me as well.

ktlint {
  version = "0.43.0"
}

[gabrielle-cozart]

Okay, so I've been running with version 0.43.0 since this was last updated, and I just now started seeing this issue again.

> Task :app:runKtlintFormatOverMainSourceSet FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:runKtlintFormatOverMainSourceSet'.
> A failure occurred while executing org.jlleitschuh.gradle.ktlint.worker.KtLintWorkAction
   > Java heap space

[JLLeitschuh]

Can you open a new issue? Also, can you provide additional information?

For example, a build scan link would be helpful, version of the plugin you are using, Gradle version, Ktlint version, ect...