[KSP] Performance regression vs KAPT

[trevjonez]

While running tests using dagger + dagger-android (2.49) on KSP (1.9.20-1.0.14) I found a huge performance regression. In order to get timing info about dagger/ksp/kapt I adjusted the kotlin strategy to be in-process which also slows things down greatly. These builds were ran with --rerun-tasks

So that asserts that things do compile, but that we are still ~3x slower with ksp using the default daemon strategy.

Then further when running with in-process in an attempt to get trace data on dagger/ksp itself we require much larger heap sizes to avoid OOM plus it pushes us to MUCH longer build times making a full build as the test bed not really feasible. Even weirder I don't know how to explain one build taking an hour then the next with the exact same params taking 3.

To compensate for that slowdown the scenario I am submitting data from a much smaller subproject where we still see a significant % of slowdown.

the scenarios ran are:

assembleLibKapt {
    tasks = [":benefitsComponents:kaptDebugKotlin", "--rerun"]
    gradle-args = ["-Dscan.tag.KspDagger", "-Dscan.tag.assembleKapt", "-PuseKsp=false"]
    run-using = tooling-api
    daemon = warm
    warm-ups = 1
    iterations = 2
}

assembleLibKsp {
    tasks = [":benefitsComponents:kspDebugKotlin", "--rerun"]
    gradle-args = ["-Dscan.tag.KspDagger", "-Dscan.tag.assembleKsp", "-PuseKsp=true"]
    run-using = tooling-api
    daemon = warm
    warm-ups = 1
    iterations = 2
}

profiler invocation used is: gradle-profiler --benchmark --profile async-profiler-all --scenario-file bench.scenarios assembleLibKapt assembleLibKsp

flame charts suggest to me that the time use is dominated by KSP calls for things like getDeclaredFields, getDeclaredType, getElementsAnnotatedWith, etc.

Flame chart SVGs via zip so that github doesn't mangle the files and break the interactive nature of them.
assembleLibKsp-8.5-cpu-simplified-flames.zip
assembleLibKapt-8.5-cpu-simplified-flames.zip

This flame chart is from one of the builds that took over an hour which exhibits the same pattern as what is seen in the others targeting the smaller library.
fullProject-assembleKsp-8.5-cpu-simplified-flames.zip

It isn't immediately clear to me if this is an issue with how dagger is using KSP or just an issue in KSP itself?

[bcorso]

Thanks for posting your findings @trevjonez!

I can't say I'm too surprised by this since we've had years to optimize for Javac but haven't really done any optimization for KSP yet (so far we've only been focused on correctness).

I can take a look through the flame charts and see if anything sticks out.

[bcorso]

One thing I noticed is that the KAPT flame chart looks like there's incremental processing going on (IncrementalProcessor.process(...)).

It might help to first compare both of them using a clean build with no incremental processing. That way we can rule out if the regression only occurs for incremental processing or also in clean builds.

[trevjonez]

I am fairly certain with the --rurun flag these should be effectively the same as a clean run of the ksp/kapt tasks. I did it that way to try and get the flame chart to have as little extra noise as possible given the tracing is picking up everything in gradle too.

That said I'll play with the scenarios a bit more tomorrow and setup clean + incremental just to be sure.

[trevjonez]

I still owe you some repro projects but I did end up rerunning my out of process benchmarks with the latest KSP (1.0.16) and the time is at least consistent now so the time balloon on 1.9.21-1.0.15 was caused by the memory leak in KSP.

[bcorso]

@trevjonez just to clarify, you're seeing the regression in KSP build times are gone with KSP 1.0.16, meaning it's about the same as your build times with Javac now?

[trevjonez]

No it still is a big regression from KAPT. but initially it would take like 15 minutes on a clean KSP build then second run on hot daemons would be 30+ and usually OOM crash out before a third run could finish. It was an obvious indication of a memory leak but someone else reported it to KSP and they fixed leak that so now we should be able to get more accurate timing info for the dagger processor optimization.

[bcorso]

Okay, thanks for the clarification on that.

[ritesh-singh]

Any update? I recently upgraded to KSP from KAPT and noticed similar performance degradation.
Project isn't using dagger-android

[bcorso]

@ritesh-singh can you pin point where the performance regression is in your build or provide a repro of the regression you are seeing?

We're still waiting on more information that will help identify the regression.

[trevjonez]

I'd like to get a public reproduction project setup but I need some sort of medium to large public android project ideally not using hilt so we can focus in on only the main dagger ksp processor for now.

Additionally it looks like KSP2 preview runs in the gradle daemon which I suspect will make profiling easier too since it won't require reconfiguring every kotlin task as in-process.

[trevjonez]

Ok here is a project/branch that is setup to quickly hit both KSP/KAPT.
https://github.com/trevjonez/santa-tracker-android/tree/tjones/ksp-profiling

gradle-profiler --benchmark --profile async-profiler-all --scenario-file bench.scenarios assembleKapt assembleKsp

Initial run via kotlin daemon has me worried it isn't a big enough project to show the issue?

Running in-process makes ksp slower overall? Bizarre.

Though can it even be trusted with the change of daemon/in-process?

Regardless, hopefully having a project to test against will be helpful for someone better at tool profiling than myself.

[trevjonez]

I am going to plumb this into my build scan setup so I can see the task level details of where time is spent when using the kotlin daemon.

[trevjonez]

It is big enough that KSP processing task runs longer than just the KAPT processing task stub generation excluded

[bcorso]

Thanks @trevjonez, I appreciate the time spent producing these.

Note that we also have some projects within google that show regressions with KSP, and even some particular targets that have significant regressions e.g. 20s with KAPT to 2min with KSP, which should be good places to start looking for obvious places of performance issues. However, we currently only have data at the Bazel action level (i.e. total compilation time) and we're working with the Google Kotlinc team to get better tooling support to be able to get full pprofs. Once we have that it should be much easier for us to identify where these performance issues are coming from. I'll keep you posted on this thread as we get more information.

[audkar]

I'll keep you posted on this thread as we get more information.

@bcorso please keep us updated with any findings you get. This is such a huge build speed regression that it's considered a blocker level. We had to roll back the KAPT to KSP migration due to this issue.

If there are any findings/workarounds, please share them. It would help us to adjust our technical roadmap.

[PhilGlass]

To add one more data point: Dagger is the only processor we have that still runs through kapt, and app:kaptGenerateStubsDebugKotlin + app:kaptDebugKotlin takes around 30-35 seconds. After switching Dagger from kapt to KSP app:kspDebugKotlin went from 15-20 seconds to around 28 minutes. The Gradle/Kotlin daemons don't appear to be under memory pressure in either case.

I'm not sure which characteristics of our project are interesting - we have ~1M lines of Kotlin according to cloc, ~450 Gradle modules, and we still use dagger-android extensively. Our generated component is ~55k lines long.

Let me know if there's some more useful/actionable information I can capture and share privately!

[ting-yuan]

A couple of optimizations in KSP will be available in the next release, hopefully by next week if not this week. They reduced the build time of a certain workload from 66s to 27s. I'm going to dive into the workload by @trevjonez and share what I find later.

[pettermahlen]

Are there any updates on when that next release will be availabe? Looks like no new releases have been made since https://github.com/google/dagger/releases/tag/dagger-2.51.1 in March.