How to determine cause for erroneous coverage? (reported: 0%, actual: 100%)

[milgner]

Describe the bug
I just noticed that the coverage report showed 0% coverage for some classes which there already are tests. The logs show that these tests are being executed successfully, too.

What is even more confusing is that other classes in the same sub-project and package correctly report 100% coverage.

This is a multi-project build. In the main build.gradle.kts:

dependencies {
    kover(project(":app"))
    kover(project(":api"))

    for (dir in File("${rootDir.absolutePath}/plugins").list { dir, name ->
        File("${dir.absolutePath}/$name").isDirectory && name != "build"
    }) {
        kover(project(":plugins:$dir"))
    }
}

allprojects {
  // ...
  apply(plugin = "org.jetbrains.kotlinx.kover")
  // ...
}

I suspect that this might be related to merging of coverage information from multiple projects but am not sure how. There are other projects that depend on this one and, as such, could confuse coverage tooling by having show up the class in their classpath (even though not used directly, only through their interface) without testing them.

In order to determine possible causes, we also made an experimental branch using only Jacoco and it correctly shows the coverage.

Errors
There are no errors: everything runs except that the reported data is incorrect.

Expected behavior
Test coverage should be >0% (100% for this class)

Ideally there

Reproducer
Unfortunately, open-sourcing the project will still require a couple of months before it's useful & ready 😬

Which is why the issue is titled as such: what should I do to determine the cause for the incorrect coverage myself?

Reports

io.redigital.spectre.spectre_host.SpectreHostRepositoryHibernateTest > can save spect:re hosts PASSED
io.redigital.spectre.spectre_host.SpectreHostRepositoryHibernateTest > maps connector information PASSED
io.redigital.spectre.spectre_host.SpectreHostRepositoryHibernateTest > can find spect:re hosts by id PASSED
io.redigital.spectre.spectre_host.SpectreHostRepositoryHibernateTest > can find spect:re hosts by hostname PASSED
io.redigital.spectre.spectre_host.SpectreHostRepositoryHibernateTest > can update spect:re hosts PASSED
io.redigital.spectre.spectre_host.SpectreHostRepositoryHibernateTest > can remove spect:re hosts PASSED

Environment

• Kover Gradle Plugin version: 0.7.5
• Gradle version: 8.5
• Kotlin project type: Kotlin/JVM
• Coverage Toolset (if customized in build script): no customization, only Kover
• Other context important for this bug: OpenJDK 17, Multi-project build

[shanshin]

Hi,
could you clarify when you run the koverHtmlReport task without specifying other Gradle command,SpectreHostRepositoryHibernateTest tests are executed?

[milgner]

I usually only executed ./gradlew clean check which would depend on the kover tasks and generate a report. Running ./gradlew check on the top-level project where most of the sub-projects don't have any changes causes Gradle to skip test execution for some and the resulting coverage reports are missing these tests.
Thus, running ./gradlew koverHtmlReport on an already-built project doesn't execute any tests and consequently only builds new reports where everything shows up as 0% coverage. (couldn't reproduce this behaviour just now, might have been a fluke)

I now tried running ./gradlew clean koverHtmlReport in the root project instead but the results are the same as running ./gradlew clean check.

One thing I noticed though: looking at the coverage reports inside the sub-project which contains the test, the coverage is shown as 0%, too, so the issue cannot be related to merging the various projects and sub-project structure.
In fact, I tried running ./gradlew clean :plugins:core.spectre:koverHtmlReport and, after running all the tests inside the project, it produced a report at plugins/core.spectre/build/reports/kover/html/index.html which also misses some of the tests being run.

One thing I forgot: we're using kotest and the DescribeSpec variant. Runinng on the JVM, kotest uses the JUnit platform. Not sure whether that's relevant, though: most of the tests are picked up just fine. I'll try to determine whether there are other tests missing from the reports and whether I can discern a pattern.

[shanshin]

Could you reproduce the problem by creating a single project to which the SpectreHostRepositoryHibernateTest test will be copied?

[shanshin]

@milgner, did you manage to create a producer?

[milgner]

@milgner, did you manage to create a producer?

Unfortunately, I did not... For the time being, we switched to Jacoco. Not sure when there might be time for a closer investigation. But we are planning to work on the build system in a little while, that could be an opportunity.