Checkstyle failing with bump pitest-maven from 1.8.1 to 1.9.0

[rnveach]

Identified in bump PR at checkstyle/checkstyle#11761 ,

Caused by: org.pitest.util.PitError: Coverage generation minion exited abnormally!
Please copy and paste the information and the complete stacktrace below when reporting an issue
VM : OpenJDK 64-Bit Server VM
Vendor : Eclipse Adoptium
Version : 11.0.15+10
Uptime : 47314
Input -> 
 1 : -Dclassworlds.conf=/usr/share/apache-maven-3.8.6/bin/m2.conf
 2 : -Dmaven.home=/usr/share/apache-maven-3.8.6
 3 : -Dlibrary.jansi.path=/usr/share/apache-maven-3.8.6/lib/jansi-native
 4 : -Dmaven.multiModuleProjectDirectory=/home/runner/work/checkstyle/checkstyle
BootClassPathSupported : false
    at org.pitest.coverage.execute.DefaultCoverageGenerator.gatherCoverageData (DefaultCoverageGenerator.java:148)
    at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage (DefaultCoverageGenerator.java:90)
    at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage (DefaultCoverageGenerator.java:52)
    at org.pitest.mutationtest.tooling.MutationCoverage.runAnalysis (MutationCoverage.java:148)
    at org.pitest.mutationtest.tooling.MutationCoverage.runReport (MutationCoverage.java:138)
    at org.pitest.mutationtest.tooling.EntryPoint.execute (EntryPoint.java:124)
    at org.pitest.mutationtest.tooling.EntryPoint.execute (EntryPoint.java:54)
    at org.pitest.maven.RunPitStrategy.execute (RunPitStrategy.java:35)
    at org.pitest.maven.AbstractPitMojo.analyse (AbstractPitMojo.java:504)
    at org.pitest.maven.AbstractPitMojo.execute (AbstractPitMojo.java:434)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

It is not clear what the issue is with this bump. Stack trace did not provide any details. From release notes it was said Maven Site Plugin should be a specific version but we are already on the recommended version.

Similar issue can be seen at checkstyle/checkstyle#11760 with isolated bump pitest-junit5-plugin from 0.16 to 1.0.0

[rnveach]

Running mvn -e --no-transfer-progress -P"pitest-annotation" clean test-compile org.pitest:pitest-maven:mutationCoverage -Dverbose=true (note verbose set) I see the following:

8:28:39 PM PIT >> FINE : MINION : java.lang.AbstractMethodError: Receiver class org.pitest.junit5.JUnit5TestUnitFinder does not define or inherit an implementation of the resolved method 'abstract java.util.List findTestUnits(java.lang.Class, org.pitest.testapi.TestUnitExecutionListener)' of interface org.pitest.testapi.TestUnitFinder.
8:28:39 PM PIT >> FINE : MINION : at org.pitest.mutationtest.config.PrioritisingTestUnitFinder.findTestUnits(PrioritisingTestUnitFinder.java:20)

[rnveach]

It seems that both pitest maven and pitest-junit5-plugin need to be bumped at the same time. When I did this locally, the build passed.

[rnveach]

Updated checkstyle PR, if it resolves then this issue this can be closed.

I feel though that pitest's error reporting could be a bit better. I wouldn't expect it to tell me both plugins need to be updated but just saying Coverage generation minion exited abnormally! is too vague and could atleast provide some details of the actual issue which was the abstract method errror.

[rnveach]

There is a surviving mutation at checkstyle/checkstyle#11761 (comment) which was being killed before. It seems like this might be regression before this new version as I can confirm locally tests fail with the mutation applied and it is the same tests pitest reported being killed by in the old version.

[hcoles]

@rnveach

Comment I accidentally left on the PR reproduced below.

---

I've taken a look at the surviving mutant. I managed to reproduce the issue once, but the mutant was killed on all subsequent runs.

Looking at the code, I'm reasonable confident that this is an existing test order dependency issue in the checkstyle suite, rather than an issue introduced in pitest 1.9.0.

The MESSAGE_QUERY_MAP map in XpathFileGeneratorAstFilter is static state, which will be modified by any test that executes accept.

Depending on which tests have been run in the JVM before, the state required to make XpathFileGeneratorAstFilterTest pass may already be in MESSAGE_QUERY_MAP before the mutant is introduced (possibly as a result of XpathFileGeneratorAstFilterTest itself having been previously run).

Although pitest aims to be fully deterministic, it cannot guarantee the same order of tests in every run. This is because the order is (partially) determined by test timings, which can be effected by external factors. Time outs due to (presumed) infinite loops also mean that pitest cannot gurantee that the same number of jvms launches occurs for each run.

Assuming this is indeed an order dependency due to static state, it can be mitigated by ensuring the finishLocalSetup method of XpathFileGeneratorAstFilter is run after a test modifies the static state, or before a test executes that relies upon it.

The simplest fix for this particular issue might be to add this to XpathFileGeneratorAstFilterTest

@BeforeEach
public void ensureCleanStartingState() {
    new XpathFileGeneratorAstFilter() {
        @Override
        public void finishLocalSetup() {
            super.finishLocalSetup();
        }
    }.finishLocalSetup();
}

It's unfortunately a little messy as it needs to overcome the access restrictions on finishLocalSetup

[rnveach]

Thanks @hcoles . I added the changes and CI is passing now. I did not think this was an order issue but what you said makes sense.

I think we are good now and this is fully resolved, but I wanted to point out my comment at #1045 (comment) as I did not see you mention it.

[rnveach]

Closing this as main issue is resolved