Issue #12430: Resolve Pitest suppression for SeparatorWrapCheck

[arinmodi]

Issue #12430: Resolve Pitest suppression for SeparatorWrapCheck

Diff Regression config: https://gist.githubusercontent.com/arinmodi/6b46ea3817e1e421ee98eb01f1bed89d/raw/1185fc6ae40afee5804c8ef069c74c06a63cbaba/config.xml

[nrmancuso]

@arinmodi please generate check regression report by github action as specified at #12341, let's see if we can find some failure in SeparatorWrapCheck from your change.

[arinmodi]

@nrmancuso , I am confused which file uri i have to pass for regression configs

[nrmancuso]

@nrmancuso , I am confused which file uri i have to pass for regression configs

See example PRs in parent issue, you need to create a config in a gist and share raw link

[arinmodi]

Github, generate report

[arinmodi]

Did i correctly generate report ?

[github-actions]

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/2dbafe5_2022194115/reports/diff/index.html

[arinmodi]

There is no exception or differences in report, so Do i have to find a new test case ?

[nrmancuso]

There is no exception or differences in report, so Do i have to find a new test case ?

@arinmodi please analyze code and try to determine if we can find a test case, and share analysis here

[arinmodi]

Github, generate report

[github-actions]

https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/2dbafe5_2022091956/reports/diff/index.html

[arinmodi]

@nrmancuso, violations found in above report

[nrmancuso]

@nrmancuso, violations found in above report

Good, please restore code, and create new test case/ test input based on your findings and remove suppression.

[arinmodi]

Little confusion,

There is no difference in checkstyle link, is difference in other links considered or just differences of checkstyle link is considered ?

[nrmancuso]

@arinmodi the idea of the report is to find differences in check behavior on a bunch of projects; where we find the differences doesn't matter, we just need to add such code to our tests.

[arinmodi]

@arinmodi the idea of the report is to find differences in check behavior on a bunch of projects; where we find the differences doesn't matter, we just need to add such code to our tests.

Got it, That's what I think also, but it's better to ask

[arinmodi]

where can I find this file given as path ?

[nrmancuso]

@arinmodi see link on the left of the row to view source file, also see projects file at https://github.com/checkstyle/contribution/blob/master/checkstyle-tester/projects-to-test-on-for-github-action.properties to help you to locate these projects online or clone locally

[arinmodi]

@nrmancuso, Test File Added

[nrmancuso]

@arinmodi mutation is still surviving, please investigate . Make sure that config that produced violation in regression report is identical to the one in your new input file and that input file code matches.

[arinmodi]

I am getting these error when building the project

[nrmancuso]

I am getting these error when building the project

@arinmodi when asking for help, it is best to follow a pattern similar to the bug report template using the console:

1. Show the command you are running
2. Show the output of the command
3. Share what you are expecting
4. Share what is actually happening

Example:
I am expecting to build and verify checkstyle, but I get the following errors instead and project fails to build:

➜  ~ mvn clean verify
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.045 s
[INFO] Finished at: 2022-11-21T08:11:38-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/home/nick). Please verify you invoked Maven from the correct directory. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MissingProjectException
➜  ~ 

This allows others to hopefully reproduce your problem; screenshots are rarely helpful except in the case of website content or some other UI focused issue.

[arinmodi]

I am trying to run testCommaOnNewLine test, but I got the following error and checkstyle build failed with following error

Executing pre-compile tasks...
Running 'before' tasks
Checking sources
Copying resources... [checkstyle]
Parsing java… [checkstyle]
Checking dependencies… [checkstyle]
Dependency analysis found 0 affected files
Errors occurred while compiling module 'checkstyle'
javac 11.0.13 was used to compile java sources
Finished, saving caches…
Compilation failed: errors: 100; warnings: 4
Executing post-compile tasks...
Compilation failed: errors: 100; warnings: 4
Synchronizing output directories...
21-11-2022 19:33 - Build completed with 100 errors and 4 warnings in 5 sec, 807 ms

D:\Open-Source-Projects\checkstyle\src\main\java\com\puppycrawl\tools\checkstyle\api\JavadocTokenTypes.java:22:55
java: package com.puppycrawl.tools.checkstyle.grammar.javadoc does not exist

[nrmancuso]

@arinmodi it looks like you have cleaned but not generated antlr classes. Run mvn clean verify first, always run this first if you run into any compilation problems.

[arinmodi]

I am trying to analyze the pitest report but got following error.

Command :

groovy .ci/pitest-survival-check-xml.groovy "pitest-whitespace"

Error :

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
D:\Open-Source-Projects\checkstyle\.ci\pitest-survival-check-xml.groovy: 5: unable to resolve class groovy.util.slurpersupport.GPathResult
 @ line 5, column 1.
   import groovy.util.slurpersupport.GPathResult
   ^

D:\Open-Source-Projects\checkstyle\.ci\pitest-survival-check-xml.groovy: 6: unable to resolve class groovy.util.slurpersupport.NodeChildren
 @ line 6, column 1.
   import groovy.util.slurpersupport.NodeChildren
   ^

2 errors

[nrmancuso]

@arinmodi make sure that you are using the same version of groovy that we use in CI

[arinmodi]

@arinmodi make sure that you are using the same version of groovy that we use in CI

which version ci using ?

[nrmancuso]

@arinmodi examine pitest execution in CI, all details of installation are there

[nrmancuso]

Items:

[nrmancuso]

One more item:

[nrmancuso]

Ok to merge

[nrmancuso]

@arinmodi please look through other pitest suppression files for trim usage, see if we can find similar cases

[rnveach]

Not merging yet as I see no CI runs in my view.

[nrmancuso]

@arinmodi please rebase on latest master, let's see if we can get CI running again

[arinmodi]

@arinmodi please rebase on latest master, let's see if we can get CI running again

Done

[arinmodi]

By the way, In PR #12317 Kevin is not generating any regression report and still able to find the errors, how he is doing that ? I want to try the same

[rnveach]

He has dealt with other PRs with similar changes. Also looking at the code, sometimes it can just be seen on how to create a case to kill the mutation. Regression is needed if your unsure or production code needs to be changed to resolve the mutation.

[arinmodi]

He has dealt with other PRs with similar changes. Also looking at the code, sometimes it can just be seen on how to create a case to kill the mutation. Regression is needed if your unsure or production code needs to be changed to resolve the mutation.

so it requires experience ?

[nrmancuso]

@arinmodi it requires only careful analysis of code and consideration of all possible inputs; experience is helpful but not required :)