Understanding PIT testing

[ThatSneakyCoder]

Okay so I have studied PIT as much as I could and the following is what I have understood;

PIT mutates each line by changing some bits of code on that line for example it can simply remove the trim() in the line
return valueOf(AccessModifierOption.class, modifierName.trim().toUpperCase(Locale.ENGLISH)); and make it
return valueOf(AccessModifierOption.class, modifierName.toUpperCase(Locale.ENGLISH));
Well why is it doing that?

Simply to check whether the test suits that we have already written are able to catch that change in code or not. If all our tests pass without a problem even though trim() was removed then that means that our test suits are not good/enough so we need to write more tests to catch that change in code that PIT made (or kill that mutation).

Now some parts where I have a question:

Question1)

Q.1.1) What is the purpose of input files at all.

Q.1.2) How running tests on input files through verifyWithInlineConfigParser() method and showing respective error on respective lines in input files going to kill mutation in a totally different java file where the mutation was reported through PIT.

Q.1.3) Most importantly why is this way of testing that checkstyle is using better than writing pure JUnit tests?

Question2)

Q.2.1) Say if I make the input files then how do I know what code to write in it.

Q.2.2) How violations being reported on the lines in the input files going to fix the mutation?

Q.2.3) How do we know what config to mention at the top of the input files as the documentation is not present for AccessModifierOption as I think it is an enum thats why.

I do have a lot in my head but I can't visually graph what is happening between the files in the sense which file is communicating with which file for what reason. As I know about conventional pure JUnit test cases, it is difficult to understand this a little bit and I cannot proceed further until some help is provided. I need to connect what is happening.

@nrmancuso @romani

[nrmancuso]

1.1 The purpose of input files is to use Checkstyle in a realistic manner, as opposed to pure unit tests.
1.2 PIT mutates the check code, not input files. Input files are our test cases to hit different code in a given check
1.3 This way of testing (input file vs pure unit test) is better because it is much closer to real usage of checkstyle, think of our JUnit tests as closer to end to end testing rather than unit testing. Also, consider bug reporting. Users give us input code, not test code :)

2.1 If this question is directly related to pitest only, knowing what to put in input file is a matter of analyzing the check and the code you need to hit, then meeting such conditions (if they exist)
2.2 If we add new input code that will kill a mutation, this means that we have input now that will fail a test if code is mutated
2.3 Config used in file depends on the code you need to hit. Analyze the code to find a path that will allow you to reach the mutation and cause the check to fail in some way.

The "big idea" of pitest is that it mutates code in a check. If no test fails when the code is mutated, this mutation has "survived". In order to kill the mutation, we need to find input code (and create a test) that will fail if this mutation is present.

[nrmancuso]

In the specific case of the mutation you are working on in #12340 (comment), the recommendation to do #12340 (comment) is because this will cause the mutated code to fail this test, since we would not be trimming the tab character from the config property, causing checkstyle to fail.

[nrmancuso]

I do have a lot in my head but I can't visually graph what is happening between the files in the sense which file is communicating with which file for what reason.

Think of our JUnit tests like our bug report template, create config, create input, run checkstyle, get results. The tests just go one step further and verify the results against the expectation (the comments in the file) . This is a very common pattern in Checkstyle.

[romani]

Please keep in mind that sometimes we just need to remove extra code that actually never benefits behavior and simply takes CPU resources. We cleaned a lot of such dead code. Migration from pure Unit tests to more end to end helped us a lot to simply code. Pitest also helping a lot to catch such code.