Introduce `--noop` option to run Noop mutators that does not change the source code (AST) #1465

maks-rafalko · 2021-01-07T12:47:03Z

This is needed for debugging purposes, to understand how and why Infection kills mutant even if no change is made for the given source code line.

If, using --noop options, we see killed mutants, it means that Infection environment somehow breaks the tests

Covered by tests
Doc PR: Add docs for --noop option. site#206

Ideally, when you run

bin/infection --noop

you should see the following picture:

Processing source code files: 407/407
.: killed, M: escaped, U: uncovered, E: fatal error, T: timed out, S: skipped

UMMMMMMMMMM                                          (11 / 11)

11 mutations were generated:
       0 mutants were killed
       1 mutants were not covered by tests
      10 covered mutants were not detected
       0 errors were encountered
       0 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 0%
         Mutation Code Coverage: 90%
         Covered Code MSI: 0%

so, mutants are either not covered by tests or escaped. It means tests are green for each noop mutator that just don't change the code.

If we get killed mutants using --noop option, it indicates the issues with the test suite:

probably tests suite can't work with random order
probably --threads=x makes tests suite broken (tests can't run in parallel)
something else

In order to further debug the issue, developer ca use --log-verbosite=all and text logger and analyze infection.log file to understand what was going wrong.

Interesting, running Infection for iself with 4 threads and --noop option uncovers issues with our tests suite as well:

bin/infection --noop --log-verbosity=all --threads=4

infection.log

60587   │ 76) /home/maksrafalko/apps/infection/src/TestFramework/PhpUnit/Config/Builder/MutationConfigBuilder.php:253    [M] Foreach_
60588   │ 
60589   │ --- Original
60590   │ +++ New
60591   │ 
60592   │ $ '/home/maksrafalko/apps/infection/vendor/phpunit/phpunit/phpunit' '--configuration' '/tmp/infection/phpunitConfiguration.aa6dcebeef88e4d0f2c093bac66eb6d1.infection.xml'
60593   │   PHPUnit 9.5.0 by Sebastian Bergmann and contributors.
60594   │   
60595   │   Runtime:       PHP 7.4.13
60596   │   Configuration: /tmp/infection/phpunitConfiguration.aa6dcebeef88e4d0f2c093bac66eb6d1.infection.xml
60597   │   Random Seed:   1610016633
60598   │   
60599   │   Testing 
60600   │   E
60601   │   
60602   │   Time: 00:00.004, Memory: 10.00 MB
60603   │   
60604   │   There was 1 error:
60605   │   
60606   │   1) Infection\Tests\TestFramework\PhpUnit\Config\Builder\MutationConfigBuilderTest::test_it_preserves_white_spaces_and_formatting
60607   │   fopen(/tmp/infection-test/FileSystemTestCase93939/interceptor.autoload.a1b2c3.infection.php): failed to open stream: No such file or directory
60608   │   
60609   │   /home/maksrafalko/apps/infection/vendor/infection/include-interceptor/src/IncludeInterceptor.php:111
60610   │   /home/maksrafalko/apps/infection/vendor/thecodingmachine/safe/generated/filesystem.php:339
60611   │   /home/maksrafalko/apps/infection/src/TestFramework/PhpUnit/Config/Builder/MutationConfigBuilder.php:91
60612   │   /home/maksrafalko/apps/infection/tests/phpunit/TestFramework/PhpUnit/Config/Builder/MutationConfigBuilderTest.php:122
60613   │   
60614   │   ERRORS!
60615   │   Tests: 1, Assertions: 0, Errors: 1.
60616   │ 
60617   │ 
60618   │ 77) /home/maksrafalko/apps/infection/src/TestFramework/PhpUnit/Config/Builder/MutationConfigBuilder.php:259    [M] MethodCallRemoval
60619   │ 
60620   │ --- Original
60621   │ +++ New
60622   │ 
60623   │ $ '/home/maksrafalko/apps/infection/vendor/phpunit/phpunit/phpunit' '--configuration' '/tmp/infection/phpunitConfiguration.651f3d9b82e0931c14af0ac02b8caf70.infection.xml'
60624   │   PHPUnit 9.5.0 by Sebastian Bergmann and contributors.
60625   │   
60626   │   Runtime:       PHP 7.4.13
60627   │   Configuration: /tmp/infection/phpunitConfiguration.651f3d9b82e0931c14af0ac02b8caf70.infection.xml
60628   │   Random Seed:   1610016633
60629   │   
60630   │   Testing 
60631   │   .......E
60632   │   
60633   │   Time: 00:00.008, Memory: 10.00 MB
60634   │   
60635   │   There was 1 error:
60636   │   
60637   │   1) Infection\Tests\TestFramework\PhpUnit\Config\Builder\MutationConfigBuilderTest::test_interceptor_is_included
60638   │   fopen(/tmp/infection-test/FileSystemTestCase68815/interceptor.autoload.a1b2c3.infection.php): failed to open stream: No such file or directory
60639   │   
60640   │   /home/maksrafalko/apps/infection/vendor/infection/include-interceptor/src/IncludeInterceptor.php:111
60641   │   /home/maksrafalko/apps/infection/vendor/thecodingmachine/safe/generated/filesystem.php:339
60642   │   /home/maksrafalko/apps/infection/src/TestFramework/PhpUnit/Config/Builder/MutationConfigBuilder.php:91
60643   │   /home/maksrafalko/apps/infection/tests/phpunit/TestFramework/PhpUnit/Config/Builder/MutationConfigBuilderTest.php:513
60644   │   
60645   │   ERRORS!
60646   │   Tests: 8, Assertions: 11, Errors: 1.

So, it looks like we have tests that can't work in parallel:

 fopen(/tmp/infection-test/FileSystemTestCase93939/interceptor.autoload.a1b2c3.infection.php): failed to open stream: No such file or directory

tests/phpunit/Mutator/NoopMutatorTest.php

sanmai

Great idea!

Below is just nitpicking.

sanmai · 2021-01-08T09:13:57Z

src/Mutator/MutatorFactory.php

-                $mutators[$mutator->getName()] = $mutator;
+            if ($ignored !== []) {
+                $mutator = new IgnoreMutator(new IgnoreConfig($ignored), $mutator);
+            }


It appears to me the mutator factory does slightly more than it should. There could be a thin decorator object, that would be super easy to test.

In the same light adding IgnoreMutator could a job of a similar object, yet we're already here so it might be easier to leave it as it is for time being.

TBH, I didn't get the idea here, or this file still looks ok for me. I left it as is for now.

…he source code (AST) This is needed for debugging purposes, to understand how and why Infection kills mutant even if no change is made for the given source code line. If, using `--noop` options, we see killed mutants, it means that Infection environment somehow breaks the tests

maks-rafalko · 2021-01-14T22:11:54Z

Great idea!

This one is from mbj, the author of ruby's Mutant lib. Agree - cool idea!

@infection/core here is an invite link to our discord channel https://discord.gg/ZUmyHTJ - there are many other channels of mutation testing libs for different languages, as well as a #general channel, where people discuss interesting things. Feel free to join!

maks-rafalko · 2021-01-14T22:21:46Z

@sanmai auto-merge worked here 👍

maks-rafalko added DX Developer Experience Feature labels Jan 7, 2021

maks-rafalko added this to the 0.21.0 milestone Jan 7, 2021

sanmai self-requested a review January 7, 2021 13:10

theofidry self-requested a review January 7, 2021 14:06

BackEndTea reviewed Jan 7, 2021

View reviewed changes

tests/phpunit/Mutator/NoopMutatorTest.php Outdated Show resolved Hide resolved

sanmai approved these changes Jan 8, 2021

View reviewed changes

theofidry approved these changes Jan 8, 2021

View reviewed changes

maks-rafalko added 3 commits January 15, 2021 01:04

Use intersection of types for mocks

a2add37

Remove extra dot in the option description

bee2c55

maks-rafalko mentioned this pull request Jan 14, 2021

Add docs for --noop option. infection/site#206

Merged

maks-rafalko force-pushed the noop-mutations branch from f3e0881 to bee2c55 Compare January 14, 2021 22:09

maks-rafalko enabled auto-merge (squash) January 14, 2021 22:12

maks-rafalko merged commit d133849 into master Jan 14, 2021

maks-rafalko deleted the noop-mutations branch January 14, 2021 22:20

renovate bot mentioned this pull request Jun 6, 2021

chore(deps): update dependency infection/infection to v0.28.1 shrikeh/scuba-diving#104

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `--noop` option to run Noop mutators that does not change the source code (AST) #1465

Introduce `--noop` option to run Noop mutators that does not change the source code (AST) #1465

maks-rafalko commented Jan 7, 2021 •

edited

sanmai left a comment •

edited

sanmai Jan 8, 2021

maks-rafalko Jan 14, 2021

maks-rafalko commented Jan 14, 2021 •

edited

maks-rafalko commented Jan 14, 2021

Introduce --noop option to run Noop mutators that does not change the source code (AST) #1465

Introduce --noop option to run Noop mutators that does not change the source code (AST) #1465

Conversation

maks-rafalko commented Jan 7, 2021 • edited

sanmai left a comment • edited

Choose a reason for hiding this comment

sanmai Jan 8, 2021

Choose a reason for hiding this comment

maks-rafalko Jan 14, 2021

Choose a reason for hiding this comment

maks-rafalko commented Jan 14, 2021 • edited

maks-rafalko commented Jan 14, 2021

Introduce `--noop` option to run Noop mutators that does not change the source code (AST) #1465

Introduce `--noop` option to run Noop mutators that does not change the source code (AST) #1465

maks-rafalko commented Jan 7, 2021 •

edited

sanmai left a comment •

edited

maks-rafalko commented Jan 14, 2021 •

edited