How would I go about figuring out why a mutant is being marked as escaped when I can't reproduce it manually?

[dkarlovi]

Let's say I get this mutant diff

36) /project/src/Bridge/Github/GithubProfile.php:228    [M] FalseValue

--- Original
+++ New
@@ @@
                 return true;
             }
         }
-        return false;
+        return true;
     }
 }

It's saying my tests don't detect the return value change. But, if I go into my code and change it like this, the tests in fact fail. Could this be due to coverage in some way?

[sanmai]

From the coverage report you should be able to find tests covering this line. Try running only them to see if they could catch the change.

[dkarlovi]

Yes, I'm trying to create a reproducer, this is why I'm asking for hints. 😀

[sanmai]

As an additional option, you could try git bisect to locate the bad commit behind this regression.

[maks-rafalko]

It can also be a bug in Infection, like this one #1523

As a general recommendation, please try to run Infection with --debug --log-verbosity=all and check text logger output.

https://infection.github.io/guide/how-to.html#How-to-debug-Infection

[dkarlovi]

OK, checking the debug log, it seems Infection doesn't run the tests which are covering the changes?

When I manually edit the line and run the entire test suite, I get 16 failures:

Failures

1) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges with data set #2
2) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges with data set #0
3) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges with data set #1
4) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges with data set #3
5) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository with data set #0
6) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository with data set #4
7) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository with data set #6
8) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository with data set #1
9) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository with data set #5
10) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository with data set #3
11) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository with data set #2
12) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges with data set #0
13) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges with data set #1
14) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges with data set #4
15) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges with data set #3
16) Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges with data set #2

Coverage for the line seems to agree:

Coverage

      <line nr="228">
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges#0"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges#2"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges#4"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges#1"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanPlanUserChanges#3"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanFetchTemplates#default"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanFetchTemplates#custom"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges#2"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges#3"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges#0"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreatePlanAttributeChanges#1"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository#5"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository#0"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository#3"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository#6"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository#1"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository#4"/>
        <covered by="Sigwin\Ariadne\Test\Bridge\Github\GithubProfileTest::testCanCreateRepository#2"/>
      </line>

The command Infection says it ran in the logs is this

'/project/vendor/bin/phpunit' '--configuration' '/tmp/infection/phpunitConfiguration.3696f64facb837718ce24de1658cff16.infection.xml' '--filter' '/GithubProfileTest\:\:testCanPlanUserChanges with data set "0"|GithubProfileTest\:\:testCanPlanUserChanges with data set "2"|GithubProfileTest\:\:testCanPlanUserChanges with data set "4"|GithubProfileTest\:\:testCanPlanUserChanges with data set "1"|GithubProfileTest\:\:testCanPlanUserChanges with data set "3"|GithubProfileTest\:\:testCanFetchTemplates with data set "default"|GithubProfileTest\:\:testCanFetchTemplates with data set "custom"|GithubProfileTest\:\:testCanCreatePlanAttributeChanges with data set "2"|GithubProfileTest\:\:testCanCreatePlanAttributeChanges with data set "3"|GithubProfileTest\:\:testCanCreatePlanAttributeChanges with data set "0"|GithubProfileTest\:\:testCanCreatePlanAttributeChanges with data set "1"|GithubProfileTest\:\:testCanCreateRepository with data set "5"|GithubProfileTest\:\:testCanCreateRepository with data set "0"|GithubProfileTest\:\:testCanCreateRepository with data set "3"|GithubProfileTest\:\:testCanCreateRepository with data set "6"|GithubProfileTest\:\:testCanCreateRepository with data set "1"|GithubProfileTest\:\:testCanCreateRepository with data set "4"|GithubProfileTest\:\:testCanCreateRepository with data set "2"/'

But, when I run the above command manually, it runs 2 tests, not the 16 it seems like it's trying to?

Runtime:       PHP 8.3.0
Configuration: /tmp/infection/phpunitConfiguration.3696f64facb837718ce24de1658cff16.infection.xml

..                                                                  2 / 2 (100%)

Time: 00:00.033, Memory: 6.00 MB

OK (2 tests, 10 assertions)

[dkarlovi]

Ah, the filter changed in PHPUnit 10.

This doesn't work:

'/project/vendor/bin/phpunit' '--filter' '/GithubProfileTest\:\:testCanPlanUserChanges with data set "0"/'

This works:

'/project/vendor/bin/phpunit' '--filter' '/GithubProfileTest\:\:testCanPlanUserChanges with data set #0/'

[dkarlovi]

Infection bug, opened #1899