Escaped mutants in CI

[martinssipenko]

I'm facing a situation where some covered mutants are being reported as not detected. This issue only happens on CI, if I run Infection locally then they are marked as killed. If I SSH into my CI environment and manually apply the mutation to my code and then run PHPUnit manually the test cases do fail with ErrorException: Undefined property: stdClass::$soemthing.

Here is a Playground link that merely replicates my issue: https://infection-php.dev/r/4l0v. In my CI environment the DecrementInteger and IncrementtInteger mutations are being reported as escaped (not detected).

I'm a bit stuck with this one so I'd appreciate any help or suggestions that could help me troubleshoot the problem.

[sanmai]

What versions are you using? Do you have failOnWarning and failOnRisky set in your phpunit.xml?

Try using the latest version of Infection. I believe there was a similar issue fixed before.

[martinssipenko]

I'm using the latest Infection version 0.26.4 and I do have failOnRisky and failOnWarning set to true in my phpunit.xml.

[martinssipenko]

One thing I could add is that I generate my code coverage outside of Infection and use --skip-initial-tests and --coverage= options. Could this be a reason for it? The strange part is that it works as expected when I run in on my machine (same Infection version).

[maks-rafalko]

I would suggest to debug this issue, see here https://infection.github.io/guide/how-to.html#How-to-debug-Infection (infection --log-verbosity=all --debug)

In the generated infection.log file, there will be exact commands for each Mutant that Infection runs under the hood, so executing it probably will give you more information. Option --debug preserves all the generated files by Infection, so you can copy those command lines and execute on CI directly.

You can leave just one mutator that is escaped and mutate just 1 file to reduce the number of generated mutations.

[martinssipenko]

Thanks for the tips! I've found out that my CI environment does not have a default php.ini file and thus the error_reporting directives value is "":

php > var_dump(ini_get('error_reporting'));
string(0) ""

This is despite PHP documentation suggesting there is a default value that depends on the PHP version you are using (either E_ALL or E_ALL & ~E_NOTICE & ~E_STRICT & ~E_DEPRECATED prior to PHP 8.0.0).

Not having this defined lead to Notices not being converted into errors on my CI which lead to escaping mutants. When I tried running Infection by setting error_reporting to E_ALL (32767): php -d error_reporting=32767 ./vendor/bin/infection it works as expected.

I will now go and fix my CI env missing php.ini, but this makes me think if this can somehow be prevented by Infection itself? For example by adding a ini setting in Infections generated phpunit.xml that would set error_reporting to E_ALL. @maks-rafalko What do you think?

[sanmai]

It appears to me if you can file an issue with this suggestion, we'll see how it goes.

Here it is: #1660