Line CodeCoverage is not a reliable source of truth

[Slamdunk]

Question	Answer
Infection version	0.26.16
Test Framework version	PHPUnit 9.5.26
PHP version	8.1.12
Platform	Ubuntu

Hi, the line code coverage is not a reliable source of truth for asserting the final MSI score.

Source

final class Foo
{
    public function bar(): string
    {
        $var
            =
            'a'
            .
            'b'
        ;
        
        return $var;
    }
}

Test

/**
 * @covers \Foo
 */
final class FooTest extends TestCase
{
    public function testBar(): void
    {
        self::assertSame('ab', (new Foo())->bar());
    }
}

Code Coverage

WARNING: this is the correct code-coverage according to xDebug and PCOV drivers, NO bugs of phpunit/php-code-coverage are involved

Settings

phpunit.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
    bootstrap="vendor/autoload.php"
    colors="true"
    verbose="true"
    failOnRisky="true"
>
    <testsuites>
        <testsuite name="tests">
            <directory>tests</directory>
        </testsuite>
    </testsuites>

    <coverage processUncoveredFiles="true">
        <include>
            <directory suffix=".php">src</directory>
        </include>
    </coverage>
</phpunit>

infection.json

{
    "source": {
        "directories": ["src"]
    },
    "timeout": 3,
    "logs": {
        "text": "php://stdout"
    },
    "mutators": {
        "Concat": true
    },
    "minMsi": 100,
    "minCoveredMsi": 100
}

Output

$ vendor/bin/phpunit

PHPUnit 9.5.26 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.1.12
Configuration: /home/tessarotto/repos/test-infection/phpunit.xml.dist

.                                                                   1 / 1 (100%)

Time: 00:00.002, Memory: 8.00 MB

OK (1 test, 1 assertion)

$ vendor/bin/infection

Processing source code files: 1/1
.: killed, M: escaped, U: uncovered, E: fatal error, X: syntax error, T: timed out, S: skipped, I: ignored

U                                                    (1 / 1)

1 mutations were generated:
       0 mutants were killed
       0 mutants were configured to be ignored
       1 mutants were not covered by tests
       0 covered mutants were not detected
       0 errors were encountered
       0 syntax errors were encountered
       0 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 0%
         Mutation Code Coverage: 0%
         Covered Code MSI: 0%

Note: to see escaped mutants run Infection with "--show-mutations" or configure file loggers.

Please note that some mutants will inevitably be harmless (i.e. false positives).
Escaped mutants:
================

Timed Out mutants:
==================

Skipped mutants:
================

Not Covered mutants:
====================

1) /home/tessarotto/repos/test-infection/src/Foo.php:13    [M] Concat

--- Original
+++ New
@@ @@
 {
     public function bar() : string
     {
-        $var = 'a' . 'b';
+        $var = 'b' . 'a';
         return $var;
     }
 }

Time: 0s. Memory: 16.00MB

 [ERROR] The minimum required MSI percentage should be 100%, but actual is 0%. Improve your tests!

[Slamdunk]

Could be a possible solution to store information about CC into a [class][method] key store instead of a [line] one?

[sanmai]

There was a similar issue with arrays, fixed by #1027.

[theofidry]

this is the correct code-coverage according to xDebug and PCOV drivers, NO bugs of phpunit/php-code-coverage are involved

I'm likely talking more to myself than anything but I really wish CS would not change what is covered, i.e. that the coverage would be bound to the statement, not how the statement is presented.

[Slamdunk]

That would be ideal, but I think that ship sailed a long time ago for PHP

[Slamdunk]

With the land of sebastianbergmann/php-code-coverage#964 in php-code-coverage:9.2.21 Line CodeCoverage is now as following:

And so Infection is now happy:

Processing source code files: 1/1
.: killed, M: escaped, U: uncovered, E: fatal error, X: syntax error, T: timed out, S: skipped, I: ignored

.                                                    (1 / 1)

1 mutations were generated:
       1 mutants were killed
       0 mutants were configured to be ignored
       0 mutants were not covered by tests
       0 covered mutants were not detected
       0 errors were encountered
       0 syntax errors were encountered
       0 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 100%
         Mutation Code Coverage: 100%
         Covered Code MSI: 100%

Note: to see escaped mutants run Infection with "--show-mutations" or configure file loggers.

Please note that some mutants will inevitably be harmless (i.e. false positives).
Escaped mutants:
================

Timed Out mutants:
==================

Skipped mutants:
================

Not Covered mutants:
====================

Time: 0s. Memory: 16.00MB

🚀

[maks-rafalko]

Thank you for your work @Slamdunk!