Infection (?) marks covered code as not covered

[sanmai]

Question	Answer
Infection version	master
Test Framework version	PHPUnit 6.5.7
PHP version	PHP 7.0.27-1
Platform	Linux
Github Repo	sanmai/LazyLINQ

TLDR: fixed by #256

Steps to reproduce:

git clone https://github.com/sanmai/LazyLINQ 
cd LazyLINQ
composer install

Now run tests to verify 100% coverage. And run Infection.

$ php vendor/bin/phpunit

Summary:                   
  Classes: 100.00% (1/1)    
  Methods: 100.00% (43/43)  
  Lines:   100.00% (181/181)

$ php vendor/bin/infection --log-verbosity=2 --threads=$(nproc)

Processing source code files: 2/2
Creating mutated files and processes: 151/151
.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out

........SS............S.....S.....................   ( 50 / 151)
.......E....S..SS..E..............S...............   (100 / 151)
S......S..........S...S...........................   (150 / 151)
.                                                    (151 / 151)

151 mutations were generated:
     137 mutants were killed
      12 mutants were not covered by tests
       0 covered mutants were not detected
       2 errors were encountered
       0 time outs were encountered

Metrics:
         Mutation Score Indicator (MSI): 92%
         Mutation Code Coverage: 92%
         Covered Code MSI: 100%

It'll show there were not covered mutants. Indeed we can see them in the log:

$ grep -A10 'Not covered mutants' infection-log.txt 
Not covered mutants:
====================

1) src/LazyLINQ/LazyCollection.php:265    [M] PublicVisibility

--- Original
+++ New
@@ @@
-    public function distinct(callable $comparer = null)
+    protected function distinct(callable $comparer = null)

But if we apply this mutant by hand:

sed -i 's/public function distinct/protected function distinct/g' src/LazyLINQ/LazyCollection.php

We'll get a failing test:

$ php vendor/bin/phpunit

There was 1 error:

1) LazyLINQ\LazyCollectionTest::testDistinct
Error: Call to protected method LazyLINQ\LazyCollection::distinct() from context 'LazyLINQ\LazyCollectionTest'

Clearly, this mutant isn't "not covered" by tests.

[sanmai]

Turns out it's not a problem of the Infection. It's a problem with clover.xml.

For example, there is a function on line 670, but there's not a word about it in the log:

<line num="657" type="stmt" count="1"/>
<line num="660" type="stmt" count="5"/>
<line num="672" type="method" name="anonymousFunction:672#2533" complexity="3" crap="3" count="1"/>
<line num="673" type="stmt" count="1"/>
<line num="674" type="stmt" count="1"/>

Like it never existed.

[tomtomau]

@sanmai if you do multiple runs do you get inconsistent results around the number not detected?

This is something we're facing at the moment on our project and I'm still trying to isolate the problem.

[sanmai]

Well, no. I'm getting consistent results in a sense that these methods are always missing.

[sanmai]

As an experiment I added an option to mutate whole files even if only mostly covered by tests. And with that option I get:

$ php ../infection/bin/infection --mostly-covered --threads=$(nproc)


151 mutations were generated:
     149 mutants were killed
       0 mutants were not covered by tests
       0 covered mutants were not detected
       2 errors were encountered
       0 time outs were encountered

Metrics:
         Mutation Score Indicator (MSI): 100%
         Mutation Code Coverage: 100%
         Covered Code MSI: 100%

This option is trivial to implement once PR #235 gets merged.

[pwm]

@sanmai Thanks for investigating this, I ran into the same issue today and was baffled by it. I'm a complete newcomer to infection and wondering how is --mostly-covered defined and how it relates to --only-covered?

[BackEndTea]

This has been fixed without the --mostly-covered option, instead it should be fixed by default in 0.9.0. Which is in beta release atm.

[pwm]

@BackEndTea ah, awesome, thanks for the info!

[whatthefrog]

Similar issue here, coverage-clover.xml file does not have an entry for <line num="318" /> :-(

seems that it would be related to the fact that the array value on that line is a boolean ???? I mean really ???

315:        $this->DB->insert('table', array(
316:            'field'           => $value,
317:            'field2'          => $value2,
318:            'field3'          => true,
...
        ));

      <line num="315" type="stmt" count="6"/>
      <line num="316" type="stmt" count="6"/>
      <line num="317" type="stmt" count="6"/>
-- missing ???
      <line num="319" type="stmt" count="6"/>
      <line num="320" type="stmt" count="6"/>
      <line num="323" type="stmt" count="6"/>

[BackEndTea]

@whatthefrog What version of infection are you running? #655 Should have fixed the possible issues with array coverage by checking if the parent array is covered. If you are using a version older than 0.13.0 i would suggest upgrading.

If the issue persists, please open an issue so we can investigate.

[whatthefrog]

I am using infection 0.9.3, I indeed realize it is quite old, did not pay attention at the beginning, as using php-7.0

0.10.0 requires php-7.1

ps: great powerful tool by the way