List of source directories is not respected in regard of more extensive coverage report

[ndench]

Question	Answer
Infection version	0.16.3
Test Framework version	PHPUnit 8.5.4
PHP version	7.4.5
Platform	e.g. Ubuntu 16.04
Github Repo	-

In order to run infection with 100% msi, we break our business logic into a "functional core" which can be thoroughly unit tested and kill every mutant. Then we run infection only over the functional core by using the "--testsuite=unit" test framework option and by using source.directories in infection.json.dist. With version 0.15.* this works exactly as expected, with mutants only being applied to our functional core:

• 178 files mutated
• 940 mutants generated

With version 0.16.3, infection is mutating our entire codebase:

• 1578 files mutated
• 10211 mutants generated

phpunit.xml

<?xml version="1.0" encoding="UTF-8"?>

<!-- https://phpunit.de/manual/current/en/appendixes.configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/8.3/phpunit.xsd"
        backupGlobals="false"
        colors="true"
        bootstrap="config/bootstrap.php"
>
   <php>
       <ini name="error_reporting" value="-1" />
       <server name="APP_ENV" value="test" force="true" />
       <server name="SYMFONY_PHPUNIT_VERSION" value="8.3.5" />
       <server name="SHELL_VERBOSITY" value="-1" />
   </php>

   <testsuites>
       <testsuite name="All">
           <directory>tests/</directory>
       </testsuite>

       <!-- Different test types you might run separately -->
       <testsuite name="Unit">
           <directory>tests/Unit/</directory>
       </testsuite>
       <testsuite name="Integration">
           <directory>tests/Integration/</directory>
       </testsuite>
       <testsuite name="Message">
           <directory>tests/Message/</directory>
       </testsuite>
       <testsuite name="Contract">
           <directory>tests/Contract/</directory>
       </testsuite>
       <testsuite name="Model">
           <directory>tests/Model</directory>
       </testsuite>
       <testsuite name="Task">
           <directory>tests/Task</directory>
       </testsuite>

       <testsuite name="Functional">
           <directory>tests/Functional/</directory>
       </testsuite>

       <testsuite name="FunctionalAdmin">
           <directory>tests/Functional/Admin</directory>
       </testsuite>
       <testsuite name="FunctionalExport">
           <directory>tests/Functional/Export</directory>
       </testsuite>
       <testsuite name="FunctionalPlaybook">
           <directory>tests/Functional/Playbook</directory>
       </testsuite>
       <testsuite name="FunctionalUser">
           <directory>tests/Functional/User</directory>
       </testsuite>
       <testsuite name="FunctionalNegotiate">
           <directory>tests/Functional/Negotiate</directory>
       </testsuite>
       <testsuite name="FunctionalActivityStream">
           <directory>tests/Functional/ActivityStream</directory>
       </testsuite>
   </testsuites>

   <filter>
       <whitelist>
           <directory>./src/</directory>
       </whitelist>
   </filter>

   <listeners>
       <listener class="Hyra\Tests\Resources\TestListener\TestTidierListener" />
   </listeners>
</phpunit>

infection.json.dist

{
 "timeout": 30,
 "source": {
   "directories": [
     "src/Security",
     "src/*/Core",
     "src/*/Action",
     "src/Integrations/*/Action",
     "src/Algorithms/Derrida/Butler/Core"
   ],
   "excludes": [
     "Migrations"
   ]
 },
 "testFrameworkOptions": "--testsuite=Unit",
 "mutators": {
   "@default": true,
   "IdenticalEqual" : false,
   "NotIdenticalNotEqual": false,
   "MBString": false
 },
 "logs": {
   "text": "build\/infection\/log.txt",
   "debug": "build\/infection\/debug.txt",
   "summary": "build\/infection\/summary.txt"
 }
}

I tried stepping through with xdebug to track down the issue but got lost as so much has changed from 0.15 to 0.16. Happy to keep debugging if someone can help point me in the right direction?

[sanmai]

That's a BC break in this version: it always mutates "covered" files unless you exclude them with a filter or from coverage altogether.

The easiest way to do the latter is to configure phpunit.xml in include only the files you want to be mutated in the whitelist.

   <filter>
       <whitelist>
           <directory>./src/</directory>
       </whitelist>
   </filter>

There's also a blacklist of sorts if that's more convenient.

Just for the completeness of the report, would you mind showing your infection.json.dist?

[ndench]

@sanmai thanks for the information, I've updated the issue to include my infection.json.dist.

Just to clarify, are you saying we filter the coverage generation to only include the "functional core" and anything code that's not covered by unit tests will then not be mutated?

Is this an expected BC break? It's not specified in the changelog or the blog post.

[sanmai]

Unfortunately, it isn't explicitly specified. Nor thought over much (by me), for that matter.

Yes, it should work that if you filter the coverage to include only things you want to be mutated, then only included files will be mutated. I use this feature every time when I need to iterate quickly over a piece of code where I couldn't care about all other code.

You have to keep your configuration as it is, though.

[maks-rafalko]

I would say this is a bug, not an expected BC break between 0.15 and 0.16.

From our docs:

source.directories - array, contains all folders with source code you want to mutate. Can be ., but make sure to exclude vendor in this case.

So, the original purpose of this config setting is to limit all the folders inside the project to those that really need to be mutated. Now this is not the case.

Please see here, I've pushed a test that works for 0.15 but fails for 0.16.

• 0.15: bin/infection - works
• 0.15: bin/infection --only-covered - works
• 0.16: bin/infection - does not work <------ here is a bug
• 0.16: bin/infection --only-covered - works

We already have a similar bug and fixed it for 0.16.2 - #1238 - but that one was about source.excludes folders, not source.directories.

Explanation of the issue:

In 0.16, we have 2 sources of files to look at:

1. from SourceFileCollector - this is an iterable that respects both source.directories and source.excludes
2. from coverage report (PhpUnitXmlCoverageTraceProvider) - this is an iterable built from code coverage report. Before 0.16.2 it didn't respect source.excludes, now it does. But at the same time now it does not respect source.directories, and this is exactly the reason of the issue that @ndench is experiencing.

So, given the following project structure:

src
├── ToMutate
│   └── SourceClass.php
└── ToNotMutate
    └── ToNotMutateSourceClass.php

and the following infection.json config:

{
    "source": {
        "directories": [
            "src/ToMutate"
        ]
    }
}

we should not mutate src/NotToMutate folder as it was in 0.15.

Hot to fix?

I didn't have enough time to dig into the code yet, but I would say that the fix will be similar to source.excludes one's in #1238 - FilteredEnrichedTraceProvider::provideTraces() should return filtered out collection respecting infection.json config.

[sanmai]

I didn't have enough time to dig into the code yet, but I would say that the fix will be similar to source.excludes one's in #1238 - FilteredEnrichedTraceProvider::provideTraces() should return filtered out collection respecting infection.json config.

Otherwise put, we have to find every file inside those source directories, and check against this list for every file we get from the coverage report. If a covered file isn't there, then we should skip it. True it is annoying, but not impossible.

Files in source directories will act as a whitelist in this scheme.

[maks-rafalko]

Otherwise put, we have to find every file inside those source directories, and check against this list for every file we get from the coverage report. If a covered file isn't there, then we should skip it.

I see it exactly the same :)

[theofidry]

I think another usage in the wild is https://github.com/nelmio/alice, it currently does not work in 0.16 due to this (if you want to try it run make tm)

[sanmai]

Excludes should have been working, are they not?

If not, that's a slightly different issue.

[theofidry]

Can be, I did not have the chance to investigate yet to know if it’s exactly that issue or another edge case

…

On Fri 15 May 2020 at 11:20, Alexey Kopytko ***@***.***> wrote: Excludes should have been working, are they not? If not, that's a slightly different issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1264 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHPVANIYRI5D2IELCMYOCTRRUCOVANCNFSM4MSKIN6A> .

[maks-rafalko]

Released https://github.com/infection/infection/releases/tag/0.16.4

Thank you for reporting and waiting for the fix ;)