All mutants escapes when using function-mocking and patchwork

[Loki3000]

Question	Answer
Infection version	0.26.19
Test Framework version	PHPUnit 9.6.3
PHP version	8.1.2
Platform	Linux Mint
Github Repo	https://github.com/Loki3000/infection_issue

Wrong test results when using function-mocker (https://github.com/lucatume/function-mocker). All mutants are escapes.

If using function mocking all mutants escapes.

Simple class with a one method and one external function:

<?php
class myClass
{
    private $status='test';
    public function checkStatus()
    {
        return $this->status==myFunc();
    }
}

function myFunc()
{
    return 'FOO';
}

Test class with mocking myFunc:

<?php
use PHPUnit\Framework\TestCase;
use tad\FunctionMocker\FunctionMocker;
final class myClassTest extends TestCase
{
    protected function setUp(): void
    {
        FunctionMocker::setUp();
        parent::setUp();
    }
    protected function tearDown(): void
    {
        parent::tearDown();
        FunctionMocker::tearDown();
    }
    public function testCheckStatus(): void
    {
         //now myFunc always returns 'test'
        FunctionMocker::replace('myFunc', 'test');
        $class=new myClass();
        $this->assertTrue($class->checkStatus());
    }
}

PHPUnit returns no errors

PHPUnit 9.6.3 by Sebastian Bergmann and contributors.
Random Seed:   1676196100
.                                                                   1 / 1 (100%)
Time: 00:00.022, Memory: 16.00 MB
OK (1 test, 1 assertion)

trying Infection

1 mutations were generated:
       0 mutants were killed
       0 mutants were configured to be ignored
       0 mutants were not covered by tests
       1 covered mutants were not detected
       0 errors were encountered
       0 syntax errors were encountered
       0 time outs were encountered
       0 mutants required more time than configured

one mutant generated and escaped. Lets check log

Escaped mutants:
================
1) /var/infection_test/src/myClass.php:9    [M] Equal
--- Original
+++ New
@@ @@
     private $status = 'test';
     public function checkStatus()
     {
-        return $this->status == myFunc();
+        return $this->status != myFunc();
     }
 }

Try to make same changes in code and run phpunit again:

PHPUnit 9.6.3 by Sebastian Bergmann and contributors.
Random Seed:   1676196349
F                                                                   1 / 1 (100%)
Time: 00:00.023, Memory: 18.00 MB
There was 1 failure:
1) myClassTest::testCheckStatus
Failed asserting that false is true.
/var/infection_test/tests/myClassTest.php:24
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

Configs, test files and logs is in https://github.com/Loki3000/infection_issue

[maks-rafalko]

I'm pretty sure that function-mocker (and/or its underlying Patchwork2 lib) conflicts with Infection in place of intercepting files. Infection uses Stream Wrappers and particularily https://github.com/infection/include-interceptor to change the code in a runtime, so other stream wrapper break Infection.

We had similar issue with dg/bypass-final (spoiler: it was fixed on their side by respecting other registered stream wrappers, not replacing them):

• Probem when using Infection with bypass-finals #1275
• Using Infection library fails because of another stream wrapper dg/bypass-finals#9 (comment)

However, after a quick look I don't see that Patchwork2 uses Stream Wrapper, seems like there is another black magic. Anyway, personally I'm not using this lib and not going to, so if you want to understand the reason - I'm afraid you are the best person who can do it.

Probably patchwork's maintainers can give some ideas where to look at and what can be the issue? @antecedent

[antecedent]

Patchwork does use stream wrappers; specifically, it does so from Patchwork\CodeManipulation\Stream. I will try to come up with some solution along the lines of the analogous issue that you linked, in a couple of days or so.

[sanmai]

You can see here how it was fixed in the other library:

dg/bypass-finals@97abcd8

In a nutshell, stream_get_meta_data() exposes the name of a previous wrapper class.

[antecedent]

Many thanks to everyone who provided helpful hints! I think I might be done with the fix. @Loki3000, could you try again with Patchwork 2.1.22 2.1.23 and see if everything is okay now?

[Loki3000]

@antecedent unfortunately TypeError appears after update:

Infection - PHP Mutation Testing Framework version 0.26.19

PHP Fatal error:  Uncaught TypeError: closedir(): supplied resource is not a valid Directory resource in /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php:124
Stack trace:
#0 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(124): closedir()
#1 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(220): Patchwork\CodeManipulation\Stream::Patchwork\CodeManipulation\{closure}()
#2 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(132): Patchwork\CodeManipulation\Stream::bypass()
#3 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(253): Patchwork\CodeManipulation\Stream::alternate()
#4 [internal function]: Patchwork\CodeManipulation\Stream->dir_closedir()
#5 /home/loki/.config/composer/vendor/symfony/finder/Iterator/RecursiveDirectoryIterator.php(137): closedir()
#6 /home/loki/.config/composer/vendor/symfony/finder/Iterator/RecursiveDirectoryIterator.php(119): Symfony\Component\Finder\Iterator\RecursiveDirectoryIterator->isRewindable()
#7 [internal function]: Symfony\Component\Finder\Iterator\RecursiveDirectoryIterator->rewind()
#8 [internal function]: FilterIterator->rewind()
#9 [internal function]: FilterIterator->rewind()
#10 [internal function]: FilterIterator->rewind()
#11 /home/loki/.config/composer/vendor/infection/infection/src/TestFramework/Coverage/BufferedSourceFileFilter.php(76): FilterIterator->rewind()
#12 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(240): Infection\TestFramework\Coverage\BufferedSourceFileFilter->__construct()
#13 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(1173): Infection\Container::Infection\{closure}()
#14 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(754): Infection\Container->get()
#15 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(231): Infection\Container->getBufferedSourceFileFilter()
#16 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(1173): Infection\Container::Infection\{closure}()
#17 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(739): Infection\Container->get()
#18 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(234): Infection\Container->getCoveredTraceProvider()
#19 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(1173): Infection\Container::Infection\{closure}()
#20 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(744): Infection\Container->get()
#21 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(520): Infection\Container->getUnionTraceProvider()
#22 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(1173): Infection\Container::Infection\{closure}()
#23 /home/loki/.config/composer/vendor/infection/infection/src/Container.php(1044): Infection\Container->get()
#24 /home/loki/.config/composer/vendor/infection/infection/src/Command/RunCommand.php(369): Infection\Container->getMutationGenerator()
#25 /home/loki/.config/composer/vendor/infection/infection/src/Command/BaseCommand.php(75): Infection\Command\RunCommand->executeCommand()
#26 /home/loki/.config/composer/vendor/symfony/console/Command/Command.php(312): Infection\Command\BaseCommand->execute()
#27 /home/loki/.config/composer/vendor/symfony/console/Application.php(1022): Symfony\Component\Console\Command\Command->run()
#28 /home/loki/.config/composer/vendor/symfony/console/Application.php(314): Symfony\Component\Console\Application->doRunCommand()
#29 /home/loki/.config/composer/vendor/symfony/console/Application.php(168): Symfony\Component\Console\Application->doRun()
#30 /home/loki/.config/composer/vendor/infection/infection/bin/infection(83): Symfony\Component\Console\Application->run()
#31 /home/loki/.config/composer/vendor/bin/infection(120): include('...')
#32 {main}
  thrown in /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php on line 124

[antecedent]

Sorry. I had left in a bug in the stream wrapper's opendir(). It should be fixed now. Please retry with 2.1.24.

[Loki3000]

@antecedent thanks for fix! But same error still remains in next string

Infection - PHP Mutation Testing Framework version 0.26.19


Running initial test suite...

PHPUnit version: 9.6.3

    8 [============================] 2 secs

Generate mutants...

Processing source code files:    0/0
.: killed, M: escaped, U: uncovered, E: fatal error, X: syntax error, T: timed out, S: skipped, I: ignored



0 mutations were generated:
       0 mutants were killed
       0 mutants were configured to be ignored
       0 mutants were not covered by tests
       0 covered mutants were not detected
       0 errors were encountered
       0 syntax errors were encountered
       0 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 0%
         Mutation Code Coverage: 0%
         Covered Code MSI: 0%

Generated Reports:
         - infection.log

Please note that some mutants will inevitably be harmless (i.e. false positives).
PHP Fatal error:  Uncaught TypeError: rmdir(): Argument #2 ($context) must be of type resource or null, int given in /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php:128
Stack trace:
#0 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(128): rmdir()
#1 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(220): Patchwork\CodeManipulation\Stream::Patchwork\CodeManipulation\{closure}()
#2 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(132): Patchwork\CodeManipulation\Stream::bypass()
#3 /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php(284): Patchwork\CodeManipulation\Stream::alternate()
#4 [internal function]: Patchwork\CodeManipulation\Stream->rmdir()
#5 /home/loki/.config/composer/vendor/symfony/filesystem/Filesystem.php(724): rmdir()
#6 /home/loki/.config/composer/vendor/symfony/filesystem/Filesystem.php(184): Symfony\Component\Filesystem\Filesystem::box()
#7 /home/loki/.config/composer/vendor/symfony/filesystem/Filesystem.php(150): Symfony\Component\Filesystem\Filesystem::doRemove()
#8 /home/loki/.config/composer/vendor/infection/infection/src/Event/Subscriber/CleanUpAfterMutationTestingFinishedSubscriber.php(58): Symfony\Component\Filesystem\Filesystem->remove()
#9 /home/loki/.config/composer/vendor/infection/infection/src/Event/EventDispatcher/SyncEventDispatcher.php(59): Infection\Event\Subscriber\CleanUpAfterMutationTestingFinishedSubscriber->onMutationTestingWasFinished()
#10 /home/loki/.config/composer/vendor/infection/infection/src/Process/Runner/MutationTestingRunner.php(123): Infection\Event\EventDispatcher\SyncEventDispatcher->dispatch()
#11 /home/loki/.config/composer/vendor/infection/infection/src/Engine.php(136): Infection\Process\Runner\MutationTestingRunner->run()
#12 /home/loki/.config/composer/vendor/infection/infection/src/Engine.php(74): Infection\Engine->runMutationAnalysis()
#13 /home/loki/.config/composer/vendor/infection/infection/src/Command/RunCommand.php(377): Infection\Engine->execute()
#14 /home/loki/.config/composer/vendor/infection/infection/src/Command/BaseCommand.php(75): Infection\Command\RunCommand->executeCommand()
#15 /home/loki/.config/composer/vendor/symfony/console/Command/Command.php(312): Infection\Command\BaseCommand->execute()
#16 /home/loki/.config/composer/vendor/symfony/console/Application.php(1022): Symfony\Component\Console\Command\Command->run()
#17 /home/loki/.config/composer/vendor/symfony/console/Application.php(314): Symfony\Component\Console\Application->doRunCommand()
#18 /home/loki/.config/composer/vendor/symfony/console/Application.php(168): Symfony\Component\Console\Application->doRun()
#19 /home/loki/.config/composer/vendor/infection/infection/bin/infection(83): Symfony\Component\Console\Application->run()
#20 /home/loki/.config/composer/vendor/bin/infection(120): include('...')
#21 {main}
  thrown in /home/loki/.config/composer/vendor/antecedent/patchwork/src/CodeManipulation/Stream.php on line 128

[antecedent]

For the time being, please replace your src/CodeManipulation/Stream.php with the one from Patchwork's master (raw source here) to see if the situation improves. If so, I will make another release. My apologies for the lengthy process.

[Loki3000]

Now works well. Thanks!

[maks-rafalko]

Thank you @antecedent & @Loki3000