Error "Expected a value identical to 1. Got: 62" when using PHPUnit 10.2

[acelaya]

Question	Answer
Infection version	0.27.0
Test Framework version	PHPUnit 10.2.0
PHP version	8.2.6
Platform	Alpine
Github Repo	- https://github.com/shlinkio/shlink

When running infection 0.27 together with PHPUnit 10.2 (only in one of my test suites) I get the next error (with very verbose output -vvv):

Infection - PHP Mutation Testing Framework version 0.27.0

[notice] You are running Infection with PCOV enabled.
[warning] Skipping the initial test run can be very dangerous.
It is your responsibility to ensure the tests are in a passing state to begin.
If this is not done then mutations may report as caught when they are not.

In Assert.php line 2074:
                                               
  [Webmozart\Assert\InvalidArgumentException]  
  Expected a value identical to 1. Got: 62     
                                               

Exception trace:
  at /home/shlink/vendor/webmozart/assert/src/Assert.php:2074
 Webmozart\Assert\Assert::reportInvalidArgument() at /home/shlink/vendor/webmozart/assert/src/Assert.php:814
 Webmozart\Assert\Assert::same() at /home/shlink/vendor/infection/infection/src/TestFramework/Coverage/JUnit/JUnitTestFileDataProvider.php:84
 Infection\TestFramework\Coverage\JUnit\JUnitTestFileDataProvider->getTestFileInfo() at /home/shlink/vendor/infection/infection/src/TestFramework/Coverage/JUnit/MemoizedTestFileDataProvider.php:57
 Infection\TestFramework\Coverage\JUnit\MemoizedTestFileDataProvider->getTestFileInfo() at /home/shlink/vendor/infection/infection/src/TestFramework/Coverage/JUnit/JUnitTestExecutionInfoAdder.php:101
 Infection\TestFramework\Coverage\JUnit\JUnitTestExecutionInfoAdder->createCompleteTestLocation() at /home/shlink/vendor/infection/infection/src/TestFramework/Coverage/JUnit/JUnitTestExecutionInfoAdder.php:88
 Infection\TestFramework\Coverage\JUnit\JUnitTestExecutionInfoAdder->testExecutionInfoAdder() at /home/shlink/vendor/infection/infection/src/TestFramework/Coverage/UnionTraceProvider.php:59
 Infection\TestFramework\Coverage\UnionTraceProvider->provideTraces() at n/a:n/a
 iterator_to_array() at /home/shlink/vendor/infection/infection/src/IterableCounter.php:64
 Infection\IterableCounter::bufferAndCountIfNeeded() at /home/shlink/vendor/infection/infection/src/Mutation/MutationGenerator.php:85
 Infection\Mutation\MutationGenerator->generate() at n/a:n/a
 iterator_to_array() at /home/shlink/vendor/infection/infection/src/IterableCounter.php:64
 Infection\IterableCounter::bufferAndCountIfNeeded() at /home/shlink/vendor/infection/infection/src/Process/Runner/MutationTestingRunner.php:70
 Infection\Process\Runner\MutationTestingRunner->run() at /home/shlink/vendor/infection/infection/src/Engine.php:134
 Infection\Engine->runMutationAnalysis() at /home/shlink/vendor/infection/infection/src/Engine.php:74
 Infection\Engine->execute() at /home/shlink/vendor/infection/infection/src/Command/RunCommand.php:377
 Infection\Command\RunCommand->executeCommand() at /home/shlink/vendor/infection/infection/src/Command/BaseCommand.php:75
 Infection\Command\BaseCommand->execute() at /home/shlink/vendor/symfony/console/Command/Command.php:326
 Symfony\Component\Console\Command\Command->run() at /home/shlink/vendor/symfony/console/Application.php:1063
 Symfony\Component\Console\Application->doRunCommand() at /home/shlink/vendor/symfony/console/Application.php:320
 Symfony\Component\Console\Application->doRun() at /home/shlink/vendor/symfony/console/Application.php:174
 Symfony\Component\Console\Application->run() at /home/shlink/vendor/infection/infection/bin/infection:83
 include() at /home/shlink/vendor/bin/infection:120

I have reproduced this error when I run infection skipping initial tests and instructing it to use already generated coverage and junit log files.

> vendor/bin/phpunit --order-by=random -c phpunit-api.xml --testdox --colors=always --log-junit=build/coverage-api/junit.xml
> vendor/bin/infection --threads=max --only-covered --skip-initial-tests --coverage=build/coverage-api --min-msi=80 --configuration=infection-api.json5 -vvv

Based on the exception message, it seems PHPUnit 10.2 is generating junit log files with a different structure, but I cannot tell the difference on first sight. I can provide both files if needed.

Everything works properly with PHPUnit 10.1

I'm also trying to to let infection run the tests itself, to make sure it passes in that case. However, I'm finding an unrelated issue which is preventing this, which is that infection seems to be ignoring testFrameworkOptions from my infection-api.json5 file, and is using the wrong phpunit config (it is using default one, phpunit.xml.dist, which is used for a different test suite).

I have tried --test-framework-options flag and it seems to be ignored too.

I'll try to bypass this somehow and report that issue separately if needed.

[acelaya]

I have managed to work around the problem above with testFrameworkOptions, but I have realized for this specific test suite I cannot let infection run the tests and rely on the code coverage generated by PHPUnit.

The reason is that this is an E2E test suite, where the code under test is running on different processes, and the process running PHPUnit cannot collect code coverage.

What I do there is generate code coverage more or less manually with some helpers, and then pass that coverage to infection.

[kniziol]

I downgraded PHPUnit to 10.1, but it didn't solve the problem. It doesn't matter if we use the --skip-initial-tests option and already generated code coverage.

I get a little bit different error message:

Expected a value identical to 1. Got: 2

but the stack trace seems to be the same:

Exception trace:
  at vendor/webmozart/assert/src/Assert.php:2074
Webmozart\Assert\Assert::reportInvalidArgument() at vendor/webmozart/assert/src/Assert.php:817
Webmozart\Assert\Assert::same() at vendor/infection/infection/src/TestFramework/Coverage/JUnit/JUnitTestFileDataProvider.php:84
Infection\TestFramework\Coverage\JUnit\JUnitTestFileDataProvider->getTestFileInfo() at vendor/infection/infection/src/TestFramework/Coverage/JUnit/MemoizedTestFileDataProvider.php:57
Infection\TestFramework\Coverage\JUnit\MemoizedTestFileDataProvider->getTestFileInfo() at vendor/infection/infection/src/TestFramework/Coverage/JUnit/JUnitTestExecutionInfoAdder.php:101
// ...

[maks-rafalko]

@kniziol looks like you have a different issue described here #1833

[acelaya]

I think the issues could actually be related.

Since I'm generating the code coverage "manually", I might be duplicating tests in the report, because it's not easy to reliably map test methods with source code being run.

I will try to solve this and see if there's any difference.

[kniziol]

@kniziol looks like you have a different issue described here #1833

@maks-rafalko Correct. Created a single <testsuite> only (in phpunit.xml.dist) and the issue is gone.

<testsuites>
    <testsuite name="default">
        <directory>tests</directory>
    </testsuite>
</testsuites>

It's not a solution, its' a workaround.

[acelaya]

In the end, it is actually not related, but it gave me an idea.

As I mentioned some comments above, I generate the code coverage myself, because the tests and the source under test are running in different processes, making it impossible for PHPUnit to collect the coverage using the regular approach.

The trickier thing for me is determining the code coverage ID consistently. Usually, this ID is a combination of test class + test method name + data provider name.

In my case, I couldn't determine the last two reliably, so I went with the class only. That seemed to work in terms of determining percentage of covered code, and to use infection and other tools that require a code coverage report.

However, I have realized the junit report which is still generated by PHPUnit, references tests using class + method + data provider, resulting in a mismatch with how they are referenced in the code coverage report.

I still don't know if this has anything to do. I'm trying to find a way for my logic to use the three elements as the coverage ID, in order to see if it solves this issue.

[acelaya]

I managed to generate proper coverage IDs, but it didn't solve this.

I don't really see a difference in either coverage reports or junit logs from PHPUnit 10.1 to 10.2, so I'm not sure why this fails.

I'll continue debugging a bit further later today.

[acelaya]

Ok, it took me some time, but I finally found the difference in the junit.xml file which is causing this.

With PHPUnit 10.1, the junit.xml file has an empty name attribute in the topmost testsuite node:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="229" assertions="894" errors="0" failures="0" skipped="0" time="16.916708">
    ...
  </testsuite>
</testsuites>

While in PHPUnit 10.2, it has a value containing the path to PHPUnit's config file.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
-  <testsuite name="" tests="229" assertions="894" errors="0" failures="0" skipped="0" time="16.916708">
+  <testsuite name="/home/shlink/phpunit-api.xml" tests="229" assertions="894" errors="0" failures="0" skipped="0" time="16.916708">
    ...
  </testsuite>
</testsuites>

If I manually edit that file setting name="", then I can run infection successfully afterwards.

I'm not sure what would be the course of action here though. In other test suites this doesn't seem to be a problem.

[acelaya]

I have tracked down the difference in PHPUnit's code.

In v10.1, this call was empty, which results in an empty name for the topmost test suite: https://github.com/sebastianbergmann/phpunit/blob/10.1.0/src/TextUI/Configuration/Xml/TestSuiteMapper.php#L42

In v10.2, they pass the config file path: https://github.com/sebastianbergmann/phpunit/blob/10.2.0/src/TextUI/Configuration/Xml/TestSuiteMapper.php#L44

[acelaya]

Ok, I found the reason for this, and it was on me.

I mentioned some comments ago that I generate the code coverage report myself for this particular test suite, and that I had some trouble defining the right coverage IDs.

Well, I found a few cases in which the coverage ID was not set at all, generating empty entries.

It seems that infection was cross-referencing those with the parent <testcase> with name="" from the junit file, masking this.

As soon as PHPUnit 10.2 stopped generating that empty name, infection was not able to cross reference those with anything, resulting in this error.

It's good, because once I have fixed this, I've found the amount of killed mutants I have is actually higher. This behavior was throwing false negatives.

Anyway, I'll close this now 😅