Code Coverage does not work with PHPUnit 9.3 PHAR and Xdebug

[jrfnl]

Q	A
PHPUnit version	9.3.2
PHP version	7.4.6
Installation Method	PHAR
Xdebug version	2.9.6
OS	Windows

Summary

When attempting to create a code coverage report, the run exits nearly immediately with a Fatal error: Uncaught Error: Call to undefined function PHPUnit\xdebug_set_filter().

I was looking forward to exploring the new path/branch coverage. Unfortunately ran into this before even adding the new pathCoverage="true" config option.

Current behavior

> phpunit --disallow-test-output --coverage-html ./build/coverage-html/

PHPUnit 9.3.2 by Sebastian Bergmann and contributors.

Fatal error: Uncaught Error: Call to undefined function PHPUnit\xdebug_set_filter() in phar://path/to/phpunit-9.3.2.phar/php-code-coverage/Driver/Xdebug2Driver.php:63
Stack trace:
#0 phar://path/to/phpunit-9.3.2.phar/php-code-coverage/Driver/Driver.php(88): PHPUnit\SebastianBergmann\CodeCoverage\Driver\Xdebug2Driver->__construct(Object(PHPUnit\SebastianBergmann\CodeCoverage\Filter))
#1 phar://path/to/phpunit-9.3.2.phar/phpunit/TextUI/TestRunner.php(336): PHPUnit\SebastianBergmann\CodeCoverage\Driver\Driver::forLineCoverage(Object(PHPUnit\SebastianBergmann\CodeCoverage\Filter))
#2 phar://path/to/phpunit-9.3.2.phar/phpunit/TextUI/Command.php(102): PHPUnit\TextUI\TestRunner->run(Object(PHPUnit\Framework\TestSuite), Array, Array, true)
#3 phar://path/to/phpunit-9.3.2.phar/phpunit/TextUI/Command.php(71): PHPUnit\TextUI\Command->run(Array, true)
#4 path\to\phpunit-9.3.2.phar(1043): PHPUnit\TextUI\Command::main()
#5 {main}

Next PHPUnit\TextUI\Exception: Call to undefined function PHPUnit\xdebug_set_filter() in phar://path/to/phpunit-9.3.2.phar/phpunit/TextUI/Command.php on line 73

PHPUnit\TextUI\Exception: Call to undefined function PHPUnit\xdebug_set_filter() in phar://path/to/phpunit-9.3.2.phar/phpunit/TextUI/Command.php on line 73

Call Stack:
    0.0380    1089944   1. {main}() path\to\phpunit-9.3.2.phar:0
    0.6750   13514728   2. PHPUnit\TextUI\Command::main() path\to\phpunit-9.3.2.phar:1043

How to reproduce

Using the latest PHPUnit 9.3.x phar, run the phpunit --disallow-test-output --coverage-html ./build/coverage-html/ command on a project.

Relevant part of the phpunit.xml config file used for the run:

    <coverage includeUncoveredFiles="true" processUncoveredFiles="false">
        <include>
            <directory suffix=".php">./PHPCSUtils/</directory>
        </include>
        <report>
            <clover outputFile="build/logs/clover.xml"/>
        </report>
    </coverage>

Expected behavior

That the tests would be run and a code coverage report would be created.

[sebastianbergmann]

Probably also caused by humbug/php-scoper#399.

[jrfnl]

@sebastianbergmann Please let me know if you'd like me to test a new phar before release (once available).

[sebastianbergmann]

Looks like this is not related to humbug/php-scoper#399, but still to PHP-Scoper. The version of JetBrains/phpstorm-stubs bundled with PHP-Scoper 0.13.2 is missing xdebug_set_filter().

@theofidry (or somebody else familiar with PHP-Scoper, maybe @sebastianfeldmann?) Can you tell what I need to work around this issue while we wait for a new release of PHP-Scoper? A pull request to either the PHP-Scoper configuration that PHPUnit uses or to PHPUnit's build automation where PHP-Scoper is used would be much appreciated.

[sebastianbergmann]

@jrfnl Thank you for offering to help, Juliette. Right now I do not know how to produce something for you test.

[sebastianbergmann]

@jrfnl In the meantime, you should probably be fine with using PCOV instead of Xdebug.

[jrfnl]

@sebastianbergmann No worries & no rush. In the mean time, I'll just use PHPUnit 9.2.x to generate code coverage and I'll patiently anticipate the moment when I can run the tests with branch/path coverage ;-)

[jrfnl]

@sebastianbergmann Just tested & found working. Thanks for the quick turn-around.

I do have some observations. /cc @dvdoug

1. Old versus new config

If both the old as well as the new config are present, the new config is ignored (at least the pathCoverage="true" part is) and the old config is used.

I'd expect:

• The new config to take precedence.
• No validator error to be shown.

Reasoning:
Having one config file containing both the old as well as the new configuration for coverage allows for running code coverage on high/low supported PHP versions which may need different PHPUnit versions with one configuration file.

Of course, this can be done with a separate phpunit-8.xml file to be passed to the lower PHPUnit run, but:

1. That makes the setup more error prone as any change in one config file would also need to be made in the other, which is easily forgotten.
2. This makes the setup more fiddly, as depending on the PHPUnit version used, you now suddenly have to the -c phpunit-8.xml command line argument.

2. Memory usage

I'm not surprised that the path/branch coverage uses up more memory, I am surprised at the amount and wonder if there is a memory leak somewhere.

First coverage run with the old config active (no path/branch coverage):

Time: 05:32.443, Memory: 56.00 MB
Tests: 2598, Assertions: 4131, Skipped: 5.

Second coverage run with the new config active, results straight away in a Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 28672 bytes) in phar://path/to/phpunit-9.3.3.phar/php-code-coverage/Driver/Xdebug2Driver.php on line 88 with a memory_limit in the php.ini file of 256M.

Mind: this error appears before any tests are even run.

Third coverage run with the new config active and -d memory_limit 1G succeeds with these stats:

Time: 28:38.428, Memory: 458.00 MB
Tests: 2598, Assertions: 4131, Skipped: 5.

Are these known issues ? and/or design/implementation choices ? Or should I open issues for these ?

[dvdoug]

Peak memory usage is because the array emitted from Xdebug with all of the data...is large. Commenting out https://github.com/sebastianbergmann/php-code-coverage/blob/master/src/RawCodeCoverageData.php#L55 so that the additional data is discarded immediately after collection doesn't affect memory usage in my testing.

[sebastianbergmann]

Having one config file containing both the old as well as the new configuration for coverage allows for running code coverage on high/low supported PHP versions which may need different PHPUnit versions with one configuration file.

A phpunit.xml configuration file that was valid for PHPUnit 9.2 still works with PHPUnit 9.3. However, you cannot mix PHPUnit 9.2 and PHPUnit 9.3 format in the same XML configuration.

You know what I think about wanting to support multiple PHP versions and multiple PHPUnit versions in the same codebase. We disagree on this topic and all I can say is: I am sorry that this not convenient to deal with in an environment such as yours, but it is not a use case that I want to handle or optimize for.

2. Memory usage

Xdebug needs to do a lot more work when branch/path coverage is used. That being said, code coverage performance issues are out of scope for this issue tracker as the respective functionality is implemented in php-code-coverage.

Mind: this error appears before any tests are even run.

php-code-coverage performs pre-processing of the sourcecode files that are configured in phpunit.xml.

Are these known issues?

Yes.

There is also a performance-related issue that is not limited to branch/path coverage: sebastianbergmann/php-code-coverage#789

[dvdoug]

Of course, this can be done with a separate phpunit-8.xml file to be passed to the lower PHPUnit run, but:

1. That makes the setup more error prone as any change in one config file would also need to be made in the other, which is easily forgotten.

2. This makes the setup more fiddly, as depending on the PHPUnit version used, you now suddenly have to the `-c phpunit-8.xml` command line argument.

You still need to know the version in use, but you can keep using the 9.2 config and pass --path-coverage on the CLI at least until PHPUnit 10