Tests take 4 times more time since PhpUnit 8

[VincentLanglet]

Q	A
PHPUnit version	8.2.x
PHP version	7.2.x
Installation Method	Composer

I use this command line to run tests.

symfony/vendor/bin/phpunit --testdox -c symfony/ symfony/tests/Unit

With this phpunit.xml.dist

<?xml version="1.0" encoding="UTF-8"?>

<!-- https://phpunit.de/manual/current/en/appendixes.configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/6.5/phpunit.xsd"
         backupGlobals="false"
         colors="true"
         bootstrap="config/bootstrap.php"
         failOnWarning="true"
>
    <php>
        <ini name="error_reporting" value="-1" />
        <env name="APP_ENV" value="test" />
        <env name="DATABASE_URL" value="mysql://sagesconseil:pleaseChangeMe@127.0.0.1/asv_test" />
        <env name="SYMFONY_DEPRECATIONS_HELPER" value="weak" />
    </php>

    <filter>
        <whitelist>
            <directory>src</directory>
            <exclude>
                <directory>src/Admin</directory>
                <directory>src/Entity</directory>
                <directory>src/Form/Helper</directory>
                <directory>src/Form/Type</directory>
                <directory>src/Resources</directory>
            </exclude>
        </whitelist>
    </filter>

    <testsuites>
        <testsuite name="All tests">
            <directory>tests</directory>
        </testsuite>
    </testsuites>

    <listeners>
        <listener class="\DAMA\DoctrineTestBundle\PHPUnit\PHPUnitListener" />
    </listeners>
</phpunit>

With phpunit 7, it takes 8 seconds.

With phpunit 8, it takes 31 seconds.

[epdenouden]

Hello @VincentLanglet!

Thanks for taking the time to post an issue with timing differences between PHPUnit v7 and v8. I tried to search for the project shown in the images, but I think it is private, right?

Question: do you see this 20+ seconds difference when you switch the project back-and-forth between PHPUnit v7 and v8? What changes do you see when running composer install?

It is impossible to tell from the configuration and console output what the reason for the difference is. If I had to debug something like this in my own projects, a few things immediately catch my eye:

• judging by SYMFONY_DEPRECATIONS_HELPER you are using Symfony-PHPUnit bridge, correct?
• there's database interaction and a Doctrine <listener>
• --testdox has had a major refactoring since v7; the printer itself isn't much slower (a bit with the ANSI-color rendering, but insignificant even on slow laptops)

Perhaps @sebastianbergmann has heard about similar changes in runtime from the community. The PHPUnit self-tests are quite extensive and did not show any significant changes in runtime and resource usage between 7 and 8.

[VincentLanglet]

@epdenouden Yes it's a private project for the work. Not sure if I can make a reproductible repository.

Yes, I just have to

• Run my test in phpunit 8.2 in 30+ seconds
• Downgrading phpunit from 8.2 to 7.5 (Removing sebastian/type, downgrading sebastian/global-statefrom 3.0.0 to 2.0.0, downgrading phpunit/php-code-coverage from 7.0.5 to 6.1.4)
• Run my test in phpunit 7.5 in 10 seconds

I do use symfony/phpunit-bridge, I do have database interaction, I don't know for the listener.

[epdenouden]

@VincentLanglet Thanks for the clarification! I do know there was a lot going on surrounding code coverage. Unfortunately I know very little about this part of PHPUnit.

Some questions that come to mind:

• do you run the tests with code coverage? (I think not, based on the config)
• is it any faster when you change the config to colors="false"?
  Maybe there is an undiscovered performance 'leak' in the new prettifier.

[sebastianbergmann]

I am not aware of any performance degration of the scale you describe. I would not be surprised if this were related to Symfony Bridge or similar extensions that wrap PHPUnit.

[VincentLanglet]

I don't run with code coverage. If I do, it's not wanted.
colors=false does nothing.

I am not aware of any performance degration of the scale you describe. I would not be surprised if this were related to Symfony Bridge or similar extensions that wrap PHPUnit.

So the same version of symfony bridge produce a more performant bridge for phpunit 7 than phpunit 8 ?

[sebastianbergmann]

That would be a question for the maintainers of the Symfony Bridge. I do not know whether they have different code paths depending on the version of PHPUnit used.

[VincentLanglet]

For information, on circle.ci my test were running in 1-2 seconds with phpunit 7 and now run in 3-5 seconds with phpunit 8.

[snapshotpl]

In my framework-less test suite I see 10% performance degration in phpunit 8 against v7

[epdenouden]

In my framework-less test suite I see 10% performance degration in phpunit 8 against v7

That is significant. I'm busy with on-demand data providers, giving you back some performance. ;-) Will have a look at comparing 7.x and 8.x with xdebug tracing after that, but it will be a while before I get to it. Any suggestions and help welcome of course. 🔮

[sebastianbergmann]

@epdenouden You may want to give @beberlei's Tideways profiler a try while investigating this.

[epdenouden]

@epdenouden You may want to give @beberlei's Tideways profiler a try while investigating this.

Ohhhh toys tools! Thanks Sebastian :)

[bizley]

We noticed the performance hit as well (~10 times slower) after upgrade from 7 (with Symfony 4.2) to 8 (Symfony 4.3) - PHP version has not been changed (7.2). But running it with n flag (no ini configuration) like

php -n vendor/bin/phpunit

makes it fast again.

[epdenouden]

@bizley thank you for this bit of information! This provides a good starting point for comparison testing.

[epdenouden]

@bizley Rereading, you are talking about the -n flag of PHP itself. So most likely the difference lies in different module configuration, opcaches, etc. Did you investigate the local configuration?

[bizley]

Yes, it's for PHP itself.

I haven't investigated it since the configuration remain the same. But I'm happy to help (just let me know what you need).

[bizley]

I did a little experiment and run the test from my local env with PHP 7.3 (previous was dockerized with PHP 7.2) and it was slow again. Then I switched off xdebug (2.7.1) - and tests were fast. Looks like xdebug is the reason for this performance drop. I know, it's expected but it was never so huge.

[sebastianbergmann]

Stating the obvious here: a performance degradation is only a performance degredation in PHPUnit if the only difference between two test executions is the version of PHPUnit used. You cannot compare "PHPUnit X without Xdebug" with "PHPUnit Y!=X with Xdebug" and conclude that ""PHPUnit Y!=X" is slower.

[bizley]

Of course you are right! Nevertheless PHPUnit is slower no matter with xdebug or not. Here are my results (tests set the same in all cases, PHP 7.2):

PHPUnit	-n flag	Time	Memory
7.5.14	no	2.82 s	32 MB
7.5.14	yes	716 ms	28 MB
8.2.5	no	29.62 s	34 MB
8.2.5	yes	4.09 s	30 MB

[epdenouden]

@bizley Thank you for those statistics! Obviously Xdebug makes things a lot slower, however I will investigate deeper after the data provider improvements. That will have me cleaning up quite a bit of old stuff. For example, loading the providers can cause annotation data being read twice, instantiating extra reflection objects and whatnot.

My eye is drawn to the difference in slowing down: without php.ini it is 4x but with it is 10x. Statistics/lies, signal/noise, etc.

[sebastianbergmann]

@MichelHartmann, @sebastianfeldmann, and @SenseException have used git-bisect to track down this performance degradation to #3673.

Checking the type of values passed to willReturn() and enforcing that they are compatible with the declared return type of a stubbed method is additional work that was previously not performed. This additional work costs time. However, the degree of performance degradation observed here is unexpected.

[mrtnzagustin]

Hi there, i have a problem when executing ./vendor/bin/simple-phpunit in a docker testing stage container. Each time i make a build, test are slow cause simple-phpunit dowload phpunit. How can i download it with other command to mantain it in cache?

[sebastianbergmann]

Neither simple-phpunit nor Docker are on topic in this repository.

[crisnunez1]

Hi, I have the same performance problem but with version 8.5.5, even with very simple testing it can take more than 25 seconds.