DX: Use ParaUnit to speed up tests

[Wirone]

Currently test suites in Github Actions take ages. This PR's goal is reducing time needed for running test suite(s).

Initially started with ParaTest, but after encountering problems with test loader I've changed to ParaUnit.

Benchmark:

	Process Lint	Fast Lint
PHPUnit	00:26:54 😩	00:00:56
ParaUnit	00:04:48	00:00:28 ❤️

[Wirone]

FYI: Test execution comparison, run locally on MacBook Pro M1, without smoke tests (466 test classes, 33572 tests):

	Process Lint	Fast Lint
PHPUnit	00:26:54 😩	00:00:56
ParaUnit	00:04:48	00:00:28 ❤️

ParaUnit is 2x faster than regular PHPUnit using Fast Linter (FAST_LINT_TEST_CASES=1), and over 5x faster using Process Linter. 5 minutes of whole suite is actually something that can be run locally without much problem, but 27 minutes? Naaahhhhh.

[Wirone]

FYI: about 462 files with NO TESTS EXECUTED, example with --debug:

PROCESS STARTED: /app/tests/RuleSet/Sets/PERRiskySetTest.php
'php' '/app/vendor/phpunit/phpunit/phpunit' '--configuration=/app/phpunit.xml' '--extensions=Paraunit\Parser\JSON\TestHook\BeforeTest,Paraunit\Parser\JSON\TestHook\Error,Paraunit\Parser\JSON\TestHook\Failure,Paraunit\Parser\JSON\TestHook\Incomplete,Paraunit\Parser\JSON\TestHook\Risky,Paraunit\Parser\JSON\TestHook\Skipped,Paraunit\Parser\JSON\TestHook\Successful,Paraunit\Parser\JSON\TestHook\Warning' '--group=auto-review' '/app/tests/RuleSet/Sets/PERRiskySetTest.php'


PROCESS TERMINATED: /app/tests/PregTest.php
 - with class name: N/A

PROCESS FULL OUTPUT:

PHPUnit 9.6.9 by Sebastian Bergmann and contributors.

No tests executed!

This test is not a part of auto-review group. So this is just difference between how PHPUnit and ParaUnit prepares test set - former filters actual set before running, while the latter runs everything but for some classes gets filtered result from the upstream.

Let's sum it up:

• 471 test classes processed
• 462 files with NO TESTS EXECUTED
• 9 test classes with @group auto-review

Everything is correct 🙂. It's not optimal, since 98% of test classes in Auto Review job are run for nothing, but the result is OK. Same with tests mentioned here - these 2 classes are from group covers-nothing which is excluded from code coverage.

[keradus]

did you tried to compare results of coverage generated by PHPUnit and by paraunit? what's the difference?

[Wirone]

No, I did not verify it, but I'll do it 👍.

[Wirone]

In terms of coverage it looks like everything works as it should:

• Coverage: 93.15% (23631/25368) for DX: Progress output refactor #6848
• Coverage: 94.47% (23942/25344) for this MR

[keradus]

i have the feeling that number should not be different.

[kubawerlos]

The number is different because the code is having global state, e.g. https://github.com/PHP-CS-Fixer/PHP-CS-Fixer/blob/v3.21.1/src/Tokenizer/CT.php#L99-L104 - running it with PHPUnit make that if covered (or not covered) all the time with the same result, ParaUnit make it non-deterministic.

If you run PHPUnit like ./vendor/bin/phpunit --order-by=random you will get slightly different coverage for each run.

[keradus]

I find this very confusing. Maybe we could at least document it in codebase (not only in PR) ?

numbers diff is impressive 👍🏻

[keradus]

tbh, I never heard about it.

when thinking about parallel execution of PHPUnit, somehow I always had in mind https://github.com/paratestphp/paratest , and to my understanding maintainer of it, Slamdunk, is also more close to PHPUnit itself.

would you mind to share this and that about 2 solutions and arguments for decision?

[Jean85]

Hi, Paraunit creator here. Paratest was in disuse when I created Paraunit, but Slamdunk took it over later and revived it. Now it has been used into Laravel, hence the growing popularity.

The main difference between the two tools is basically in the approach: Paratest is "hand on", uses @internal components of PHPUnit and has a tighter integration with it (hence some additional capabilities like being able to hook into specific output formats or being able to filter by groups); Paraunit is "hands off", just executes PHPUnit from the outside, hence the simpler integration but less functionality.

BTW I know Slamdunk and he even invited me to co-maintaint Paratest, but I have to admit I'm not so well versed in it.

[keradus]

Thanks for the details!

side note: yeah, it could be interesting to have the parallel executor available, instead of few possibilities. I could suggest one step further - consider to incorporate it to PHPUnit natively [I know Sebastian wanted to have some internal cleanups first, but if I recall right, lot of them already in place]

[Wirone]

I find this very confusing. Maybe we could at least document it in codebase (not only in PR) ?

@keradus let's finish #6839, then such change would be transparent to users because Composer script would switch from PHPUnit to ParaUnit and everything would work as we expected 🙂. This is advantage of that PR, which I'm fighting for for months 😆.

[coveralls]

Pull Request Test Coverage Report for Build 5474591896

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+1.3%) to 94.521%

---

Files with Coverage Reduction	New Missed Lines	%
src/Console/Output/ErrorOutput.php	2	92.42%

Totals
Change from base Build 5474542201:	1.3%
Covered Lines:	24102
Relevant Lines:	25499

---

💛 - Coveralls