Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DX: Use ParaUnit to speed up tests #6883

Merged
merged 5 commits into from
Jul 8, 2023
Merged

DX: Use ParaUnit to speed up tests #6883

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
adaca94
PoC: Use ParaTest to speed up tests
Wirone Apr 4, 2023
a445af9
PoC: Test compatibility with Symfony 6 using ^6.1 and PHP8.1
Wirone Apr 4, 2023
5bdba7f
PoC: Run tests on PHP8.0 too
Wirone Apr 4, 2023
aa859fa
PoC: Use ParaUnit instead of ParaTest
Wirone Apr 5, 2023
82b04ac
Revert previous CI matrix
Wirone Jul 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,6 @@ jobs:
php-version: '8.2'
job-description: 'with calculating code coverage'
calculate-code-coverage: 'yes'
phpunit-flags: '--testsuite coverage --exclude-group covers-nothing --coverage-clover build/logs/clover.xml'

- operating-system: 'ubuntu-20.04'
php-version: '8.2'
Expand Down Expand Up @@ -143,10 +142,15 @@ jobs:
run: sed 's/enforceTimeLimit="true"/enforceTimeLimit="false"/g' phpunit.xml.dist > phpunit.xml

- name: Run tests
if: matrix.calculate-code-coverage != 'yes'
env:
PHP_CS_FIXER_IGNORE_ENV: ${{ matrix.PHP_CS_FIXER_IGNORE_ENV }}
FAST_LINT_TEST_CASES: ${{ matrix.FAST_LINT_TEST_CASES }}
run: vendor/bin/phpunit ${{ matrix.phpunit-flags || '--exclude-group auto-review' }}
run: vendor/bin/paraunit run ${{ matrix.phpunit-flags || '--exclude-group auto-review' }}

- name: Collect code coverage
if: matrix.calculate-code-coverage == 'yes'
run: vendor/bin/paraunit coverage --testsuite coverage --exclude-group covers-nothing --clover build/logs/clover.xml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you tried to compare results of coverage generated by PHPUnit and by paraunit? what's the difference?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I did not verify it, but I'll do it 馃憤.

Copy link
Member Author

@Wirone Wirone Jul 3, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of coverage it looks like everything works as it should:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i have the feeling that number should not be different.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The number is different because the code is having global state, e.g. https://github.com/PHP-CS-Fixer/PHP-CS-Fixer/blob/v3.21.1/src/Tokenizer/CT.php#L99-L104 - running it with PHPUnit make that if covered (or not covered) all the time with the same result, ParaUnit make it non-deterministic.

If you run PHPUnit like ./vendor/bin/phpunit --order-by=random you will get slightly different coverage for each run.


- name: Upload coverage results to Coveralls
if: matrix.calculate-code-coverage == 'yes'
Expand Down
1 change: 1 addition & 0 deletions composer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
"symfony/stopwatch": "^5.4 || ^6.0"
},
"require-dev": {
"facile-it/paraunit": "^1.3 || ^2.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh, I never heard about it.

when thinking about parallel execution of PHPUnit, somehow I always had in mind https://github.com/paratestphp/paratest , and to my understanding maintainer of it, Slamdunk, is also more close to PHPUnit itself.

would you mind to share this and that about 2 solutions and arguments for decision?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, Paraunit creator here. Paratest was in disuse when I created Paraunit, but Slamdunk took it over later and revived it. Now it has been used into Laravel, hence the growing popularity.

The main difference between the two tools is basically in the approach: Paratest is "hand on", uses @internal components of PHPUnit and has a tighter integration with it (hence some additional capabilities like being able to hook into specific output formats or being able to filter by groups); Paraunit is "hands off", just executes PHPUnit from the outside, hence the simpler integration but less functionality.

BTW I know Slamdunk and he even invited me to co-maintaint Paratest, but I have to admit I'm not so well versed in it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the details!

side note: yeah, it could be interesting to have the parallel executor available, instead of few possibilities. I could suggest one step further - consider to incorporate it to PHPUnit natively [I know Sebastian wanted to have some internal cleanups first, but if I recall right, lot of them already in place]

"justinrainbow/json-schema": "^5.2",
"keradus/cli-executor": "^2.0",
"mikey179/vfsstream": "^1.6.11",
Expand Down