feat: implement --shard option
#12546
Conversation
marionebl
force-pushed
the
6270
branch
2 times, most recently
from
7934540
to
3052d3b
Compare
Co-authored-by: Simen Bekkhus <sbekkhus91@gmail.com>
Todo
|
|
@SimenB I applied all requested changes and changed the CircleCI config to make use of the new feature. Let me know :) |
There was a problem hiding this comment.
Well done, this is incredibly thorough and quick!
From a quick review, it appears that this only shards the spec files themselves and not the tests within the suites. This implementation would be better than nothing, but to me the true solution would be to shard the total test count between machines. This is how the --shard option works in @playwright/test for single or many test suites.
# shards all tests within a single test file
npx playwright test --shard=1/3 tests/one.test.ts
# shards all tests across 2 test files
npx playwright test --shard=1/3 tests/one.test.ts tests/two.test.ts
# shards all tests across all project test files
npx playwright test --shard=1/3So if you had 2 test files one with 20 tests and the other with 80, both commands below would run 50 tests, never the same test twice.
npx playwright test --shard=1/2 # 50 tests run
npx playwright test --shard=2/2 # 50 tests runI imagine most jest setups have good distribution and separation of test files such that the test suite sharding solution would still be incredibly useful. I'm not aware if the jest runner setup is suitable for sharding on a per-test basis (i.e. running a consistent subset of tests for for a single test file/suite), maybe a jest maintainer can opine.
I'm just a spectator so feel free to ignore my thoughts. 👍🏼
Good shout! I've toyed with the concept and discarded if after initial tests in a code base I maintain. My observations:
This assumes we need full blown transpilation before listing test cases which I'm not 100% confident about maybe @SimenB can chime in. |
|
Yeah, we don't want to execute the test files before splitting up as that would require transpilation as mentioned, but also actually running the test file (but not the |
|
One thing to explore for this repo is to shard in a way that mostly splits off
So we should probably, funnily enough, implement our of test scheduler and not use the default one. That said of course, the gains here are already quite massive 😅 |
| const shardEnd = shardSize * options.shardIndex; | ||
|
|
||
| return [...tests] | ||
| .sort((a, b) => (a.path > b.path ? 1 : -1)) |
There was a problem hiding this comment.
I wonder if we should be more clever here. Thoughts on calling await this.sort(tests) and then splitting off from there? and then using index % options.shardIndex === 0 or something (not that exactly since it won't work 😅) since sort by default tries to schedule slowest/failing/largest first.
Might be overkill, dunno
There was a problem hiding this comment.
I've thought about this one a bit - my team will use a jump consistent hash (https://arxiv.org/pdf/1406.2294.pdf) based on the filename, sort based on that and then apply modulo. That should make for nicely distributed and very stable shards.
In a later iteration I believe we could try and balance the shards to contain a similar amount of predicted test run time based on historical data, not sure yet what that would like yet though 😅
There was a problem hiding this comment.
TL;DR If you don't object I'll give a smarter shard algorithm a crack; thoughts on pulling in a jump consistent hashing lib as dependency vs. having it in source?
There was a problem hiding this comment.
Worth a go! Might be best to do that in a separate PR though, just to not block this one 🙂 I'm very interested in seeing that happen, though!
There was a problem hiding this comment.
I went ahead and added simple hashing of the test path, results are encouraging. The distribution of e2e test went from
108
102
0
0
to
51
48
62
49
Command used:
λ yarn jest --listTests --shard='1/4' | grep e2e | wc -l && yarn jest --listTests --shard='2/4' | grep e2e | wc -l && yarn jest --listTests --shard='3/4' | grep e2e | wc -l && yarn jest --listTests --shard='4/4' | grep e2e | wc -l
For CI this means:
There was a problem hiding this comment.
Oh yeah, this is awesome! 8 minutes CI is a dream 😀
Co-authored-by: Simen Bekkhus <sbekkhus91@gmail.com>
Co-authored-by: Simen Bekkhus <sbekkhus91@gmail.com>
| const relativeTestPath = path.relative( | ||
| test.context.config.rootDir, | ||
| test.path, | ||
| ); |
There was a problem hiding this comment.
a change in the logic to make the sharding stable between machines with different paths (test.path is an absolute path, so the hash differs between machines). Not really an issue except that we test the sharding in this repo, so we need to be stable 🙂
|
4 minute CI, gotta love it EDIT: or even 3 😅
|
|
Huh, test is failing on windows node@14, but not 12, 16 or 17... |
| test, | ||
| }; | ||
| }) | ||
| .sort((a, b) => a.hash.localeCompare(b.hash)) |
There was a problem hiding this comment.
Will produce inconsistent results across machines with different locale settings - maybe forcing it to a specific locale would be the best option here?
There was a problem hiding this comment.
would have thought that didn't impact file paths, but I've been surprised by FS behaviour before 😛
|
Test failures:
|
|
This time it failed with jest jasmine, not circus... something is off. I only have access to node 12 on windows, but I'll try dig |
|
Thank you again @marionebl! |
|
Thank you for getting it across the line! 🎉 |
|
QQ for you both @SimenB @marionebl - when running coverage with |
|
@therynamo Istanbul report supports it natively now: jamestalmage/istanbul-combine#2 |
|
@llwt - woah! Nice! So I just need to make sure that jest allows for the Wow - great find! Thank you for the reply too 👍 |
|
You'll need to merge the reports manually, yeah. Or use a CI/reporter that does it for you (e.g. coveralls) |
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Summary
Fixes #6270
Test plan
shardtest)