Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(test runner): Use timing file to balance shards #30388

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ofirpardo-artlist
Copy link

This feature came from this issue: #17969

The idea is to use the JSON reporter and extract the duration of each test, then map the test groups based on the duration so the shards are all as equal as possible, the amount of tests per shard is not relevant here, only the duration of the tests.
If there are tests that are not recorded in the JSON report they will be distributed like they currently do based on the amount.

This is still in a POC state to get some feedback, but the functionality works well.

Some open questions:

  • Not sure about the behavior of non-parallel tests/files
  • Optionally adding a config flag as well, currently it's only available from CI
  • Do we want to use the JSON report, or create a custom 'TestDetails' report which only includes the necessary data

At the end of the day unless you come up with some cloud solution, the 'infra' for this feature would need to be done mostly manually, a static JSON report is optional but might get irrelevant very fast.
On my project I've managed it by averaging the durations of the last 10 successful runs, which is being updated to a CDN from CI.

Some more details on current POC can be seen here:
#17969 (comment)
#17969 (comment)

@ofirpardo-artlist
Copy link
Author

ofirpardo-artlist commented Apr 16, 2024

@microsoft-github-policy-service agree company="Artlist"

This comment has been minimized.

Copy link
Contributor

Test results for "tests 1"

1 flaky ⚠️ [webkit-library] › library/browsercontext-clearcookies.spec.ts:52:3 › should remove cookies by name

26922 passed, 628 skipped
✔️✔️✔️

Merge workflow run.

@anujshah3
Copy link

Hi @ofirpardo-artlist
Is there an expected release timeline for this PR? This is a much needed feature!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants