ci: Add workflow to detect flaky tests #7497
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new script & workflow that can be run to detect flaky tests.
It runs tests a given amount of times (10 to try it out), and logs errors if they happen, else log the average run time (note this is a bit ~~ as it just takes the duration to run the given test, separated by run count - so parallelization will skew this a bit. Still it should give a decent insight how long a given test takes).
Update:
This now runs automatically on PRs that change any browser integration tests. It will run all changed integration tests 100x, separate from the rest of the integration tests.
note that this is not merge blocking, but will report if it fails.
You can also choose to run this manually via workflow_dispatch, in which case it will run all tests 5x (for now).