test(ci): Adjust detectFlakyTests to account for multiple tests in a file
#11653
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lms24
commented
Apr 17, 2024
| /** | ||
| * The number of browsers we run the tests in. | ||
| */ | ||
| const NUM_BROWSERS = 3; |
There was a problem hiding this comment.
So, we previously set this to 4 but afaict at max, we test 3 browsers. Some tests potentially even only run on Chrome. So I think it's ok to reduce this in favour of more often running each test. My super unscientific testing says we currently run way shorter than 30min.
size-limit report 📦
|
mydea
approved these changes
Apr 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our flaky test detector uses a couple of heuristics to determine how many times each changed test should be run. We previously relied on the assumption that one test path (file) only contained one file. In some cases this is not true because it's totally valid to contain multiple
sentryTesttest runs in one file.This PR adds a small test detection heuristic to the detector to roughly count the number of tests in a file. This should reduce the number of times each test is running in case files with multiple tests have changed. Furthermore, this PR also refactors the run count determination logic a bit to make it easier to understand.
An example of this is a PR of mine (#11631) where the test detector would run into the 60 minutes timeout because each changed test file contained multiple (~5) test files. So the actual number of tests would be 5x higher than what our heuristic assumed.