New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make it possible to automatically rerun selected flaky tests #1629
Comments
If somebody wants to work on this, please comment here, to avoid multiple contributors working on the same task. |
I would like to work on this issue. |
@chungngoops thank you! Feel free to ask any questions you have before implementing it 👍 |
FYI the current state can be found here: |
@chungngoops I wanted to have a play with one possible way of doing it in #1680, as it's causing us a lot of pain at the moment (our build should not be failing as it is right now, and we need to fix this urgently). Consider #1680 an experimental approach though - if you have a better way, it'd be good to hear about it. |
@rnorth: I have a similar approach, mine just directly implement |
@chungngoops thanks - I think being able to specify the number of retries is a nice thing to have, so I will add as an annotation property. Shall we just use #1680? I think I'd like to try and complete this before next week. |
There are a few tests that fail due to the underlying Docker issues. Some examples:
Since we agreed in #1594 that we should only rerun the tests that are known to be flaky, we should have a mechanism (e.g. with a help form JUnit) to rerun them automatically, N times (where N is ~1..3)
Such test cases (important! not classes) should be explicitly marked (e.g. with an annotation) as flaky. This will also make it possible to find usages of that annotation and discover all flaky tests in the project.
"Flaky test" here means a test that fails randomly, due to some timing issues, or CIs problem with resource allocation or the underlying Docker issues
The text was updated successfully, but these errors were encountered: