Capability to rerun failed tests cases (Retry Mechanism)

[reshadat]

Is your feature request related to a problem? Please describe.
There are a couple of tests that are flaky. These are mostly environment issues hence it is not possible to handle such cases.

Describe the solution you'd like
We can have a mechanism to easily retry tests, There should be a configuration to allow only few tests to retry and no of retries.

Describe alternatives you've considered
Can use BATS_TEST_COMPLETED variable to check and add failed tests to a global level variable, and use that to run tests again. It is not entirely impossible, but native support would be great.

Additional context
TestNG creates a retry xml file that can be used to rerun tests. We can have a simpler approach where a file contains test file name and tests name. that can be parsed to rerun as needed.

[martin-schulze-vireso]

I am unsure whether you are asking for a retry maechanism that retries tests within the same bats run or in subsequent runs. I am also unsure why exactly you want to Limit the number of tests to rerunas opposed to the number of retries. What should happen if the limit is reached?

[reshadat]

It can work either ways. Either rerun it after it failed, or run it collectively at the end.

The reason to allow only few tests to rerun is to limit the unnecessary retries. If the tests has reached the max retires, last result is the final result of that test

This is the same philosophy used in TestNG(Retry Analyzer).

[martin-schulze-vireso]

Well, we already have #483 in the pipeline which does seem like a fit for half of your request. However, I am see this as a better fit to speed up local development, where one might want to only work with the failing tests.

On the CI server, I'd expect a more automated solution that retries flaky tests for a given amount of times and then gives up, which I would put into the "within one bats run" category.

The reason to allow only few tests to rerun is to limit the unnecessary retries.

Let me try to rephrase that to see if I got you right: Let's say you got 3 tests A, B and C which are flaky. Now you want to give each test 3 tries to get it right but you also want to limit the maximum number of tests run to say 4? So if A fails 3/3 tests and B passes on first try, then C is skipped because the maximum number of tests has been executed? Is that what you mean?

If the tests has reached the max retires, last result is the final result of that test

I would have assumed that as soon as a test passes a try, we report that test as passed. Your suggestion sounds more like: run the test n times and take the last result regardless of the other tries.

[reshadat]

Well, we already have #483 in the pipeline which does seem like a fit for half of your request.

Went through the PR, and seems like this would be a great feature for me

On the CI server, I'd expect a more automated solution that retries flaky tests for a given amount of times and then gives up, which I would put into the "within one bats run" category.

Agreed, however there is not straightforward way right now to retry only failed tests in a bats file.

Let me try to rephrase that to see if I got you right: Let's say you got 3 tests A, B and C which are flaky. Now you want to give each test 3 tries to get it right but you also want to limit the maximum number of tests run to say 4? So if A fails 3/3 tests and B passes on first try, then C is skipped because the maximum number of tests has been executed? Is that what you mean?

No, what I mean is that each failing tests gets 3 retry attempts, if last iteration is failed, it can be marked as failed/rerun failed. As soon as a test passes, it is not retried again.

I would have assumed that as soon as a test passes a try, we report that test as passed. Your suggestion sounds more like: run the test n times and take the last result regardless of the other tries.

Only if the test fails, it is retried. Hence it may pass first time, and in that case it is not retried.

This may sound confusing, my query is very much influenced with how TestNG works.