Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducing fail tests #5454

Open
Tracked by #5456
richtja opened this issue Aug 4, 2022 · 0 comments
Open
Tracked by #5456

Reproducing fail tests #5454

richtja opened this issue Aug 4, 2022 · 0 comments

Comments

@richtja
Copy link
Contributor

richtja commented Aug 4, 2022

Is your feature request related to a problem? Please describe.
When some of the tests from job fails, it would be good to be able to re-run only the failed tests with the same configuration to reproduce the failures. It is more efficient to run only failed tests and skip the passed ones.

Describe the solution you'd like
Let's create avocado configuration option for reproducing failed tests. We can use avocado recipe file to store the active avocado configuration for each failed test. If the configuration will be stored in recipe file, it would be easier to move the configuration to another machine.

This feature should look like this:
Running avocado job:

$ avocado run --store-failed-configuration=/tmp/fail.json /bin/true /bin/false examples/tests/passtest.py examples/tests/failtest.py examples/tests/errortest.py 
JOB ID     : 2c7b8348c00b3ab711061b76995d8db216d03446
JOB LOG    : /home/jarichte/avocado/job-results/job-2022-08-04T14.05-2c7b834/job.log
 (3/5) examples/tests/passtest.py:PassTest.test: STARTED
 (1/5) /bin/true: STARTED
 (1/5) /bin/true: PASS (0.02 s)
 (3/5) examples/tests/passtest.py:PassTest.test: PASS (0.01 s)
 (2/5) /bin/false: STARTED
 (2/5) /bin/false: FAIL (0.01 s)
 (4/5) examples/tests/failtest.py:FailTest.test: STARTED
 (5/5) examples/tests/errortest.py:ErrorTest.test: STARTED
 (4/5) examples/tests/failtest.py:FailTest.test: FAIL: This test is supposed to fail (0.04 s)
 (5/5) examples/tests/errortest.py:ErrorTest.test: ERROR: This should end with ERROR. (0.04 s)
RESULTS    : PASS 2 | ERROR 1 | FAIL 2 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/jarichte/avocado/job-results/job-2022-08-04T14.05-2c7b834/results.html
JOB TIME   : 2.45 s
Failed tests were stored in  /tmp/fail.json

Reproducing failed tests:

avocado run /tmp/fail.json
JOB ID     : 97c28bcdccaeb762060a192bad75f73048728823
JOB LOG    : /home/jarichte/avocado/job-results/job-2022-08-04T14.08-97c28bc/job.log
 (1/3) /bin/false: STARTED
 (2/3) examples/tests/failtest.py:FailTest.test: STARTED
 (1/3) /bin/false: FAIL (0.01 s)
 (3/3) examples/tests/errortest.py:ErrorTest.test: STARTED
 (2/3) examples/tests/failtest.py:FailTest.test: FAIL: This test is supposed to fail (0.04 s)
 (3/3) examples/tests/errortest.py:ErrorTest.test: ERROR: This should end with ERROR. (0.03 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 2 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/jarichte/avocado/job-results/job-2022-08-04T14.08-97c28bc/results.html
JOB TIME   : 2.12 s
@richtja richtja added this to Long Term (Next Q) Backlog in Avocado Kanban via automation Aug 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Avocado Kanban
  
Long Term (Next Q) Backlog
Status: Long Term (Next Q) Backlog
Development

No branches or pull requests

1 participant