Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to write test cases to JSON file #366

Merged
merged 14 commits into from Oct 24, 2022

Conversation

cpdeethree
Copy link
Contributor

@cpdeethree cpdeethree commented Oct 23, 2022

Adds the ability to optionally write out all of the individual unit test results to json file.

Intended to resolve #365

@cpdeethree
Copy link
Contributor Author

@EnricoMi let me know what you think about this approach!

@EnricoMi EnricoMi changed the title feat: option to write out test cases to json Add option to write test cases to JSON file Oct 23, 2022
Copy link
Owner

@EnricoMi EnricoMi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good PR, just minor changes required.

README.md Outdated Show resolved Hide resolved
composite/action.yml Outdated Show resolved Hide resolved
python/test/test_action_script.py Outdated Show resolved Hide resolved
python/test/test_publisher.py Outdated Show resolved Hide resolved
python/test/test_action_script.py Outdated Show resolved Hide resolved
python/test/test_publisher.py Outdated Show resolved Hide resolved
python/test/test_publisher.py Outdated Show resolved Hide resolved
python/test/test_publisher.py Outdated Show resolved Hide resolved
@github-actions
Copy link

github-actions bot commented Oct 23, 2022

Test Results (reference)

       63 files  ±  0         63 suites  ±0   30m 57s ⏱️ + 1m 0s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

♻️ This comment has been updated with latest results.

cpdeethree and others added 4 commits October 23, 2022 13:22
Co-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Enrico Minack <github@enrico.minack.dev>
@cpdeethree
Copy link
Contributor Author

@EnricoMi I think this is ready for your eyes again. Thanks for the quick and thorough feedback!

@EnricoMi
Copy link
Owner

I have added this to the CI/CD, so this feature is used there as well: 5d01af9

Please pull from your remote git.

@github-actions
Copy link

github-actions bot commented Oct 23, 2022

Test Results (Docker Image)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

♻️ This comment has been updated with latest results.

@EnricoMi
Copy link
Owner

I am going to rework class UnitTestCaseResults today, which is difficult to serialize atm.

@cpdeethree
Copy link
Contributor Author

I am going to rework class UnitTestCaseResults today, which is difficult to serialize atm.

I suspected the failure was related to this. Thanks! Looking forward to what you come up with!

@EnricoMi
Copy link
Owner

After reworking the UnitTestCaseResults type, I could reproduce the error that we saw in the CI/CD, which wasn't produced by tests before. I have also beautified the structure of cases because the tuple-keys were not very meaningful and could not be JSON-serialized as they were.

I have also added an example to the README.md.

@EnricoMi
Copy link
Owner

Tests are failing due to PyPi outage: https://status.python.org/incidents/77qzc7qsb5wb

@github-actions
Copy link

github-actions bot commented Oct 24, 2022

Test Results (Test File)

97 tests  ±0   80 ✔️ ±0   3m 25s ⏱️ ±0s
  1 suites ±0   17 💤 ±0 
  1 files   ±0     0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

♻️ This comment has been updated with latest results.

Copy link
Owner

@EnricoMi EnricoMi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@github-actions
Copy link

github-actions bot commented Oct 24, 2022

Test Results (Windows python installed)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

Test Results (macOS python installed)

     27 files   -        36       27 suites   - 36   10m 31s ⏱️ - 18m 44s
   384 tests +       10     375 ✔️ +         1      0 💤 ±    0  0 ±0    9 🔥 +  9 
9 165 runs   - 14 397  8 958 ✔️  - 14 100  180 💤  - 324  0 ±0  27 🔥 +27 

For more details on these errors, see this check.

Results for commit 8a4febb. ± Comparison against base commit 71fba75.

@cpdeethree
Copy link
Contributor Author

After reworking the UnitTestCaseResults type, I could reproduce the error that we saw in the CI/CD, which wasn't produced by tests before. I have also beautified the structure of cases because the tuple-keys were not very meaningful and could not be JSON-serialized as they were.

I have also added an example to the README.md.

Nice work!

@cpdeethree
Copy link
Contributor Author

@EnricoMi pypi said that they resolved the problem if you want to try retriggering the builds (I can't do it since it's my first pr here)

@EnricoMi
Copy link
Owner

@cpdeethree
Copy link
Contributor Author

Does this work for you? https://github.com/EnricoMi/publish-unit-test-result-action/actions/runs/3313904648/jobs/5472751179#step:6:2047

It does, that is precisely what I need. Thanks for helping quickly get this in!

@github-actions
Copy link

Test Results (setup-python)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@github-actions
Copy link

Test Results (Dockerfile)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@github-actions
Copy link

Test Results (Test Files)

     67 files  ±0         6 errors  358 suites  ±0   1h 6m 6s ⏱️ ±0s
1 682 tests ±0  1 396 ✔️ ±0    71 💤 ±0  208 ±0  7 🔥 ±0 
1 929 runs  ±0  1 589 ✔️ ±0  106 💤 ±0  226 ±0  8 🔥 ±0 

For more details on these parsing errors, failures and errors, see this check.

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@github-actions
Copy link

Test Results (macOS 12 python installed)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@github-actions
Copy link

Test Results (macOS python 3.6)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@github-actions
Copy link

Test Results (Linux python installed)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@github-actions
Copy link

Test Results (Linux python 3.6)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@github-actions
Copy link

Test Results (Linux 22.04 python installed)

       63 files  ±  0         63 suites  ±0   32m 32s ⏱️ + 2m 35s
     375 tests +  1       375 ✔️ +  1      0 💤 ±0  0 ±0 
23 625 runs  +63  23 121 ✔️ +63  504 💤 ±0  0 ±0 

Results for commit 2b89517. ± Comparison against base commit 71fba75.

@EnricoMi
Copy link
Owner

Thanks for the contribution!

@EnricoMi EnricoMi merged commit 1c4c5ee into EnricoMi:master Oct 24, 2022
@EnricoMi
Copy link
Owner

This has been released with v2.2.0, enjoy!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option to write out all individual test case results to json output file
2 participants