New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tests that have not run end up in SpecStateSkipped in after suite report #1320
Comments
hey @mat007 - |
Ah, I totally misunderstood what Thanks for the suggestion about looking at the test failure message. But then, is there now a way to tell the difference between a test that was not run because it was out of focus (suite’s config |
nor straightforwardly I don’t think - but at this point i should ask: what problem are you trying to solve? |
Yes, sorry, fair ask! We run tests nightly on the CI a whole bunch of times in a loop, in batches with various focuses, and record the state of each test. We wanted to start categorizing these a bit better, by separating the «real» skipped tests (typically these ~0.5%), the «not focused by design» tests (our tests are split in different «groups» so we can parallelize batch them), and the «unknown» (or «not run», or …) tests. It’s actually important for us to monitor the latter, because when it increases it means the test failure recovery mechanism we have may be starting to get flaky as well, and needs to be looked at. I hope this is clear enough 😅 I guess I could try to manually re-apply the focus/skip regexps in my after suite report to tell if a test was in our out of focus. 🤔 |
ok got it - thanks. Ginkgo doesn't have super strong first-class support for this right now. It's almost like you want something like "SkipReason" which will allow you to tell if a spec was skipped because it was out of focus, or if the user skipped it, or if an abort/timeout has occured. I could imagine adding that to the codebase though I'm not going to get to it soon :/ If you're up for doing some more work on your end, though, you can add a I think that'll get you what you want? If you want to be more granular and do this on a per-spec level you'll need to be able to connect the specs in the Sorry it's so tedious, though. I'll add |
Thanks mate, that’s a good lead! |
Hi!
Is there a way in a
ginkgo.ReportAfterSuite
handler to tell the difference between a test that has been explicitly skipped (i.e. by callingginkgo.Skip
in its code) from a test that was not run (because e.g. aginkgo.AbortSuite
happened before the test had a chance to run)?It seems all these tests end up in the
SpecStateSkipped
state. Shouldn’t the tests that never ran at all still be in theSpecStatePending
state?Thanks!
The text was updated successfully, but these errors were encountered: