Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reporting improvements for Scrutineer #3551

Open
Zac-HD opened this issue Jan 19, 2023 · 2 comments
Open

Reporting improvements for Scrutineer #3551

Zac-HD opened this issue Jan 19, 2023 · 2 comments
Labels
enhancement it's not broken, but we want it to be better

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Jan 19, 2023

As part of the explain phase, codenamed Scrutineer, we collect and analyze coverage information. This issue is about upgrading the way we report the information we collect, defined in this python module.

Looking at the following output from a failing Pandas test made me think of a few improvements:

Explanation:
    These lines were always and only run by failing examples:
        C:\hostedtoolcache\windows\Python\3.10.9\x64\lib\site-packages\_pytest\_io\saferepr.py:81
        C:\hostedtoolcache\windows\Python\3.10.9\x64\lib\site-packages\_pytest\assertion\util.py:235
        C:\hostedtoolcache\windows\Python\3.10.9\x64\lib\site-packages\numpy\core\_methods.py:49
        C:\hostedtoolcache\windows\Python\3.10.9\x64\lib\site-packages\pandas\io\formats\format.py:1306
        C:\hostedtoolcache\windows\Python\3.10.9\x64\lib\site-packages\pandas\io\formats\format.py:1312
        (and 18 more with settings.verbosity >= verbose)
  • Add some more files to the ignorelist
  • Increase the number of displayed locations from five to ten
  • Sort so that we prefer local files, then site-packages, and standard-library last of all.
@Zac-HD Zac-HD added the enhancement it's not broken, but we want it to be better label Jan 19, 2023
@Zac-HD
Copy link
Member Author

Zac-HD commented Jan 23, 2023

An alternative heuristic that might be more helpful: we can check whether there's currently an exception propagating using sys.exc_info(), and keep track of the number of exceptions which have been raised (and in all-but-one cases, caught). Then each time we add a new line to the mapping, note the exception idx (or None); and finally decline to report any lines executed only after the last exception we saw began to propagate.

Or in plain words: ignore lines that only occured after the exception-that-made-the-test-fail was raised.

Might have some pretty annoying overhead, but on the other hand we only need to run this on the minimal failing example, not all the others, so overhead can be low at cost of writing a second trace function.

@Zac-HD
Copy link
Member Author

Zac-HD commented Mar 24, 2023

Another idea: it's easy enough to track the order in which we first saw each unique location. We could then highlight the earliest divergence, while still reporting the others as useful context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement it's not broken, but we want it to be better
Projects
None yet
Development

No branches or pull requests

1 participant