Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip known-uninformative modules in Scrutineer explain mode #3439

Closed
Zac-HD opened this issue Aug 12, 2022 · 2 comments · Fixed by #3444
Closed

Skip known-uninformative modules in Scrutineer explain mode #3439

Zac-HD opened this issue Aug 12, 2022 · 2 comments · Fixed by #3444
Assignees
Labels
legibility make errors helpful and Hypothesis grokable

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Aug 12, 2022

I'm quite fond of Scrutineer - the fault-localization component of "explain mode" from #2859 - but it's certainly not perfect. One mundane failure mode is that sometimes, as for example in https://zhd.dev/ghostwriter/?q=numpy.arange, it will point to contextlib.py or Pytest's assertion-rewrite module as the first location at which control-flow diverges.

This isn't wrong, but nor is it helpful, so I propose to hand-exclude a short list of known-to-be-uninformative modules as a postprocessing step. This is conceptually inelegant, but recoups a lot of style points from me as an 80/20 easy-to-implement improvement to the user experience.

Thoughts? Suggestions for more uninformative modules, or systematic ways to build such a list?

@Zac-HD Zac-HD added the legibility make errors helpful and Hypothesis grokable label Aug 12, 2022
@Zac-HD
Copy link
Member Author

Zac-HD commented Aug 13, 2022

Alas, this turns out not to work well at all - I tried a few quick things and quickly wound up playing whack-a-mole with the entire standard library and Pytest internals, and excluding all of those as an experiment just pointed to the @given decorator on my test function. Just going to give up, since the status quo is at least easily interpretable.

@Zac-HD Zac-HD closed this as completed Aug 13, 2022
@Zac-HD
Copy link
Member Author

Zac-HD commented Aug 17, 2022

Wait, no: I'd tried dropping them from the CFG and that didn't work, but I think it would work to calculate the whole report and then drop any report lines which point to such modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
legibility make errors helpful and Hypothesis grokable
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant