Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve explain-mode performance #3015

Merged
merged 5 commits into from Jun 9, 2021

Conversation

Zac-HD
Copy link
Member

@Zac-HD Zac-HD commented Jun 9, 2021

This set of patches makes a few related changes to explain-mode, aimed at improving the test-time and user-facing performance. I think this is the best fix #2989 is going to get.

  • Move all the not-run-under-coverage tests to nocover (duh)
  • Use __slots__ and some micro-optimisations in the Tracer class
  • Only run the tracer if shrinking is enabled, though still starting immediately after the first failure (i.e. often in keep-generating mode)
  • Disable the tracer immediately if a DeadlineExceeded error is raised. The performance overhead of tracing interacts badly with deadlines, and "explaining" it in terms of some line of code is almost certainly wrong anyway.

@Zac-HD Zac-HD added tests/build/CI about testing or deployment *of* Hypothesis performance go faster! use less memory! legibility make errors helpful and Hypothesis grokable labels Jun 9, 2021
@Zac-HD Zac-HD requested a review from Zalathar June 9, 2021 02:57
Small improvements, but this is called very very often while we're tracing and so even micro-optimisations are worth it.
For two reasons: our analysis is just not relevant to deadline exceeded failures, and running under a tracer adds some non-representative slowdowns.

Note that this code handles the case where we first discover an unrelated bug, start shrinking, and then find a deadline-exceeding input.
With shrinking disabled you get *a* result, but the "always observed for failing examples" part is pretty vacuous.
@Zac-HD Zac-HD merged commit 014f1b8 into HypothesisWorks:master Jun 9, 2021
@Zac-HD Zac-HD deleted the tune-scrutineer branch June 9, 2021 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
legibility make errors helpful and Hypothesis grokable performance go faster! use less memory! tests/build/CI about testing or deployment *of* Hypothesis
Projects
None yet
Development

Successfully merging this pull request may close these issues.

@settings(phases=) API makes it very easy to accidentally enable Phase.explain
2 participants