New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve performance of small tests that use rejection sampling #2030
Conversation
7040de6
to
be14bad
Compare
Despite the failing build, this should actually pass and be ready for review. It's just failing because Cloudflare issues are breaking most of the internet right now. |
be14bad
to
7579868
Compare
@@ -70,6 +70,7 @@ def test_unsat_sets_of_samples(x): | |||
assert False | |||
|
|||
|
|||
@settings(suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not really comfortable with this since it seems to defeat the purpose of the test, but after merging #2031 and rebasing I don't think it will be needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, agreed. I think it's not totally defeating the point of the test, but it definitely weakens it. Either way, it's no longer necessary.
7579868
to
903e1cc
Compare
903e1cc
to
1b0867c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Fixes #1864.
I think this problem must always have been latent in Hypothesis but the previous implementation of the tree had a bug which masked it somehow, because the issue is actually intrinsic to what we were doing: The rejection sampling on small state spaces forces the necessary novel prefix to be progressively longer and longer, creating an accidentally quadratic problem where generating the Nth example takes O(N) time.
This PR fixes it by implicitly capping all rejection sampling loops to 20 iterations by building logic for marking longer sequences of consecutive discards as invalid into
ConjectureData
.Earlier versions of this PR had an independent performance improvement to
generate_novel_prefix
which turned out to change the distribution enough that I decided to pull it out for now (it causes us to miss some branches in coverage that we were previously hitting).