Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perform health checks for each component of a strategy #1007

Closed
Zac-HD opened this issue Dec 2, 2017 · 5 comments
Closed

Perform health checks for each component of a strategy #1007

Zac-HD opened this issue Dec 2, 2017 · 5 comments
Labels
legibility make errors helpful and Hypothesis grokable

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Dec 2, 2017

As part of our eternal quest to stop people (accidentally) misusing Hypothesis, we have a number of health checks to warn about strategies that will have poor shrinking behavior or are impossible to draw from.

A possibly more fruitful approach is that we do a bunch of random generation that basically inserts this approach [trying the minimal example] on the first draw call to every strategy it sees. We'd need some decently reliable way of distinguishing strategies (I've been vaguely meaning to do something about this with a hashing scheme but haven't worked through the details). At any rate, I think this is definitely too big a change for now.

This issue is meant to keep track of that ambition.

@Zac-HD Zac-HD added the enhancement it's not broken, but we want it to be better label Dec 2, 2017
@Zac-HD
Copy link
Member Author

Zac-HD commented Feb 14, 2018

#1112 is a real-life example of too much filtering in a sub-strategy (field for a Django model).

@Zac-HD
Copy link
Member Author

Zac-HD commented Feb 25, 2018

A closely related problem is that the error messages for failed healthchecks do not show the strategy which failed the check!

hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data.

Adding the repr of the relevant strategy would probably be helpful right now, but even more so when we can run this check on smaller parts and discover the minimal unhealthy components.

@Zac-HD Zac-HD added performance go faster! use less memory! legibility make errors helpful and Hypothesis grokable and removed performance go faster! use less memory! enhancement it's not broken, but we want it to be better labels Feb 25, 2018
@Zac-HD
Copy link
Member Author

Zac-HD commented Feb 25, 2018

Overlaps substantially with #434, which proposes yet another solution.

@Zac-HD
Copy link
Member Author

Zac-HD commented Mar 8, 2018

#1153 would also be easier to solve with more granular reporting of health check failures.

@Zac-HD
Copy link
Member Author

Zac-HD commented Oct 12, 2019

Post-#2030, I'm closing this as the remaining parts are better described by #434.

@Zac-HD Zac-HD closed this as completed Oct 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
legibility make errors helpful and Hypothesis grokable
Projects
None yet
Development

No branches or pull requests

1 participant