-
Notifications
You must be signed in to change notification settings - Fork 575
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix a swarm-testing footgun #3894
Conversation
Following patch reproduces (some of?) the ci failures consistently, e.g. diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py b/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py
index 1e321e744..4b1e90e5b 100644
--- a/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py
@@ -53,6 +53,7 @@ class FeatureFlags:
# of more features being enabled.
if self.__data is not None:
self.__p_disabled = data.draw_integer(0, 255) / 255.0
+ self.__p_disabled = 1
else:
# If data is None we're in example mode so all that matters is the
# enabled/disabled lists above. We set this up so that everything since we're forcing false but drawing true with probability 1. Possibly __p_disabled should be 254 / 255? |
Found a weird one while debugging this. If your stateful test sets a reproducerfrom hypothesis.stateful import RuleBasedStateMachine, invariant, rule
class NumberModifier(RuleBasedStateMachine):
step_count = 0
@rule()
def count_step(self):
self.step_count += 1
@invariant()
def divide_with_one(self):
assert self.step_count % 2 == 0
TestCase = NumberModifier.TestCase
import unittest
unittest.main() |
0d72399
to
16d0ee7
Compare
Nice, I accidentally force-pushed over your commit (wish I could alias https://github.com/HypothesisWorks/hypothesis/actions/runs/8028323348/job/21933383800?pr=3894#step:7:98 is unrelated, but looks like it might be an IR flake? I think I've seen similar things one or twice before. |
16d0ee7
to
77f596f
Compare
Interesting, I'll take a look. May turn out to need the same treatment as |
So, you know how we used swarm testing to choose a random subset of rules to enable on a
RuleBasedStateMachine
? Turns out that we could choose the empty set, which promptly fails the test because it's impossible to choose a rule from the empty set. Oops? This was still pretty rare even with a single-rule machine, but also a totally self-inflicted problem.Also improves some reprs so that the
status_reason
in observability output looks nicer for stateful testing, closing #3845.Also also fixes #3892, without docs because it's just improving the error message for something that already didn't work.