Bypass small examples in stateful testing

[mqvist]

I am using the stateful testing features of Hypothesis. I have the issue that Hypothesis always generates ~50 identical very simple test cases initially. These tests are not useful as they contain no variation.

I checked Hypothesis source code and found the "small_example_cap" in engine.py generate_new_examples() function, that seems to be responsible for these 50 cases. Its based on the "max_examples" setting, but it cannot be set directly.

It would be nice to be able to bypass and configure the amount of small examples in stateful tests.

[honno]

If possible, mind sharing the code being used? It might be the way the strategy is constructed for your test leads itself to generating uninteresting/repetitive examples. Also if you could share the terminal output when you run your tests with the flags --hypothesis-verbosity=verbose -s, so we can see the examples generated.

These tests are not useful as they contain no variation.

Have examples repeat can be useful, namely as it can catch bugs related to states, i.e. if you pass the same arg twice to a function and unexpectedly get different results. But yeah if it's a ton of repetition than something needs addressing.

[mqvist]

Unfortunately I cannot share the code as it's proprietary. To give a rough idea, the test setup consists of a process written in C++ and several test mocks that interface with the process via different protocols. The stateful tests communicate with the mocks to produce different test actions and read back outputs from the process.

One thing I also noticed is that the search strategy used by RuleBasedStateMachine orders the rules based on their "complexity", i.e., number of inputs and preconditions etc. This ordering may also play role here, as the initial repetitions consist of the simplest rule that has no parameters and, therefore, does not produce any variations.

I agree that some amount of repetition can be beneficial but seeing those same 50 cases every time makes one wonder it they could be skipped or at least reduced to just a few cases. Also running the 50 cases currently takes one to two minutes, so they are a bit painful to watch go by.

[mqvist]

Just to experiment, I changed the small_example_cap to zero in engine.py. This generated more interesting examples almost immediately. It would really be nice to have a way to control the small_example_cap from the Hypothesis API.

[Zac-HD]

The trouble is that small_example_cap exists because not having it causes problems in other cases. If you can provide a sanitized version of your stateful test we can try to find a solution that works for everyone; but as-is there's not much we can do.

I suspect that #3086 will also help, by improving our ability to detect redundant outputs.

[mqvist]

Thank you for the response. My suggestion was to allow control of small_example_cap, for example, by adding it to the settings. I will check #3086.

[Zac-HD]

My suggestion was to allow control of small_example_cap, for example, by adding it to the settings.

We're pretty strongly against exposing such internals though the public API. It's a pretty nasty abstraction-violation which makes the library a lot harder to learn and use, and also holds back future development via compatbility constraints.

[mqvist]

Could you add a setting that would allow indirect control of the amount of small examples? Something like no_small_examples? Then it would be possible to keep the implementation details hidden, and even deprecate the setting at some point? I could event try to create a PR for this myself.

[Zac-HD]

No, I will decline any PR adding such a setting. At minimum, every new setting adds to the size of our docs and makes Hypothesis slightly harder to understand (or be confident that you understand).

Again, if you can create a non-sensitive test that demonstrates this issue I'm confident that some other mitigations are possible; including some which are already on the issue tracker. (though, as a volunteer-driven OSS project, not on a roadmap or timeline)