Allow specifying max number of shrinks per test

[AlexeyRaga]

🚀 Feature Request

Allow specifying max number of shrinks.
The idea is not new, other property testing framework (QuickCheck,HedgeHog, FSCheck in Haskell and FSharp have it)

Motivation

Performance and friendliness with integration tests.
Also related to #4161.

Example

fc.assert(myProperty, { numRuns: 5, maxShrinks: 5 });

[dubzzz]

Any example of such option in the other libraries? How it can be configured compared to another test...?

On it's own reducing shrinker can be done at various level: the arbitrary itself (what we do with noShrink) vs the runner (what you suggest). It can also be done at different levels: how far it goes in a given level? how many levels it can run into? As actually the shrinker runs on a 2D-plan so we may want to limit one direction and not the other or a combination of both of them.

Having more context would help as it seems that you know how they do it might reduce the exploratory time required just to put a status (do/don't do) on such ticket.

---

Side-note: You may use noShrink or endOnFailure.

Like:

fc.uuid().noShrink()

or:

fc.assert(myProperty, { numRuns: 5, endOnFailure: true })

[AlexeyRaga]

@dubzzz sure,

Hedgehog: https://hackage.haskell.org/package/hedgehog-1.4/docs/Hedgehog.html#v:withShrinks
QuickCheck in Haskell: https://hackage.haskell.org/package/QuickCheck-2.14.3/docs/Test-QuickCheck.html#t:Args

Alternatives are not really the same because I do not really want to give up on shrinking...

[dubzzz]

As we already have noShrink, I think I'll try to find an option that could cover both noShrink and maxShrinks at the same time. Probably re-using noShrink but passing it a number for v3 and move to another name for v4.

The remaining part to decide is: what to stop shrinking as shrinker lives in a 2D-plan so 2 directions can be stopped separately or together.

[dubzzz]

Maybe the existing noShrink should be replaced by some kind of limitShrink({ maxDepth, maxValuesPerLevel }) or a simpler one limitShrink(maxValues) with maxValues somehow aggregating the two others.

With such option, noShrink could just be dropped in V4.

Another option would be to make a dedicated arbitrary to handle that.

[dubzzz]

I re-checked this issue as I'm planning to solve it in the next major (on it). The solution I currently came up with is to handle it at Arbitrary level as I already do for the noShrink case.

The idea is the following:

• the Value, more precisely it's attached context has to integrate the depth
• based on that the shrink can decide

It has the huge benefits not to slow down nominal users, to be implementable on user land too... But I'll need to assess the case in which the shrink gets called without any context. It can be the case when nesting arbitraries to re-use their shrink capabilities.

The real huge benefit of the approach is that it fully fulfills my plugin-based target. As for now context is untyped it does bring any concerns related to mapping on an existing context to extend it with extra pieces of data.

[moodmosaic]

Very interesting 👍 Essentially, endOnFailure: true is like a global way of disable shrinking, right?

[dubzzz]

endOnFailure is even more powerful than disabling shrinking actually. In classic usages of fast-check, it behaves fully the same with shrink being cut. But in usages such as reproducing an issue based on a known path it does not really stop shrinking, as it does allow it until you reach the target of the path and then forbid it.

endOnFailure: true can be seen as: stop immediately the execution if you fall on failure. An alternative less aggressive way to limit shrinking is probably interruptAfterTimeLimit. The later allows users to still benefit from shrinking capabilities while not spending too many time into it. Both options can be seen as global ones as they can be set for all tests using configureGlobal.