[Feature]: Dynamic test retries based on error messages

[airhorns]

🚀 Feature Proposal

It'd be really nice to be able to selectively retry tests that fail with specific error messages. It's expensive and slow to blindly retry everything, but if you know you have a flake in your CI, it'd be great to overcome it by targeting it specifically and retrying only when it occurs.

Motivation

To help suite reliability, it's nice to retry tests that you know may be flakey. If flake is localized to one test or one suite, it's easy to slap a jest.retryTimes on that suite and be done with it. But, if the flake comes from infrastructure (as it often does), many different suites might use that infrastructure and flake through no fault of their own. Adding a jest.retryTimes to all those suites would work, but can get really expensive in CI if you have some systematic real reason why all those tests are failing that isn't the flake.

Instead of blindly retrying all tests always, it'd be really nice to retry tests that fail with only a certain error message matching a regexp or similar.

Example

// in a global setup
jest.retryTimes(2, { only: [/RST_STREAM_ERROR: some grpc dependency has a flake/] })

Pitch

I can't currently do this from a custom environment as jest-circus is hardcoded to inspect the global retry config once before starting. It can't be adjusted on a per-test basis, or adjusted once the suite has started running. An alternative would be to add or extend the runtime api for adjusting retries to allow it to be called by custom environments, but by-error-message filtering seems common enough to me to include in core.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.