Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: tests/test_variants.py::test_variants_different_dtypes randomly fail with hypothesis.errors.FailedHealthCheck: Examples routinely exceeded the max allowable size #141

Open
mgorny opened this issue Feb 28, 2024 · 0 comments

Comments

@mgorny
Copy link

mgorny commented Feb 28, 2024

When running the Python test suite, I'm frequently getting health check failures on different tests. For example:

$ python -m pytest tests/test_variants.py::test_variants_different_dtypes
========================================================= test session starts =========================================================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0
rootdir: /tmp/cramjam/cramjam-python
plugins: hypothesis-6.98.13, xdist-3.5.0
collected 8 items                                                                                                                     

tests/test_variants.py FF......                                                                                                 [100%]

============================================================== FAILURES ===============================================================
_______________________________________________ test_variants_different_dtypes[snappy] ________________________________________________

variant_str = 'snappy'

    @pytest.mark.parametrize("variant_str", VARIANTS)
>   @given(arr=st_np.arrays(st_np.scalar_dtypes(), shape=st.integers(0, int(1e4))))
E   hypothesis.errors.FailedHealthCheck: Examples routinely exceeded the max allowable size. (20 examples overran while generating 8 valid ones). Generating examples this large will usually lead to bad results. You could try setting max_size parameters on your collections and turning max_leaves down on recursive() calls.
E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.data_too_large to the suppress_health_check settings for this test.

tests/test_variants.py:42: FailedHealthCheck
------------------------------------------------------------- Hypothesis --------------------------------------------------------------
You can add @seed(297719150791330741877614251129208577971) to this test or run pytest with --hypothesis-seed=297719150791330741877614251129208577971 to reproduce this failure.
_______________________________________________ test_variants_different_dtypes[brotli] ________________________________________________

variant_str = 'brotli'

    @pytest.mark.parametrize("variant_str", VARIANTS)
>   @given(arr=st_np.arrays(st_np.scalar_dtypes(), shape=st.integers(0, int(1e4))))
E   hypothesis.errors.FailedHealthCheck: Examples routinely exceeded the max allowable size. (20 examples overran while generating 9 valid ones). Generating examples this large will usually lead to bad results. You could try setting max_size parameters on your collections and turning max_leaves down on recursive() calls.
E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.data_too_large to the suppress_health_check settings for this test.

tests/test_variants.py:42: FailedHealthCheck
------------------------------------------------------------- Hypothesis --------------------------------------------------------------
You can add @seed(151523981063034797703438801667290859669) to this test or run pytest with --hypothesis-seed=151523981063034797703438801667290859669 to reproduce this failure.
======================================================= short test summary info =======================================================
FAILED tests/test_variants.py::test_variants_different_dtypes[snappy] - hypothesis.errors.FailedHealthCheck: Examples routinely exceeded the max allowable size. (20 examples overran while generating 8 v...
FAILED tests/test_variants.py::test_variants_different_dtypes[brotli] - hypothesis.errors.FailedHealthCheck: Examples routinely exceeded the max allowable size. (20 examples overran while generating 9 v...
=============================================== 2 failed, 6 passed in 153.35s (0:02:33) ===============================================

I can reproduce with 496c1ab.

My reproducer:

pip install . pytest pytest-xdist hypothesis numpy
python -m pytest tests/test_variants.py::test_variants_different_dtypes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant