Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Our pytest plugin loads all Hypothesis entry points, which may have undesirable side effects #3140

Closed
hugovk opened this issue Nov 8, 2021 · 4 comments · Fixed by #3159
Closed
Labels
bug something is clearly wrong here interop how to play nicely with other packages

Comments

@hugovk
Copy link
Contributor

hugovk commented Nov 8, 2021

If I have Hypothesis installed and run pytest, it always creates .hypothesis/unicode_data/13.0.0/charmap.json.gz, whether the project uses Hypothesis or not.

This then shows up as an untracked file, because the non-Hypotheis project has no reason to put .hypothesis in .gitignore.

Would it be feasible to avoid creating the file, or creating it in a non-local directory?

Alternative

If this is non-trivial, I can easily enough put .hypothesis in a global .gitignore file.

Minimal demo

Python 3.10
pytest
hypothesis 6.24.2

Running pytest in an empty directory without hypothesis installed:

$ mkdir /tmp/123
$ cd /tmp/123
$ pytest
======================================================= test session starts ========================================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /private/tmp/123
plugins: respx-0.18.2, xdist-2.4.0, anyio-3.3.4, timeout-1.4.2, forked-1.3.0, lazy-fixture-0.6.3, cov-3.0.0
collected 0 items

====================================================== no tests ran in 0.02s =======================================================
$ l
total 0
drwxr-xr-x   3 hugo  wheel    96B  8 Nov 10:53 .
drwxrwxrwt  43 root  wheel   1.3K  8 Nov 10:53 ..
drwxr-xr-x   6 hugo  wheel   192B  8 Nov 10:53 .pytest_cache

Installing hypothesis and running pytest:

$ python -m pip install hypothesis
Collecting hypothesis
  Using cached hypothesis-6.24.2-py3-none-any.whl (381 kB)
Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from hypothesis) (2.4.0)
Requirement already satisfied: attrs>=19.2.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from hypothesis) (21.2.0)
Installing collected packages: hypothesis
Successfully installed hypothesis-6.24.2
$ l
total 0
drwxr-xr-x   3 hugo  wheel    96B  8 Nov 10:53 .
drwxrwxrwt  43 root  wheel   1.3K  8 Nov 10:55 ..
drwxr-xr-x   6 hugo  wheel   192B  8 Nov 10:53 .pytest_cache
$ pytest
======================================================= test session starts ========================================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /private/tmp/123
plugins: respx-0.18.2, hypothesis-6.24.2, xdist-2.4.0, anyio-3.3.4, timeout-1.4.2, forked-1.3.0, lazy-fixture-0.6.3, cov-3.0.0
collected 0 items

====================================================== no tests ran in 0.03s =======================================================
$ l
total 0
drwxr-xr-x   4 hugo  wheel   128B  8 Nov 10:55 .
drwxrwxrwt  43 root  wheel   1.3K  8 Nov 10:55 ..
drwxr-xr-x   3 hugo  wheel    96B  8 Nov 10:55 .hypothesis
drwxr-xr-x   6 hugo  wheel   192B  8 Nov 10:53 .pytest_cache
$ l .hypothesis/unicode_data/13.0.0/
total 48
drwxr-xr-x  3 hugo  wheel    96B  8 Nov 10:55 .
drwxr-xr-x  3 hugo  wheel    96B  8 Nov 10:55 ..
-rw-------  1 hugo  wheel    20K  8 Nov 10:55 charmap.json.gz

Bisecting

Bisecting releases, this didn't happen with 5.48.0 and started in 5.49.0.

Bisecting commits, this started in 763e545 from PR #2738.

(Due to the frequency of Hypothesis releases, it was really easy to bisect this!)

Thank you!

@Zac-HD Zac-HD added bug something is clearly wrong here interop how to play nicely with other packages labels Nov 8, 2021
@Zac-HD
Copy link
Member

Zac-HD commented Nov 8, 2021

Thanks for the report! The weird thing here is that we've already designed Hypothesis to avoid creating the .hypothesis directory until we're actually using it, and try as I might I haven't been able to reproduce this - I even wrote a test in #3141 without success.

I think this could happen because you have some package (e.g. Pydantic) with a Hypothesis entry point installed (e.g. Pydantic), and then when pytest loads our pytest-plugin we in turn load that hypothesis-plugin and that's what creates the charmap file. If so, we'll want to defer loading Hypothesis entry points, so that they're triggered on user interaction rather than just running pytest.

The only problem is that this doesn't fit at all with your bisection! I am seriously confused.

@hugovk
Copy link
Contributor Author

hugovk commented Nov 8, 2021

Aha, it is pydantic! Uninstalling pydantic prevents it from happening, re-installing pydantic reproduces it.

(Before this, I uninstalled all pytest-* and those listed by pytest -VV.)

@Zac-HD Zac-HD changed the title Hypothesis always creates .hypothesis/unicode_data/13.0.0/charmap.json.gz when running pytest Our pytest plugin loads all Hypothesis entry points, which may have undesirable side effects Nov 10, 2021
@Zac-HD
Copy link
Member

Zac-HD commented Nov 10, 2021

Looks like we might need to refactor our pytest plugin for lazy-loading of Hypothesis then... it'll be a little annoying, but could be worse!

@hugovk
Copy link
Contributor Author

hugovk commented Nov 28, 2021

Confirmed fixed with hypothesis-6.28.0, pytest-6.2.5 and pydantic-1.8.2:

$ l
total 0
drwxr-xr-x   2 hugo  wheel    64B 28 Nov 15:52 .
drwxrwxrwt  57 root  wheel   1.8K 28 Nov 15:52 ..
$ pytest
========================================================= test session starts =========================================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /private/tmp/123
plugins: respx-0.19.0, freezegun-0.4.2, hypothesis-6.28.0, flaky-3.7.0, xdist-2.4.0, timeout-2.0.1, anyio-3.3.4, forked-1.3.0, cov-3.0.0
collected 0 items

======================================================== no tests ran in 0.03s ========================================================
$ l
total 0
drwxr-xr-x   3 hugo  wheel    96B 28 Nov 15:52 .
drwxrwxrwt  57 root  wheel   1.8K 28 Nov 15:52 ..
drwxr-xr-x   6 hugo  wheel   192B 28 Nov 15:52 .pytest_cache

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug something is clearly wrong here interop how to play nicely with other packages
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants