Running under xdist causes FileNotFoundError

[daneah]

Hey! I've just gotten a big project onto pytest and now I'm working on speeding up the test suite. Running an otherwise working suite using pytest-xdist with any number of workers causes an error via pytest-django when Django tries to load PyYAML for serialization:

.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/pytest_django/fixtures.py:108: in django_db_setup
    **setup_databases_args
.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/test/utils.py:187: in setup_databases
    serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True),
.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/db/backends/base/creation.py:77: in create_test_db
    self.connection._test_serialized_contents = self.serialize_db_to_string()
.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/db/backends/base/creation.py:121: in serialize_db_to_string
    serializers.serialize("json", get_objects(), indent=None, stream=out)
.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/core/serializers/__init__.py:128: in serialize
    s = get_serializer(format)()
.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/core/serializers/__init__.py:97: in get_serializer
    _load_serializers()
.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/core/serializers/__init__.py:153: in _load_serializers
    register_serializer(format, BUILTIN_SERIALIZERS[format], serializers)
.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/core/serializers/__init__.py:71: in register_serializer
    module = importlib.import_module(serializer_module)
.../.pyenv/versions/3.4.3/lib/python3.4/importlib/__init__.py:109: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    """

    import collections
    import decimal
    import sys
    from io import StringIO

>   import yaml
E   FileNotFoundError: [Errno 2] No such file or directory

/.../.pyenv/versions/3.4.3/envs/jstor/lib/python3.4/site-packages/django/core/serializers/pyyaml.py:12: FileNotFoundError

Django's docs state:

This serializer is only available if PyYAML is installed.

But this occurs whether PyYAML is installed or not. I'm filing here because it feels like a variation pytest-django is creating in the presence of xdist, but I'd be happy to file in pytest-xdist if it appears more likely to be implicated. Thanks for any help you can provide!

Python: 3.4.3
Django: 1.11.20
pytest: 4.4.1
pytest-django: 3.4.8
pytest-xdist: 1.28.0
PyYAML: 5.1.0
Databases: Postgres and MySQL

[blueyed]

I guess this is related to pyyaml being available on the host, but not the nodes?
How do you run xdist?

[daneah]

At this point I was just running it on my local (macOS) machine, using e.g. pytest -n 2!

[daneah]

I tried bypassing that particular execution path altogether by setting SERIALIZE to False in each database configuration, but I'm still seeing other tests where _gcd_import ultimately raises a FileNotFoundError as well, not in Django code. For that reason, it seems like this may need to move over to pytest-xdist. What are your thoughts?

[blueyed]

Try upgrading Python.
It reminded me of pytest-dev/py#207.

[daneah]

I will try it, although unfortunately we're pretty well stuck on 3.4.3 for the moment for infrastructure reasons. It'll at least be a good data point—that issue does look related. I'll report back, hopefully today or tomorrow.

[daneah]

Renamed the issue to be more general and better correlate those things

[daneah]

Upgrading to 3.5 appears to have mitigated this issue, so it looks like that's the issue. Thanks for pointing me in that direction!