More difficulties in 0.23 with async fixtures

[bmerry]

This might have the same underlying cause as #705 / #706, but I didn't see the specific error message I'm running into so I thought I should open a separate bug.

To reproduce:

conftest.py (copied verbatim from https://pytest-asyncio.readthedocs.io/en/latest/how-to-guides/run_session_tests_in_same_loop.html):

import pytest

from pytest_asyncio import is_async_test


def pytest_collection_modifyitems(items):
    pytest_asyncio_tests = (item for item in items if is_async_test(item))
    session_scope_marker = pytest.mark.asyncio(scope="session")
    for async_test in pytest_asyncio_tests:
        async_test.add_marker(session_scope_marker)

foo/__init__.py: empty

foo/test_foo.py:

import pytest_asyncio


@pytest_asyncio.fixture(scope="package")
async def bar() -> int:
    return 3

async def test_stuff(bar: int) -> None:
    pass

Gives this error:

==================================== ERRORS ====================================
_________________________ ERROR at setup of test_stuff _________________________
file /home/bmerry/work/sdp/bugs/pytest-asyncio-session/foo/test_foo.py, line 8
  async def test_stuff(bar: int) -> None:
      pass
file /home/bmerry/work/sdp/bugs/pytest-asyncio-session/foo/test_foo.py, line 4
  @pytest_asyncio.fixture(scope="package")
  async def bar() -> int:
      return 3
E       fixture 'foo/__init__.py::<event_loop>' not found
>       available fixtures: __pytest_repeat_step_number, _session_event_loop, anyio_backend, anyio_backend_name, anyio_backend_options, bar, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, check, class_mocker, cov, doctest_namespace, event_loop, event_loop_policy, foo/test_foo.py::<event_loop>, mocker, module_mocker, monkeypatch, no_cover, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, unused_tcp_port, unused_tcp_port_factory, unused_udp_port, unused_udp_port_factory, worker_id
>       use 'pytest --fixtures [testpath]' for help on them.

Python 3.10.12, pytest-asyncio 0.23.2.

[bmerry]

I forgot to include the pytest.ini:

[pytest]
asyncio_mode = auto

[seifertm]

Thanks for the reproducer. Good call for opening a separate issue!

[nzeid]

Let me know if this isn't the best place to post this, but I have a gut feeling it's very closely related to the sloppiness of fixture scopes and event loop initialization.

I was busting my head all day trying to refactor my code to get rid of the def event_loop override per version 0.23. I could not for the life of me get function-scoped asyncio fixtures to work on session-scoped asyncio test functions with simple markers/decorators. I would almost always end up with the dreaded MultipleEventLoopsRequestedError. I can understand why multiple event loops are prohibited, but I can't understand why fixtures renewing at different cadences condemns them to live in a separate event loop. This is absolutely a must-have.

For anyone who happens upon this, the workaround for now is as follows:

# This allows all fixtures to access the session-scoped event loop:
@pytest_asyncio.fixture(scope="session")
async def global_event_loop():
    return asyncio.get_running_loop()


# The function of the async fixture you want to create, but without the marker:
async def clear_db():
    async with Transaction() as tx:
        await tx.execute(sql.text("DELETE FROM some_table"))


# Use this non-async fixture to mimic the behavior of the async fixture you need.
# Run the async code using global_event_loop.
@pytest.fixture(autouse=True, scope="function")
def clear_db_wrapper(global_event_loop):
    yield
    global_event_loop.run_until_complete(clear_db())

(Edit: Fixed and clarified code.)

[seifertm]

@nzeid I'm sorry that you had to go through so much trouble. Pytest-asyncio v0.23 supports running tests in different event loops. There's a loop for each level of the test suite (session, package, module, class, function). However, pytest-asyncio wrongly assumes that the scope of an async fixture is the same as the scope of the event loop in which the fixture runs. This essentially makes it impossible to have a session-scoped fixture that runs in the same loop as a function-scoped test. See #706.

[av223119]

I'm experiencing the similar issue, which I believe is another manifestation of #706.

Namely, the error message fixture 'test_something.py::<event_loop>' not found which, unexpectedly, is generated when tests from test_something_else.py being run. The only thing these two modules have in common is that they use one shared module-scoped fixture from conftest.py. It appears that pytest-asyncio tried to use the event loop from the first module while running the second one. It looks awfully similar to #829 which, in turn, I believe is just another manifestation of #706

[iamWing]

@seifertm Hi, I'm trying to use the async fixtures at class scope and I'm having issues running the test module when there're multiple classes in that module. Here some of my test code

@pytest_asyncio.fixture(scope="class")
async def grpc_server() -> AsyncGenerator[(grpc.aio.Server, int), None]:
    global _KEYSTORE, _API_KEY_MANAGER_SERVICER

    server = grpc.aio.server()
    # add service

    port = server.add_insecure_port("[::]:0")

    await server.start()

    yield (server, port)

    await server.stop(None)

   # more clean up actions


@pytest_asyncio.fixture(scope="class")
async def grpc_stub(
    grpc_server: tuple[grpc.aio.Server, int],
):
    server, port = grpc_server

    channel = grpc.aio.insecure_channel(f"localhost:{port}")

    await channel.channel_ready()

    stub = api_key_manager_pb2_grpc.ApiKeyManagerStub(channel)

    yield stub

    await channel.close()


class TestExportServiceConfig: # non async test class
    _expected_json_str = """{
  "free_tier_daily_usage_limit": 1,
  "premium_tier_daily_usage_limit": -1
}"""

    @pytest.fixture()
    def json_path(self, tmp_path: pathlib.Path) -> str:
        return tmp_path / "service-config.json"

    def test_export_success(self, json_path: pathlib.Path):
        with does_not_raise():
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(json_path)

        with open(json_path, "r") as file:
            assert file.read() == self._expected_json_str

    def test_export_fail(self, tmp_path: pathlib.Path):
        with pytest.raises(OSError):
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(
                tmp_path / "nonexistent" / "file.json"
            )


@pytest.mark.asyncio(scope="class")
class TestGetServiceConfig: # async test class
    async def test_get_service_config(
        self, grpc_stub: api_key_manager_pb2_grpc.ApiKeyManagerStub
    ):
        expected_response = api_key_manager_pb2.GetServiceConfigResponse(
            config=api_key_manager_pb2.ApiKeyManagerServiceConfig(
                free_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "free_tier_daily_usage_limit"
                ],
                premium_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "premium_tier_daily_usage_limit"
                ],
            )
        )

        call: grpc.aio.Call = grpc_stub.GetServiceConfig(empty_pb2.Empty())

        response = await call

        assert response == expected_response
        assert await call.code() == grpc.StatusCode.OK

...

It works when I run pytest with just one class/specify a class to test, but when I run it as a module where there're multiple classes, async or not, I get the error of fixture 'something.py::<event_loop>' not found.

I'm not entirely sure if my issue is related to this thread but it seems to be the case. Tried downgrade to 0.21 directly without changing the code but that doesn't work for me. I guess the pytest.mark.asyncio(scope="value") syntax works differently between 0.21 & 0.23?

Is there any workaround I can implement to make my use case workable? Right now I can run the test on individual class but it makes it not possible for the test module to pass the CI/CD pipeline I've setup. Thanks!

Edit: Had a quick look through on the open issues and seems like my issue if also related to #829. Will come back and see if it works after I tried putting those class scope fixtures in the classes.

[iamWing]

@seifertm Hi, I'm trying to use the async fixtures at class scope and I'm having issues running the test module when there're multiple classes in that module. Here some of my test code

@pytest_asyncio.fixture(scope="class")
async def grpc_server() -> AsyncGenerator[(grpc.aio.Server, int), None]:
    global _KEYSTORE, _API_KEY_MANAGER_SERVICER

    server = grpc.aio.server()
    # add service

    port = server.add_insecure_port("[::]:0")

    await server.start()

    yield (server, port)

    await server.stop(None)

   # more clean up actions


@pytest_asyncio.fixture(scope="class")
async def grpc_stub(
    grpc_server: tuple[grpc.aio.Server, int],
):
    server, port = grpc_server

    channel = grpc.aio.insecure_channel(f"localhost:{port}")

    await channel.channel_ready()

    stub = api_key_manager_pb2_grpc.ApiKeyManagerStub(channel)

    yield stub

    await channel.close()


class TestExportServiceConfig: # non async test class
    _expected_json_str = """{
  "free_tier_daily_usage_limit": 1,
  "premium_tier_daily_usage_limit": -1
}"""

    @pytest.fixture()
    def json_path(self, tmp_path: pathlib.Path) -> str:
        return tmp_path / "service-config.json"

    def test_export_success(self, json_path: pathlib.Path):
        with does_not_raise():
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(json_path)

        with open(json_path, "r") as file:
            assert file.read() == self._expected_json_str

    def test_export_fail(self, tmp_path: pathlib.Path):
        with pytest.raises(OSError):
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(
                tmp_path / "nonexistent" / "file.json"
            )


@pytest.mark.asyncio(scope="class")
class TestGetServiceConfig: # async test class
    async def test_get_service_config(
        self, grpc_stub: api_key_manager_pb2_grpc.ApiKeyManagerStub
    ):
        expected_response = api_key_manager_pb2.GetServiceConfigResponse(
            config=api_key_manager_pb2.ApiKeyManagerServiceConfig(
                free_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "free_tier_daily_usage_limit"
                ],
                premium_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "premium_tier_daily_usage_limit"
                ],
            )
        )

        call: grpc.aio.Call = grpc_stub.GetServiceConfig(empty_pb2.Empty())

        response = await call

        assert response == expected_response
        assert await call.code() == grpc.StatusCode.OK

...

It works when I run pytest with just one class/specify a class to test, but when I run it as a module where there're multiple classes, async or not, I get the error of fixture 'something.py::<event_loop>' not found.

I'm not entirely sure if my issue is related to this thread but it seems to be the case. Tried downgrade to 0.21 directly without changing the code but that doesn't work for me. I guess the pytest.mark.asyncio(scope="value") syntax works differently between 0.21 & 0.23?

Is there any workaround I can implement to make my use case workable? Right now I can run the test on individual class but it makes it not possible for the test module to pass the CI/CD pipeline I've setup. Thanks!

Edit: Had a quick look through on the open issues and seems like my issue if also related to #829. Will come back and see if it works after I tried putting those class scope fixtures in the classes.

Ended up commented out the scope=class lines and use the fixtures on function scope instead. Not ideal but it minimised the effort needed for refactoring