Deprecating event_loop fixture overrides

[seifertm]

Motivation

pytest-asyncio provides an asyncio event loop via the event_loop fixture. The fixture is function-scoped by default. This is useful for unit testing as it ensures a high level of isolation between test cases. However, as soon as users want to write larger tests that span multiple test cases the scope of the event_loop fixture needs to be extended.

Extending the scope of the event_loop fixture is currently achieved by reimplementing the fixture with an appropriate scope. This approach leads to code duplication, because users are forced to implement the fixture body over and over again. Users are also known to modify the fixture implementation and extend it with custom setup or teardown code.

Several problems arise from this: For one, changes to the fixture implementation in pytest-asyncio would require all users to change their fixture implementations accordingly. For another, requiring users to implement their own event_loop fixture somewhat defeats the purpose of pytest-asyncio in the first place.

Proposed solution

Pytest nodes are discovered by a hierarchy of collectors. Pytest marks can be located at different levels of this hierarchy (see Marking whole classes or modules).

The proposed solution is to derive the scope of the asyncio event loop from the location of the asyncio mark. Consider the following examples.

Strict mode

Test functions

@pytest.mark.asyncio
async def test_a():
    ...

@pytest.mark.asyncio
async def test_b():
    ...

Each function test_a and test_b have their own asyncio mark. Therefore, the event loop scope will be limited to each test function.

Test classes

When using test classes the scope of the event loop can be per-class or per-function, depending on the location of
the asyncio mark.

class MyTestsInFunctionScopedLoop:
    @pytest.mark.asyncio
    async def test_a(self):
        ...

    @pytest.mark.asyncio
    async def test_b(self):
        ...

The functions test_a and test_b each run in their own loop, because the asyncio marker is directly attached to them.

@pytest.mark.asyncio
class MyTestsInClassScopedLoop:
    async def test_a(self):
        ...

    async def test_b(self):
        ...

The functions test_a and test_b run in a common loop, because the asyncio marker is attached to the class.

Module-scoped loops

If the entire module has the asyncio mark all tests are run in a module-scoped event loop.

pytestmark = pytest.mark.asyncio

async def test_a():
    ...

class MyTests:
    async def test_b(self):
        ...

Auto mode

Function scope

async def test_a():
    ...

Class scope

@pytest.mark.asyncio
class MyTests:
    async def test_a(self):
        ...

Module scope

pytestmark = pytest.mark.asyncio

async def test_a(self):
    ...

Session scope

pytestmark = pytest.mark.asyncio # in root conftest.py

async def test_a(self):
    ...

Considered alternatives

Dynamic fixture scopes

pytest-dev/pytest#6101

Introducing dynamic fixture scope in pytest is more effort compared to the proposed solution as it introduces new features to pytest.

[Tinche]

That answers scope (although switching to global by default from function by default is a raw deal), but what if I want to test on uvloop?

[seifertm]

While you cannot change the event loop without the fixture override, you could change the event loop policy to return a different type of loop. Testing on uvloop should be as easy as setting the event loop policy. According to their documentation:

import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

To my knowledge, pytest-asyncio respects existing event loop policies as of v0.17.

(although switching to global by default from function by default is a raw deal)

Can explain what you mean by that? I have the feeling you're seeing some issue that I missed :)

[Tinche]

For uvloop: where would I put this code though? In an autouse global fixture I guess? We'd need to document this I suppose.

For the default scope: I think the default should be function-scoped loops, right? Testing frameworks should default to best test design choices.

[seifertm]

I totally agree, the default scope for the event loop should remain at function scope. Only when the user wants to change the default scope to something else, they need to take action. I amended the original post of the discussion with examples for pytest-asyncio auto mode.

The loop policy for uvloop can either be set in conftest.py as in the example above or in a fixture as you suggested. When using a fixture, it should even be possible to limit the loop policy in scope. That means you should be able to test one module with uvloop and another with a different event loop.

I second you opinion for documenting this. We can even do that now, regardless of the outcome of this discussion.

[schlamar]

To my knowledge, pytest-asyncio respects existing event loop policies as of v0.17.

Not really, see #591

[seifertm]

I opened a draft PR with a proof-of-concept for class-scoped loops (#620).

It's probably not a good idea to change the semantics of the existing asyncio mark as my initial post intended. Firstly, the pytest docs describe markers as being intended for test functions. Applying a marker to a class or module simply applies that marker to all test functions. Internally, we could still use markers, but it may be a better idea to them, e.g. pytest_asyncio.loop = pytest.mark.XYZ.

Secondly, changing the semantics of the existing asyncio mark will introduce more breaking changes,making it more difficult for users to migrate.

[seifertm]

The proposal did not account for parametrized event loops which have been added in v0.18.
I'm currently looking into a way to add this functionality to the new mark, for example:

@pytest.mark.asyncio_event_loop(
    policy=[
        asyncio.DefaultEventLoopPolicy,
        uvloop.EventLoopPolicy(),
    ]
)
class TestClass:
    ...

[ramnes]

We're using the auto mode and async fixtures that are session-scoped. Latest pytest-asyncio tells us that Replacing the event_loop fixture with a custom implementation is deprecated and will lead to errors in the future. (...)

I tried to remove the warning by removing our session-scoped event_loop fixture and adding pytestmark = pytest.mark.asyncio (as mentioned here) or pytestmark = pytest.mark.asyncio_event_loop to the root conftest.py file, but it gives back ScopeMismatch errors. What am I missing?

[seifertm]

Adding pytestmark = pytest.mark.asyncio_event_loop to the root conftest.py will give a module-scoped loop in the conftest module.

Apparently, there are quite a few people depending on session-scoped loops and the new release doesn't provide appropriate replacement functionality for the deprecation (see #657). I yanked the v0.22 release from PyPI to avoid wasting people's time. Sorry for the hassle.

Do you mind outlining your use case for a session scoped loop and why it cannot be achieved with a module-scoped loop?

[ramnes]

Basically setting up and setting down a database. Definitely works at the module level as well but it takes more time for no gain.

[freywaid]

Yep. Same for me. I want to setup/teardown db in a session-scoped (autouse) fixture.

[freywaid]

Any resolution for this? Or I guess I could write a work around to only setup/teardown DB once in a module-scoped fixture.