metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations #3542

kamichal · 2018-06-06T09:08:04Z

Both, python 2.7.13 and 3.5.3, pytest 3.6.1, no additional pytest plugins.

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}
def pytest_generate_tests(metafunc):
    if 'tested_branch' in metafunc.fixturenames:
        branches = KNOWN_BRANCHES.keys()
        metafunc.parametrize("tested_branch", branches, scope="session")

@pytest.fixture(scope="session")
def repo_preparation(tested_branch):
    """ If it would have a "session" scopoe then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[tested_branch] += 1
    return KNOWN_BRANCHES[tested_branch]

And the simplest reproduction:

# test_parametrization.py
import pytest

def test_nothing(repo_preparation):
    assert repo_preparation == 1  # that passes

@pytest.mark.parametrize("_", ["param_1", "param_2"])
def test_basic_parametrization(repo_preparation, _):
    # that causes parametrization confisuon
    assert repo_preparation == 1, "Repo prepared more than once."

Gives such an result:

$ pytest -v test_parametrization.py 
================================== test session starts ==================================
platform linux -- Python 3.5.3, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/kamichal/ecws/pyplayground, inifile:
collected 6 items                                                                       

test_parametrization.py::test_nothing[branch_1] PASSED                            [ 16%]
test_parametrization.py::test_basic_parametrization[branch_1-param_1] PASSED      [ 33%]
test_parametrization.py::test_nothing[branch_2] PASSED                            [ 50%]
test_parametrization.py::test_basic_parametrization[branch_1-param_2] PASSED      [ 66%]
test_parametrization.py::test_basic_parametrization[branch_2-param_1] FAILED      [ 83%]
test_parametrization.py::test_basic_parametrization[branch_2-param_2] FAILED      [100%]

======================================= FAILURES ========================================
_____________________ test_basic_parametrization[branch_2-param_1] ______________________

repo_preparation = 2, _ = 'param_1'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 2 == 1

test_parametrization.py:12: AssertionError
_____________________ test_basic_parametrization[branch_2-param_2] ______________________

repo_preparation = 3, _ = 'param_2'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 3 == 1

test_parametrization.py:12: AssertionError
========================== 2 failed, 4 passed in 0.03 seconds ===========================

Setting indirect=True in the metafunc.parametrize call fixes that problem, but there is no information that it's required for larger scopes to work properly. There is only an advice to do so, but...

The text was updated successfully, but these errors were encountered:

pytestbot · 2018-06-06T09:12:33Z

GitMate.io thinks possibly related issues are #634 (metafunc.parametrize overwrites scope), #519 (fixture scope is ignored when using metafunc.parametrize()), #570 (indirect=True in parametrize breaks fixture scopes), #244 (parametrize fails when values are unhashable), and #1111 (pytest.mark.parametrize fails with lambdas).

RonnyPfannschmidt · 2018-06-06T09:37:36Z

that looks like a bug

note that you can use fun workaround

since you already know your parameters,

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}

@pytest.fixture(scope="session", param=KNOWN_BRANCHES.keys())
def repo_preparation(request):
    """ If it would have a "session" scope then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[request.param] += 1
    return KNOWN_BRANCHES[request.param]

might do (untested)

kamichal · 2018-06-06T09:54:35Z

Thank you Ronny. I know the "workaround" very well.
The point is that I would like to take the tested_branch parameters from parser.addoption.
So, contents of that list is variable in my case, that's why I took pytest_generate_tests hook.

RonnyPfannschmidt · 2018-06-06T10:20:04Z

@kamichal i see - in that case - there is a deeper underlying issue that i wont be investigating in near future - i have no idea if any of the other core devs has other plans

winklerrr · 2018-06-19T13:51:53Z

I came across the same bug, I think...
When using metafunc.parametrize(scope=session) in combination with parametrized tests, all of my session scoped fixtures teared down after each test.

This behavior can be demonstrated very vividly with --setup-show:

SETUP    S session_fixture1 # <-- This fixture is generated with pytest_generate_tests
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test1()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2 
TEARDOWN S session_fixture1 # <-- Session scoped fixture gets tear downed
SETUP    S session_fixture1 
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test2()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2
TEARDOWN S session_fixture1

Workaround

@kamichal For me, the workaround with indirect=True also works in combination with parameters received from metafunc.config.getoption.

Just define a real fixture with the same name (in the example below my_fixture) and use the request fixture to return the indirect parametrized value:

def pytest_addoption(parser):
    parser.addoption("--my_parser_option")
    
def pytest_generate_tests(metafunc):
    if "my_fixture" in metafunc.fixturenames:
        metafunc.parametrize("my_fixture", metafunc.config.getoption("my_parser_option"), scope="session", indirect=True)

@pytest.fixture(scope="session")
def my_fixture(request):
    return request.param

def test_fixture(my_fixture):
    assert my_fixture == "world"

Then you can call it with pytest --my_parser_option hello, which will result in "hello" != "world"
I hope this helps someone.

Regards
Winklerrr

RonnyPfannschmidt · 2018-06-19T13:57:11Z

thanks for providing the workaround,
this also means we can have a better way of understanding what is wrong

as its limited to freestanding parameterization

There seems to be a bug in pytest where metafunc.parametrize breaks fixture scope. By creating a fixture and parameterizing it indirectly scope can be maintained

Implement workaround for pytest issue [#3542](pytest-dev/pytest#3542) There seems to be a bug in pytest where metafunc.parametrize breaks fixture scope. By creating a fixture and parameterizing it indirectly scope can be maintained

PShiw · 2019-12-02T19:34:41Z

I came across the same bug, I think...
When using metafunc.parametrize(scope=session) in combination with parametrized tests, all of my session scoped fixtures teared down after each test.

This behavior can be demonstrated very vividly with --setup-show:
SETUP    S session_fixture1 # <-- This fixture is generated with pytest_generate_tests
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test1()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2 
TEARDOWN S session_fixture1 # <-- Session scoped fixture gets tear downed
SETUP    S session_fixture1 
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test2()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2
TEARDOWN S session_fixture1
Workaround

@kamichal For me, the workaround with indirect=True also works in combination with parameters received from metafunc.config.getoption.

Just define a real fixture with the same name (in the example below my_fixture) and use the request fixture to return the indirect parametrized value:
def pytest_addoption(parser):
    parser.addoption("--my_parser_option")
    
def pytest_generate_tests(metafunc):
    if "my_fixture" in metafunc.fixturenames:
        metafunc.parametrize("my_fixture", metafunc.config.getoption("my_parser_option"), scope="session", indirect=True)

@pytest.fixture(scope="session")
def my_fixture(request):
    return request.param

def test_fixture(my_fixture):
    assert my_fixture == "world"
Then you can call it with pytest --my_parser_option hello, which will result in "hello" != "world"
I hope this helps someone.

Regards
Winklerrr

Worked perfectly in my case. Thanks for the workaround.

rassie · 2021-09-29T12:05:19Z

I'm sorry to be nagging, but is this something that is going to be addressed someday? Is this confirmed as a bug?

The-Compiler · 2021-09-30T08:17:27Z

I'm sorry to be nagging, but is this something that is going to be addressed someday?

As usual with open source projects, it will be addressed once someone feels like working on it (or someone pays someone to work on it).

Is this confirmed as a bug?

@RonnyPfannschmidt (a pytest core maintainer) said "that looks like a bug" above.

pytestbot added the topic: parametrize related to @pytest.mark.parametrize label Jun 6, 2018

pytestbot added the type: bug problem that needs to be addressed label Jun 6, 2018

barnabasJ mentioned this issue Sep 12, 2018

Workaround for pytest issue #3542 pytest-dev/pytest-testinfra#346

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations #3542

metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations #3542

kamichal commented Jun 6, 2018 •

edited

pytestbot commented Jun 6, 2018

RonnyPfannschmidt commented Jun 6, 2018

kamichal commented Jun 6, 2018 •

edited

RonnyPfannschmidt commented Jun 6, 2018

winklerrr commented Jun 19, 2018 •

edited

RonnyPfannschmidt commented Jun 19, 2018

PShiw commented Dec 2, 2019

Workaround

rassie commented Sep 29, 2021

The-Compiler commented Sep 30, 2021

metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations #3542

metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations #3542

Comments

kamichal commented Jun 6, 2018 • edited

pytestbot commented Jun 6, 2018

RonnyPfannschmidt commented Jun 6, 2018

kamichal commented Jun 6, 2018 • edited

RonnyPfannschmidt commented Jun 6, 2018

winklerrr commented Jun 19, 2018 • edited

Workaround

RonnyPfannschmidt commented Jun 19, 2018

PShiw commented Dec 2, 2019

Workaround

rassie commented Sep 29, 2021

The-Compiler commented Sep 30, 2021

kamichal commented Jun 6, 2018 •

edited

kamichal commented Jun 6, 2018 •

edited

winklerrr commented Jun 19, 2018 •

edited