pytest-cases not compatible with pytest-repeat

[PiQuer]

Pytest-repeat is a plugin for, well, repeating tests :)

Tested with pytest-cases 3.6.11 and pytest-repeat 0.9.1, python version 3.9.9:

$ python -m venv venv
$ . venv/bin/activate
$ pip install pytest-cases pytest-repeat

A minimal example to reproduce the issue is this test_repeat.py:

from pytest_cases import parametrize_with_cases, parametrize

@parametrize("foo", (1,2))
def case_a(foo):
    pass

@parametrize_with_cases("arg", cases='.')
def test_repeat(arg):
    assert True

When run with pytest test_repeat.py it works, but with pytest test_repeat.py --count=10 it raises the following exception:

___________________________________________________ ERROR collecting test_repeat.py ____________________________________________________
venv/lib/python3.9/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
venv/lib/python3.9/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
venv/lib/python3.9/site-packages/_pytest/python.py:262: in pytest_pycollect_makeitem
    return list(collector._genfunctions(name, obj))
venv/lib/python3.9/site-packages/_pytest/python.py:501: in _genfunctions
    fixtureinfo.prune_dependency_tree()
venv/lib/python3.9/site-packages/_pytest/fixtures.py:393: in prune_dependency_tree
    self.names_closure[:] = sorted(closure, key=self.names_closure.index)
venv/lib/python3.9/site-packages/pytest_cases/plugin.py:651: in __setitem__
    raise NotImplementedError("It is not possible to replace an element in a super fixture closure,"
E   NotImplementedError: It is not possible to replace an element in a super fixture closure,as the partitions inside it do not have the same size

Note that repeat works when the case function itself is not parametrized.

[smarie]

Thanks @PiQuer for spotting this ! Could you please let me know the version of pytest-cases you are using ? Indeed the latest version of pytest-cases should have fixed this specific line popping in the stack trace.

Thanks in advance !

[PiQuer]

I can reproduce the error with pytest 3.6.11 and with the latest master def94ce.

[smarie]

super, thanks for the confirmation ! I therefore have to investigate in order to understand why there is some fixture reordering when pytest-repeat is enabled. It seems strange that simply repeating tests would trigger re-ordering, but I do not know what they actually do...

[PiQuer]

Thanks for investigating! I guess it is practically impossible to be compatible with all pytests plugins out there, if they have a certain complexity.

I just thought it would be good to document the incompatibility. Pytest-repeat looked like a good fit to easily get statistics about unstable tests, but for me it is not top-priority to get this working ;)

[smarie]

It seems that they define a __pytest_repeat_step_number fixture statically, but they append it to the test nodes metafunc dynamically (only if the count option is active) and parametrize it dynamically : https://github.com/pytest-dev/pytest-repeat/blob/master/pytest_repeat.py#L61

I'll see what I can do

[smarie]

Well I do not clearly understand why this happens in the first place (even without pytest-cases).. I tried to run the following simple example :

import pytest

@pytest.mark.parametrize("arg", (1,))
def test_repeat(arg):
    assert True

And then put a breakpoint in pytest.fixtures.prune_dependency_tree line 393 (the line appearing in your stacktrace). So, at:
self.names_closure[:] = sorted(closure, key=self.names_closure.index)

Inspecting the variables:

>>> self.names_closure
 ['environment', 'arg', '__pytest_repeat_step_number']
>>> closure
 {'arg', 'environment'}

So the function seems to try to remove the fixture that was added by pytest-repeat (!). That's where I raise the error, as in pytest cases, self.names_closure is a tree of fixture names, while in pytest this is a list.

One workaround could be to simply accept removal of fixtures in the tree, without reordering (I currently simply ignore reordering and create a warning message). In this particular situation, this would be enough to make the plugin compatibility work.

[smarie]

It should be fixed in 3.6.12, available in a few minutes. Let me know if this works for you !

[PiQuer]

Looks good! Thanks for the fix :)

[smarie]

Thanks again @PiQuer for the contribution