tests\cases\issues\test_issue_269.py raises ValueError during collection

[jgersti]

tests\cases\issues\test_issue_269.py throws an exception during collection in most test environments with the following trace (modulo the env name)

.nox\tests-3-9-env-pytest-latest\lib\site-packages\pluggy\_hooks.py:433: in __call__
     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
 .nox\tests-3-9-env-pytest-latest\lib\site-packages\pluggy\_manager.py:112: in _hookexec
     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
 .nox\tests-3-9-env-pytest-latest\lib\site-packages\_pytest\python.py:271: in pytest_pycollect_makeitem
     return list(collector._genfunctions(name, obj))
 .nox\tests-3-9-env-pytest-latest\lib\site-packages\_pytest\python.py:498: in _genfunctions
     self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
 .nox\tests-3-9-env-pytest-latest\lib\site-packages\pluggy\_hooks.py:489: in call_extra
     return self._hookexec(self.name, hookimpls, kwargs, firstresult)
 .nox\tests-3-9-env-pytest-latest\lib\site-packages\pluggy\_manager.py:112: in _hookexec
     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
 .nox\tests-3-9-env-pytest-latest\lib\site-packages\_pytest\fixtures.py:1577: in pytest_generate_tests
     metafunc.parametrize(
 .nox\tests-3-9-env-pytest-latest\lib\site-packages\pytest_cases\plugin.py:910: in parametrize
     raise ValueError("This should not happen - please file an issue")
 E   ValueError: This should not happen - please file an issue

[smarie]

Indeed I am able to reproduce, thanks @jgersti for reporting !

The issue is due to the fact that our own version of MetaFunc.parametrize is not patched onto the metafunc object early enough, and another plugin (my fake version of the pytest-repeat plugin used in tests) has already called the function.

this should be called first:

python-pytest-cases/src/pytest_cases/plugin.py

Lines 851 to 862 in ab3b719

	@pytest.hookimpl(tryfirst=True, hookwrapper=True)
	def pytest_generate_tests(metafunc):
	"""
	We use this hook to replace the 'partial' function of `metafunc` with our own below, before it is called by pytest
	Note we could do it in a static way in pytest_sessionstart or plugin init hook but
	that way we can still access the original method using metafunc.__class__.parametrize
	"""
	# override the parametrize method.
	metafunc.parametrize = partial(parametrize, metafunc)
	# now let pytest parametrize the call as usual
	_ = yield

this should be called last:

python-pytest-cases/tests/pytest_extension/issues/test_issue_269.py

Lines 10 to 26 in ab3b719

	@pytest.hookimpl(trylast=True)
	def pytest_generate_tests(metafunc):
	"""This hook and fixture above are similar to what happens in pytest-repeat.
	See https://github.com/smarie/python-pytest-cases/issues/269
	"""
	if metafunc.function is test_repeat:
	metafunc.fixturenames.append("__my_repeat_step_number")
	def make_progress_id(i, n=2):
	return '{0}-{1}'.format(i + 1, n)
	metafunc.parametrize(
	'__my_repeat_step_number',
	range(2),
	indirect=True,
	ids=make_progress_id
	)

But unfortunately the opposite happens.

This seems to come from a change in pluggy that changed the meaning of the try_first trylast and wrapper meanings. Most probably a bug I'd guess. I'll open an issue there.

[smarie]

Issue opened: pytest-dev/pluggy#441
Until the fix is available, we can mark the test as expected to fail.

EDIT : I commented the whole tests/pytest_extension/issues/test_issue_269.py test file, let's uncomment it when ok