How to build custom items with pytest-xdist

[sar009]

I am making my custom test items using pytest_pycollect_makeitem and adding some context to the test item(although I am not sure if this is the right way of doing things). My code looks like the following.

main.py

#!/usr/bin/env python3
import inspect
import uuid

import pytest


class MyFunctionItem(pytest.Function):
    def __init__(self, my_param, *args, **kwargs):
        self.my_param: str = my_param
        super().__init__(*args, **kwargs)


class MyClassItem(pytest.Class):
    def __init__(self, my_param, *args, **kwargs):
        self.my_param: str = my_param
        super().__init__(*args, **kwargs)


class MyPlugin:

    __name__ = 'my_plugin'

    def pytest_pycollect_makeitem(self, collector, name, obj):
        items = []
        if ((inspect.isfunction(obj) or inspect.ismethod(obj))
                and (name.startswith("test_") or name.endswith("_test"))):

            fixture_info = pytest.Function.from_parent(collector, name=name, callobj=obj)._fixtureinfo
            item = MyFunctionItem.from_parent(
                collector, name=name, fixtureinfo=fixture_info, my_param=str(uuid.uuid4()))
            items.append(item)

        if inspect.isclass(obj):
            items.append(MyClassItem.from_parent(collector, name=name, obj=obj, my_param=str(uuid.uuid4())))

        return items


if __name__ == '__main__':
    pytest.main(["test.py", "-s"], plugins=[MyPlugin()])
    print("-----------------------")
    pytest.main(["test.py", "-s", "-n", "2"], plugins=[MyPlugin()])

test.py

class TestClass:
    def test_in_class_with_decorator(self, request):
        print(request.node.my_param)

    def test_in_class(self, request):
        print(request.node.my_param)


def test_out_class(request):
    print(request.node.my_param)

when running the code I get the following error.

FAILED test.py::TestClass::test_in_class_with_decorator - AttributeError: 'Function' object has no attribute 'my_param'
FAILED test.py::TestClass::test_in_class - AttributeError: 'Function' object has no attribute 'my_param'
FAILED test.py::test_out_class - AttributeError: 'Function' object has no attribute 'my_param'

How do I add context in pytest_pycollect_makeitem and access the context while using pytest-xdist

[RonnyPfannschmidt]

You code only adds the plugin to the initial process, xdist starts new processes without your plugin added

What's the use case for the custom data?

[sar009]

I am initializing an object while creating the test item to be used inside the test.

[RonnyPfannschmidt]

That use case is so unspecific that it seems appropriate to recommend a fixture

[sar009]

So I am using this for two reasons.

1. Do a heavy calculation only once and all test items use the result rather than doing the heavy calculation in every test.
2. When building the test item I also want the context used to build the test item. We have a YAML file that contains the Python test file location and various inputs. for each input, we want to build the test item and have the input in the context of the test.

[sar009]

please check the updated code below.

main.py

#!/usr/bin/env python3
import argparse
import inspect

import pytest
import yaml


class MyFunctionItem(pytest.Function):
    def __init__(self, my_complex_calc, my_test_config, *args, **kwargs):
        self.my_complex_calc: str = my_complex_calc
        self.my_test_config: dict = my_test_config
        super().__init__(*args, **kwargs)


class MyPlugin:

    __name__ = 'my_plugin'

    def __init__(self, suites):
        self.suites = suites
        self.my_complex_calc = "5"

    def pytest_pycollect_makeitem(self, collector, name, obj):
        file_name = str(collector.fspath).split("/")[-1]
        items = []
        if ((inspect.isfunction(obj) or inspect.ismethod(obj))
                and (name.startswith("test_") or name.endswith("_test"))):

            fixture_info = pytest.Function.from_parent(collector, name=name, callobj=obj)._fixtureinfo
            for each_test in self.suites[file_name]:
                item = MyFunctionItem.from_parent(
                    collector, name=name, fixtureinfo=fixture_info,
                    my_complex_calc=self.my_complex_calc, my_test_config=each_test
                )
                items.append(item)

        if inspect.isclass(obj):
            return pytest.Class.from_parent(collector, name=name, obj=obj)

        return items


def _get_suites(yaml_suites) -> dict[str, list[dict]]:
    suites: dict[str, list[dict]] = {}
    for each_file in yaml_suites:
        with open(each_file, 'r') as suite_file:
            suite_yaml = yaml.safe_load(suite_file)

        for each_test in suite_yaml.get("tests", []):
            file_name = each_test["file"]
            if suites.get(file_name) is None:
                suites[file_name] = []

            suites[file_name].append(each_test)

    return suites


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('suites', nargs=argparse.ONE_OR_MORE)
    args = parser.parse_args()
    suites = _get_suites(args.suites)

    pytest_args = list(suites.keys())
    pytest.main(pytest_args + ["-s"], plugins=[MyPlugin(suites)])
    print("-----------------------")
    pytest.main(pytest_args + ["-s", "-n", "2"], plugins=[MyPlugin(suites)])


if __name__ == '__main__':
    main()

test.py

def test_func(request):
    print()
    print(f"value of complex calc is {request.node.my_complex_calc}")
    print(f"{request.node.my_test_config['config']['name']}")

suite.yml

name: my_tests
tests:
  - name: tests with alice
    file: test.py
    config:
      name: alice
  - name: tests with bob
    file: test.py
    config:
      name: bob

run with the following command

./main.py suite.yml

[RonnyPfannschmidt]

session scope is for compute once use everywhere

if you have a custom collector to begin with, and no parameters to the plugin object - let a toplevel conftest reegister it

if you have test inputs, that sounds like a case for parameterize based on a cli arg, not a custom collector that half-implements function-collection

based on the provided information my impression is that you need a conftest based plugin that annotates your tests based on inputs

a plugin added that way is automatically available in xdist

you are still describing a abstract use-case - for someone in my position those naturally seem very matching to already preexisting features
if you can describe the use-case more concrete i may be able to give more concrete advise

[RonnyPfannschmidt]

the posted yaml example can be interpreted as a fight against the pytest architecture/apis

its not clear to me what the end goal is, but the layout compared to how pytest does things seems to me like a major friction point

[sar009]

okay, let me give it a fresh try once again.
My requirement is to build an automation framework and we already have a small framework made for the purpose that was built with pytest in place. The idea of the framework is to have test code written once and use the code with multiple inputs. So we came up with the approach of writing a YAML file(we call it suite) which follows a structure that contains the test file and the inputs for the test function. To achieve the same we used pytest_pycollect_makeitem and started creating custom test items and setting the YAML content in the context of the item which is then accessed through the request fixture.

linking previous discussion in the thread for referencing the files #957 (reply in thread)

Although everything runs fine I am not sure if this is the right way of doing things and if it violates any pytest principles and majority if in future we would not be able to use certain functionality. For example, we are not able to use xdist plugin.