Describe pytest execution in developer documentation

[renefritze]

Currently we rely, via the Makefile, to run python setup.py pytest. This is deprecated by setuptools. It installs dependencies with easy_install as opposed to pip.
Parallel to that we have the CI test scripts which all run pytest (under xvfb-run or mpirun at times) directly in the end. These take care to setup the env previous to running pytest. That means installing requirements/extras and ensuring pyMOR's extensions are build. Ideally we'd want a replacement for python setup.py pytest that does all that, but relies on metadata in pyproject.toml.
AFAICT there's currently no PEP for this, nor a project that fits completely.

Some references:

• https://pypi.org/project/pytest-runner/
• https://pypi.org/project/pip-run/
• Deprecate test command pypa/setuptools#1684
• setup.py test is going to be deprecated pytest-dev/pytest#5534
• https://pypi.org/project/tox-current-env

My current goto replacement would probably be using pip-run .[ci,optional] -- py.test --pyargs pymortests -c .ci/installed_pytest.ini. This still needs to have pip-run already installed ofc, but should take of most of the rest. It would also build a new wheel for pymor on each invocation (pip-run does not support editable installs), which is not ideal at all.

Seperate from this is the question of removing the full-test Makefile target. It (only) installs pytest plugins that are not included in our CI setup dependencies and then runs pytest. Neither @sdrave nor I use it or remember why it's there. How about you @pmli?

Ref. #910

[pmli]

I think I used full-test a couple of times. Mostly, I run specific tests directly using pytest.

[renefritze]

I think I used full-test a couple of times. Mostly, I run specific tests directly using pytest.

I'll take that as no opposition to remove it then 😉

[pmli]

I just used it when I wanted to compute the code coverage locally. If we could keep that functionality, that would be nice, but I'm also fine with just using pytest directly.

[renefritze]

I just used it when I wanted to compute the code coverage locally. If we could keep that functionality, that would be nice, but I'm also fine with just using pytest directly.

We could simply move the --cov option to the default args in setup.cfg that are always added

[sdrave]

We could simply move the --cov option to the default args in setup.cfg that are always added

This is not an option for me. First of all, I do not want to get a coverage report whenever I run py.test. It messes up the console output and takes time to compute. Also,

py.test --cov operators.py

will, for some reason run the entire test suite (as opposed to py.est operators.py --cov), and adding the option to setup.cfg has the same effect. Also, simply adding --cov does not seem like much of a burden.

Regarding a different options: How about simply removing the make targets, and document in README.md how to run the tests?

[renefritze]

--cov takes an argument @sdrave

[renefritze]

We could simply move the --cov option to the default args in setup.cfg that are always added

This is not an option for me. First of all, I do not want to get a coverage report whenever I run py.test. It messes up the console output and takes time to compute. Also,

py.test --cov operators.py

will, for some reason run the entire test suite (as opposed to py.est operators.py --cov), and adding the option to setup.cfg has the same effect. Also, simply adding --cov does not seem like much of a burden.

Regarding a different options: How about simply removing the make targets, and document in README.md how to run the tests?

I never use make test anyway, so removing that would be fine with. But removing docker_test ist not an option for me.

Also: lets just discuss this in the next meeting?

[sdrave]

--cov takes an argument @sdrave

Good to know ...

I never use make test anyway, so removing that would be fine with. But removing docker_test ist not an option for me.

I think there is not reason to remove docker_test.

[renefritze]

Dev meeting: Decided to remove make test, change readme to use pytest after building ext.

[renefritze]

The remaining work in this issue is actually to describe pytest execution in the non-containerized environment in the dev docs. This should include most common options/setups like running a single file, filtering with -k, etc.