Feature enhancement to allow turning plugin on and off

[clintonsteiner]

Description

Hello,

Using this plugin occasionally while writing new tests to prevent flakiness, but old tests still need to run without the random order.
Currently accomplishing this via pytest.ini which includes -p no:randomly and a separate one you use via -c pytest-randomly.ini

This feels rather clunky, I'd like to propose creating a pytest command line argument to turn the plugin off as needed
The other option is to create a bash variable PYTEST_USE_RANDOMLY which defaults to true, but can be flipped on and off as you go

I'm more than happy to make this change, but wanted to run it by the community before opening the PR.

[adamchainz]

to turn the plugin off as needed

This is hard to do, the plugin acts at different points in pytest's test lifecycle: collecting tests, then running modules, classes, and functions.

To force order for specific tests, you can use pytest-order, which even documents usage with pytest-randomly. Try that for your legacy tests?

[clintonsteiner]

Hi @adamchainz thank you for maintaining this plugin and taking the time to respond to my issue.

The project I'm using this for is massive, and adding markers to each test is not feasible.
Outside of being difficult to do, do you have any objection to the ability to turn this plugin off but leave it installed via a command line argument?
If would be open to the idea I will fork the repo and begin work, but didn't want to work on something that was fundamentally against the project nature.

From what I can tell, I think the change would need to occur inside https://github.com/pytest-dev/pytest-randomly/blob/main/src/pytest_randomly/__init__.py
More specifically within the hook function pytest_collection_modifyitems

Let me know what your thoughts are and thanks once again.

[adamchainz]

I think your proposal is too niche to include in pytest-randomly right now. If others +1 this issue, maybe.

Also, you may be able to achieve what you want with a custom plugin in your project. pytest plugins are very flexible and it should be possible to make one that disables pytest-randomly by default - perhaps by telling pytest to disable the plugin, or perhaps by activating the --randomly-dont-reset-seed and --randomly-dont-reorganize options. Give that a try?