envlist anti-pattern advertised in primary README example?

[ssbarnea]

The first example from the project readme advertises what proved to be an anti-pattern for tox.ini long-term maintenance: the use of explicit python versions like envlist = py37,py38, or similar py{37,36,35,27}.

The issue with this pattern is that it requires permanent maintenance of the tox.ini file in order to advertise new python versions.

As an alternative to this, I seen project maintainers migrating towards using only a non specific one like py3 which resolves to the current python3 version available on the system, while not preventing user from running with a particular version.

The interaction between envlist, skip_missing_interpreters and ignore_basepython_conflict can easily produce problems with CI/CD pipelines, some of them with very dangerous effects: running jobs using unexpected version of python and giving false-positives --- hard to spot. Over the last years I encountered several cases like this.

I opened this subject with the hope that we can document a safe and optimal reference use-example that we can advertise to users, especially as many of them do not know all the complex details on how tox runs.

Things that we may want to cover:

• able to run all environments for a single version of python (current one)
• able to run tests using a particular version of python, (like -e pyXY)
• avoid accidents where a specific environment may run using unexpected interpreter
• expose minimal version of python required but not cap the maximal one
• be able to run tests using all installed versions of python (pass if one is missing)
• be able to run tests using all officially supported versions of python (fail if one is missing)
• document how a user can define an environment that can using using a specific range of python versions like run only with py36 or py37 and not anything else (very useful for some version sensitive envs while you still want to keep them relatively portable between different operating systems, which may have different default python versions).

I do expect that covering all these may be possible only with tox4, but that is not a problem.

[gaborbernat]

The first example from the project readme advertises what proved to be an anti-pattern for tox.ini long-term maintenance: the use of explicit python versions like envlist = py37,py38, or similar py{37,36,35,27}.

I strongly disagree with that form being anti-pattern.

The issue with this pattern is that it requires permanent maintenance of the tox.ini file in order to advertise new python versions.

I consider this a strong feature. Who does not update the envlist matrix effectively means the project has not been validated with other versions. The fact that you need to update tox.ini is an explicit statement that we checked and this project does support version xy.

As an alternative to this, I seen project maintainers migrating towards using only a non specific one like py3 which resolves to the current python3 version available on the system, while not preventing user from running with a particular version.

I consider this an anti-pattern and I'm willing to document this in the README. The Python interpreter is not semver versioned, so the fact that a project works with version pyxy does not offer any guarantees that pyx(y+/-1) is also suppored. Generally CPython deprecates something in version a, and then removes it in version a + 2. This makes it non-semver, and only validating with the major version offers no guarantees that any other version will work.

The interaction between envlist, skip_missing_interpreters and ignore_basepython_conflict can easily produce problems with CI/CD pipelines, some of them with very dangerous effects: running jobs using unexpected version of python and giving false-positives --- hard to spot. Over the last years I encountered several cases like this.

• able to run all environments for a single version of python (current one)

What is the current one here? The tox version? This at worst just needs to be documented.

• able to run tests using a particular version of python, (like -e pyXY)

The like format you specified is already supported and documented.

• avoid accidents where a specific environment may run using unexpected interpreter

We do raise errors when we detect this. At worst we just need to improve the detection logic, for this we need specific examples where the current case breaks.

• expose minimal version of python required but not cap the maximal one

This is hard to do and underspecified. How do you even know what versions are available (other than the tox owns)? If the tox ones do not satisfy how do you find and pick another one?

• be able to run tests using all installed versions of python (pass if one is missing)

For this, we would first need to know what versions of python are installed. This is not generally available information, so first we would need to formalize this.

• be able to run tests using all officially supported versions of python (fail if one is missing)
  Again what is officially supported?

• document how a user can define an environment that can using using a specific range of python versions like run only with py36 or py37 and not anything else (very useful for some version sensitive envs while you still want to keep them relatively portable between different operating systems, which may have different default python versions).

tox 4 allows a list of specs for base_python, so you can do base_python = py37,py36. This needs documentation.

In general, though I'd like to fill direct issues where you think there's something actionable because such lung threads will mean we'll miss a lot of actionable parts in between juggling multiple topics...

[ssbarnea]

Maybe calling it an anti-pattern would be bit much, that is why I phrased it as a question :p

There are at least 65 projects on https://codesearch.opendev.org/?q=py3%2C&i=nope&files=tox.ini&excludeFiles=&repos= that seem to use py3. With a little bit use of blame, I can find why they endup up like this, example https://opendev.org/openstack/automaton/commit/86a214fc68e2d183bf6142c51ccde807207e36bc

At least on OpenStack projects envlist is used only for humans, on CI we never rely on it as we always have specific environments and we do not have a job like "py3".

Still, I can see why they wanted to add only a single py3 entry inside envlist, as this would make it much easier for anyone to hack the code, regardless if they have various python versions. Yep, one outside the supported range would likely fail and some projects even have extra safeguards for that.

This approach means that an average developer can just "clone & tox" from any recent system. No need to update documentation example on development guidelines or the toxenv list with an ever moving list of python versions.

I was not involved into this decision but looking back I can really understand why, after years of refreshing envlist, they some got tired and looked for alternatives that require less maintenance.

PS. I used discussions because at this stage I am not sure which actions are needed, I consider this thread more of a brainstorming, hoping to extract some conclusions that would materialize into doc change or even RFE tickets.

[gaborbernat]

I was not involved into this decision but looking back I can really understand why, after years of refreshing envlist, they some got tired and looked for alternatives that require less maintenance.

I'm not saying the problem does not exist, just that the solution used seems not the best. Perhaps we can come up with a better solution, though I'm not sure yet entirely what's the problem they want to solve.