Tox4 #163

tibortakacs · 2023-02-21T17:56:48Z

This PR introduces tox4 support.

Notes:

The tests can be run only outside of tox.
The previous API has been followed, but there is not tox3 support.

for more information, see https://pre-commit.ci

tox_conda/plugin.py

tibortakacs · 2023-05-05T15:02:34Z

@drdavella @AntoineD I would be happy to move forward with this PR. I would like to get your initial feedback though. What do you think the next sensible steps should be?

AntoineD · 2023-05-06T06:53:54Z

@tibortakacs How about having working tests?

tibortakacs · 2023-05-06T09:28:51Z

@AntoineD Sounds great for me. Which way do you prefer: having more complex code and tests but both tox3 and tox4 support, or the new major version is only for tox4 support?

AntoineD · 2023-05-06T10:01:34Z

I think the tests should only support tox 4.

tibortakacs · 2023-05-06T10:42:36Z

@AntoineD Re-reading my previous post, it was slightly confusing: I meant whether the tox-conda library itself after this PR should support both tox3 and tox4, or just tox4?

AntoineD · 2023-05-06T13:11:05Z

It should only support tox 4 for everything.

for more information, see https://pre-commit.ci

tibortakacs · 2023-06-02T23:20:00Z

@AntoineD I updated all the tests to pass. I mainly kept the tests and the logic, but of course I had to update a few of them. I used the tox.pytest infrastructure. All tests are passing locally, but unfortunately I could not make it work with tox. I suspect that since the tests call tox_run from tox, there is a conflict somewhere, and despite all my effort it does not use the temporary directory that it should. (@gaborbernat You might have some advice here.)

I am not sure when I can spend another big push on this. It works well on my project (it has been used for months now, however, in a quite simple conda context). Of course, I am happy to align with your advice, but we might consider executing the tests purely from a virtualenv (all conda calls are mocked), and we can resolve this later.

@AntoineD What do you think?

AntoineD · 2023-06-06T17:23:45Z

I am not familiar with tox 4 API so I let @gaborbernat decide what is the best.

gaborbernat · 2023-06-06T17:32:58Z

tox_conda/plugin.py

+    def _get_python(self, base_python: List[str]) -> Optional[PythonInfo]:
+        exe_path = base_python[0]
+
+        output = self._run_pure(


Shold be cached to do fresh run every time 🤔 something like virtualenv does

Do you mean we should run this only once, and cache the result?

Yeah something like https://github.com/pypa/virtualenv/blob/main/src/virtualenv/discovery/cached_py_info.py#LL28C1-L28C1

Great idea, I will do it.

gaborbernat · 2023-06-06T17:33:32Z

tox_conda/plugin.py

+        return PythonInfo(implementation, version_info, version, is_64, platform_name, extra)
+
+    @property
+    def _package_tox_env_type(self) -> str:


Any reason we use virtualenv for packaging and not conda?

I did not see any urgent need to diverge from the original tox behavior here, I have not experienced any issue by using it in conda. But happy to change if there is an issue with this. What would be preferred command here?

Duno, would make some sense to use conda environments universally rather than virtualenv in some places and conda in others. nitpick. Ultimately I'm only advisory here. @AntoineD is the maintainer of this project, if he no longer needs to do so, I'm happy to hand over the maintainership bit to you.

This is a good question. Happy to dig deeper how conda does the packaging, but it might take slightly longer. I will invest some time already in this PR to estimate how much work would be needed. If not too much, happy to do it now, if it took longer, I might prefer merging as it is not, and resolve it later. This is just an opinion though, happy to do either way.

Duno, would make some sense to use conda environments universally rather than virtualenv in some places and conda in others. nitpick. Ultimately I'm only advisory here.

I think that packaging with anaconda should be opt in because:

some people use anaconda to get the required deps in constrained IT envs, but with no actual need for anaconda packaging.

building an anaconda package take way more time! (the way tox and conda build work have some overlap)

@AntoineD is the maintainer of this project, if he no longer needs to do so, I'm happy to hand over the maintainership bit to you.

I've been wondering about that for quite some time: I not longer have enough spare time for tox-conda and as I said do not know tox 4 so I guess that tox-conda deserves a better maintainer, @tibortakacs you seem to be the best candidate!

Thank you for the insight, @AntoineD. Then, if you do not have any objection, let's keep the pip builder in this PR, and further down the line we can create a new option to enable conda build packager.

@AntoineD @gaborbernat Thank you, this is a great honor. Both tox and conda are key parts of the Python, especially in R&D communities, and tox-conda is the crucial glue between them. I am happy to act as the maintainer of the project, and see whether I can bear this responsibility on the level that this project deserves.

gaborbernat · 2023-06-06T17:35:42Z

tox_conda/plugin.py

+
+        _, conda_deps = self.conf["conda_deps"].unroll()
+        if conda_deps:
+            conda_dict["deps"] = conda_deps


Why do we need deps for Python?

This ensures that if the conda dependencies change, the environment is recreated because there will a cache change. Was the question about this?

Shouldn't these be already covered by deps durings the setup, the same way the tox uses it?

Probably you are right, I am still learning how tox does the job internally. Can you point me to the tox code to be sure that I am checking the right approach.

tibortakacs · 2023-06-06T21:54:39Z

tests/test_conda_env.py

+    assert fnmatch(
+        pip_install_command,
+        (
+            f"*conda run -p {str(proj.path / '.tox' / env_name)} --live-stream"


@gaborbernat Let me ask your advise here. All the tests are based on the fixtures (tox_project) that are copy (with minor modification) from the tox.pytest module. It works properly, when I call pytest directly, all tests are passing. However, when I use tox, it does not work well, proj.path point to the source root folder instead of the temporary directory. I am leaning towards running tox-conda test using pytest for now, but I do not particularly like this, I want to keep tox here. Is there any solution for this?

Would have to check why the path differs.

It is actually more, tox_project creates the tox environment in the .tox folder of the source folder.

for more information, see https://pre-commit.ci

Tibor Takacs and others added 9 commits February 13, 2023 17:46

Initial commit with base functionality.

f66a161

Merge find_conda functions.

bb87ba7

Rearrange CondaEnvRunner.

0fc5068

Implement conda_env.

48c40ca

Handle conda_env.

c792980

Remove unused code.

838d5e1

Installation.

7a0c51e

Reformat.

93c897a

[pre-commit.ci] auto fixes from pre-commit.com hooks

7504726

for more information, see https://pre-commit.ci

tibortakacs mentioned this pull request Feb 21, 2023

tox4 support #156

Open

gaborbernat requested review from AntoineD and dcrosta March 10, 2023 23:00

gaborbernat requested changes Mar 10, 2023

View reviewed changes

tox_conda/plugin.py Outdated Show resolved Hide resolved

gaborbernat reviewed Mar 10, 2023

View reviewed changes

tox_conda/plugin.py Outdated Show resolved Hide resolved

Tibor Takacs added 2 commits April 24, 2023 12:15

Inherit from PythonRun and add type info.

dfb59dc

Reformat plugin with isort and black.

4aed14c

dcrosta removed their request for review May 8, 2023 16:28

Tibor Takacs and others added 7 commits May 30, 2023 14:50

WIP.

6afdb36

Fixes.

1330cad

Plugin fixes.

26fe70f

Use executor instead of subprocess.

b51c6a7

Test infrastructure wip.

52a8bfb

[pre-commit.ci] auto fixes from pre-commit.com hooks

503e190

for more information, see https://pre-commit.ci

Fix test passing.

b2c68cd

Tibor Takacs and others added 16 commits June 1, 2023 22:44

4th test.

580330a

Format.

b5104ee

conda_spec test added.

e77fd71

Add conda-env test and fixes.

fcff0cc

Test for env. and spec.

7bdd51e

Add conda env. tests are ready.

311a14e

Add conda_name test.

af770cf

Format.

10868ef

Add cache tests.

3b7b57e

Add more tests and fixes.

8e863c4

Format.

b27855d

Add more tests.

9f761eb

Format.

ebe2c70

Linting fixes.

fa09579

Run tests with pytest directly.

63faefc

[pre-commit.ci] auto fixes from pre-commit.com hooks

bc0cf59

for more information, see https://pre-commit.ci

tibortakacs marked this pull request as ready for review June 2, 2023 23:14

gaborbernat reviewed Jun 6, 2023

View reviewed changes

tibortakacs commented Jun 6, 2023

View reviewed changes

zaneselvans mentioned this pull request Oct 22, 2023

Set up reproducible Python environments with conda-lock catalyst-cooperative/pudl#2968

Merged

8 tasks

trexfeathers mentioned this pull request Oct 31, 2023

Switch to tox SciTools/iris#5184

Open

trexfeathers mentioned this pull request Nov 29, 2023

Unpin tox<4 bjlittle/geovista#170

Open

3 tasks

tibortakacs changed the title ~~Tox4 PoC~~ Tox4 Jan 19, 2024

Tibor Takacs and others added 5 commits January 19, 2024 16:32

Merge remote-tracking branch 'origin/main' into tox4

610ddd8

Add conda_python option.

7935e75

Lint and type fixes.

305ff7f

[pre-commit.ci] auto fixes from pre-commit.com hooks

023cc54

for more information, see https://pre-commit.ci

Try fixing tests.

97f68b8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tox4 #163

Tox4 #163

tibortakacs commented Feb 21, 2023 •

edited

tibortakacs commented May 5, 2023

AntoineD commented May 6, 2023

tibortakacs commented May 6, 2023

AntoineD commented May 6, 2023 •

edited

tibortakacs commented May 6, 2023

AntoineD commented May 6, 2023

tibortakacs commented Jun 2, 2023 •

edited

AntoineD commented Jun 6, 2023

gaborbernat Jun 6, 2023

tibortakacs Jun 6, 2023

gaborbernat Jun 6, 2023

tibortakacs Jun 6, 2023

gaborbernat Jun 6, 2023

tibortakacs Jun 6, 2023

gaborbernat Jun 6, 2023

tibortakacs Jun 6, 2023

AntoineD Jun 7, 2023

tibortakacs Jun 7, 2023

gaborbernat Jun 6, 2023

tibortakacs Jun 6, 2023

gaborbernat Jun 6, 2023

tibortakacs Jun 6, 2023

tibortakacs Jun 6, 2023

gaborbernat Jun 6, 2023

tibortakacs Jun 6, 2023

Tox4 #163

Are you sure you want to change the base?

Tox4 #163

Conversation

tibortakacs commented Feb 21, 2023 • edited

tibortakacs commented May 5, 2023

AntoineD commented May 6, 2023

tibortakacs commented May 6, 2023

AntoineD commented May 6, 2023 • edited

tibortakacs commented May 6, 2023

AntoineD commented May 6, 2023

tibortakacs commented Jun 2, 2023 • edited

AntoineD commented Jun 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tibortakacs commented Feb 21, 2023 •

edited

AntoineD commented May 6, 2023 •

edited

tibortakacs commented Jun 2, 2023 •

edited