Move tests out-of-tree

[bollwyvl]

References

• fixes Moving tests out-of-tree? #14058

[bollwyvl]

Locally:

579K ipython-8.14.0.dev0-py3-none-any.whl

vs PyPI

797.6K ipython-8.13.0-py3-none-any.whl

[Carreau]

Yeah, I'm thorn as I can see the reason to package the test in the wheel. On some system you may want to run the test to make sure all works, and having to reach for the source in meh...

[bollwyvl]

run the test to make sure all work

That's the thing: i don't think one can right now, without the source, as it generally fails without the exact pyproject.toml, in the right path, next to the source. On conda-forge we previously got away with --pyargs IPython, but that stopped working.

I guess another alternative would be to build and distribute (and test as-distributed) a second ipython-tests package, with a == pin on ipython, which hoisted all the required test dependencies as actual dependencies.

[Carreau]

I guess another alternative would be to build and distribute (and test as-distributed) a second ipython-tests package, with a == pin on ipython, which hoisted all the required test dependencies as actual dependencies.

I think all of this might be a discussion for https://github.com/scientific-python/specs
I'd like to have a standard across all the packages. I understand wanting test to not be importable, but from last week meeting in Seattle it looks like running the test from a wheel install is fairly critical.

[bollwyvl]

running the test from a wheel install is fairly critical.

sure, but then they need to actually run with a documented approach... i guess, ideally, exactly pip install ipython[test] && pytest --pyargs IPython, without extra config files. downstream on conda-forge we're still having to do weird path stuff to run the tests.

As the test dependencies won't/shouldn't/can't/aren't consistent by platform/distribution to be installed by default, perhaps the 200kb of tests could still be out-of-tree, but with an inverted dependency, e.g.

[project.optional-dependencies]
test = [
  "pytest",
  # ...
  "ipython-tests", 
]

... and the invocation would be pytest --pyargs _IPython_Tests or something.

[Carreau]

Let me think about it. No objections if there is consensus. I'm not a big fan of having different package as long as it's hard to do the wrong things when installing/publishing.

[Carreau]

Maybe that can be IPython 9.0, few new features, but rework of the repository ?

[bollwyvl]

installing/publishing

Yep, all valid concerns, of course, and inherited by all downstreams.

IPython 9.0

Sounds fine, but even prior to that, if test-as-installed is important, having the simplest test invocation possible documented and working would not be breaking. A concrete step here would be to up-front building .whl and .tar.gz in CI, and runing the bulk of the CI jobs against the downloaded, as-installed package, rather than in-tree.

Looking to 9.x, another thing I'd love is much lazier imports of things that don't aren't used everywhere. Putting my alternate runtime environment hat on, our biggest sources of challenges are jedi and prompt-toolkit. Making these truly lazy (e.g. not excited by import IPython) would be a great boon.

rework of the repository ?

Indeed, getting on the pyproject.toml train, with an explicit PEP517 build backend, would be great, even if only declaring setuptools_meta... but flit would also do the reproducible build stuff, and is by far the easiest to reason about if no binary extensions are being built.

The only sticky thing would be keeping the hard version pin between two packages somewhere, but having a single-source-of truth for the version in one place (hint: pyproject.toml#/project/version), and a test that verifies that number appears in the pin, would likely be enough to keep broken builds from getting created, much less published.