PEP 632, deprecation of distutils and numpy.distutils

[rth]

It looks like PEP 632 with the deprecation of distutils in stdlib and removal in Python 3.12 was accepted.

How would it impact numpy.distutils ? Does it mean that distutils would be added as a stand-alone build dependency (even if it's deprecated) or are there some other long term plans for numpy.distutils?

[rgommers]

If we're just going to deal with the setuptools vendoring distutils and then eventually exposing it with a new API, here is the plan:

• Setuptools summary issue: how to handle numpy.distutils and setuptools interaction pypa/setuptools#2372
• Mailing list discussion: https://mail.python.org/pipermail/numpy-discussion/2020-September/080985.html

If we're going to do a ton of work, the time is now though to move to a build system that doesn't hate compilers and has a sane design. Here is the plan for SciPy and I'm slowly moving that forward: scipy/scipy#13615 (likely will start moving faster soon).

If that move is successful, we should do the same for NumPy.

[rgommers]

My thinking on this topic has evolved a bit. setuptools enabled its vendored copy of distutils for the first time in Sep'21, and after that broke a lot of things reverted that. Re-enabling was planned a few months later, and I said back in Jan'22 that setuptools can go ahead without worrying about numpy.distutils (in pypa/setuptools#2372 (comment)). I expected that to happen soon, and then setuptools was going to establish a new API (original estimate in Sep'21 was 6-9 months for that). So we'd wait, and then fix up numpy.distutils. We're now 15 months further, and setuptools is still very far from exposing that new API (EDIT: setuptools is re-enabling its vendored copy in 2 weeks, see here). Based on the current trajectory, I don't think my original plan is working out. So my new thinking is:

• Go with the pinning strategy that we've been using in NumPy and SciPy so far (in pyproject.toml, pin setuptools to the last version that is known to work for each release).
• Fix things up in numpy.distutils when they break on a best-effort basis.
• Projects that need significant features from numpy.distutils (Fortran support or BLAS/LAPACK detection for example) should migrate to a better build system next year. The two reasonable choices (Meson and CMake/scikit-build) are also still works-in-progress, but they're much more maintainable and performant.

[rgommers]

The question then becomes how we communicate changes. We could:

• deprecate numpy.distutils now, and say it will be removed from NumPy once Python 3.12 is our lowest supported version (should happen around 2025-2026), and will not be installed in any Python >=3.12 wheels and sdists.
• have a separate doc somewhere that gives more detailed migration advice, and for complicated cases offer help
• for projects/people that do want to stay with setuptools and need numpy.distutils features, say they are invited to help port those features to setuptools.

[rgommers]

By far the most widely used numpy.distutils feature is nested setup.py files. It would make sense to port that to setuptools I think. It does need a volunteer to work on this (or, if there's a strong need but no one volunteers, we may consider funding this work through NumPy project funds at some point). In the migration guide from numpy.distutils to other build systems (see gh-20875) I will link to this issue as the tracking issue.

[seberg]

Bumping the milestone, but if there is anyting important for the 1.23.0 release with respect to this issue, please make a note!

[rgommers]

Given that distutils was removed from the stdlib for the Python 3.12 development cycle 3 days ago (python/cpython#99061), this issue is likely to get some traffic. Here is the up-to-date info: https://numpy.org/devdocs/reference/distutils_status_migration.html.

NumPy does not build with the distutils that setuptools vendored and then keeps modifying, so I expect that if someone is trying to build NumPy with a CPython 3.12 development version, things will not work.

We are moving away from disutils, so are not going to spend the time keeping up with setuptools as a moving target. If someone wants to take this on, I'm happy to review and merge patches (however, keep in mind that it's nontrivial, patches should work also with older setuptools versions that we actually need on Python 3.8-3.11, e.g. we're pinning setuptools 59.2.0 in our pyproject.toml, and users of numpy.distutils may be using other versions). For the most recent relevant discussion on this, see #21114 (comment).

I'll link to a new tracking issue for the build system migration to Meson soon, need to summarize the current plan and status.

[tacaswell]

NumPy does not build with the distutils that setuptools vendored and then keeps modifying, so I expect that if someone is trying to build NumPy with a CPython 3.12 development version, things will not work.

I can confirm numpy does not install with either setuptool 59 or the latest (did not check any in between) with py312.

[CAM-Gerlach]

As a stopgap, someone could potentially put a forward-port of distutils on PyPI, and NumPy could depend on that at build and/or runtime. You could also potentially use the distutils vendored into Setuptools 59, which presumably has substantially fewer (if any) than in later versions, and would not be a moving target.

[mattip]

You could also potentially use the distutils vendored into Setuptools 59

Does this work to build NumPy? If I understand correctly, @tacaswell commented above that it does not.

[CAM-Gerlach]

On Setuptools <=59, Setuptools' copy of distutils doesn't replace the (missing in 3.12) stdlib version. However, NumPy could modify its imports to use that (assuming the changes/adjustments to work with that target would not be so much).

However, the forward-port on PyPI would be a much better solution overall, since it wouldn't require any changes to NumPy other than adding it to NumPy's build time and runtime requirements, and/or users manually installing it themselves. This will unblock building NumPy from source/sdist for 3.12, to not block early porting and testing to that version while the Meson migration is underway, or as a fallback plan in case it isn't fully ready. If it was not needed at runtime to support numpy.distutils, then it could be specified as a build-time requirement only, and NumPy binary wheels would not require installing it,

Of course, up to you all how you feel its best to approach that...though, IMO, we should still go ahead with the distutils forward-port regardless, for other packages that might need it.

[rgommers]

My $2c:

• Easiest is to do nothing, it's too early in the Python 3.12 release cycle to care about it, so it's not urgent for us. We typically start testing with Python development versions around March, somewhere around the alpha/beta stage.
• Second easiest is for CPython to revert the deletion and wait till right before the alpha stage to remove distutils again.
• Third option is for someone to work on one of the suggested workarounds and take responsibility for it. I recommend that that's not a NumPy maintainer, because we have more urgent/important things to do - like moving to Meson so we're actually ready for Python 3.12

[CAM-Gerlach]

Just FYI, the first alpha is typically released right around the same time as the previous version .0 release, and the nominal goal is to have as many of the planned removals in by the first alpha as practical, so that downstream packages can start testing with it early and maintainers, contributors, distributors, etc. can identify issues as soon as possible to make progress on fixing them, and provide feedback upstream.

In this case, Alpha 1 was released a bit later than usual, around a week or two ago, and while it was specifically planned to be sure that the distutils removal was included for those reasons, due to an oversight it was still left in and only removed now, for Alpha 2. It is possible that the removal is reverted and deferred to the next release, if it looks like it would still be too disruptive to the ecosystem, but that would be probably at the end of the alpha stage or during the beta, if it happens.

Its certainly a possibility its reverted, but for better or for worse, the Setuptools changes with overriding the stdlib distutils mean that any packages, whether with legacy or PEP 517 builds, that are building in an environment with Setuptools installed (which is the great majority of the time) and don't explicitly require <=59 or happen to have it (probably not so likely with CPython 3.12), will already behave the same whether or not distutils is removed from the stdlib on the CPython side. because they will always be getting the Setuptools' bundled distutils.

[tacaswell]

I also have little interest in doing the work to put a forward port on pypi. I would much rather put the effort into helping packages affected by this move to meson.

Does this work to build NumPy? If I understand correctly, @tacaswell commented above that it does not.

It does not work out of the box, but I have not tried setting the relevant ENV yet.

[rgommers]

Okay, if someone really needs NumPy for Python 3.12 right now, the meson branch should work on Linux and macOS (see #22546 (comment)). For early adopters only, and there are no docs yet - but there's a GitHub Actions CI file in that branch, so I'm sure it can be figured out without too much trouble.

EDIT: scratch that, NumPy is updated but Meson itself still relies on distutils: mesonbuild/meson#7702. Guess we'll have to wait a little while.

[eli-schwartz]

Sysconfig is not (yet) a full replacement for distutils, because Meson sneakily uses a distutils Extension() object and checks whether it depends on libpython to figure out whether extensions need to be linked to the DLL on the current platform.

I intend to discuss this with CPython soon, but in the meantime, setuptools on python 3.12 will inject distutils and Meson's use is so simple that it currently still works with setuptools' patches (obviously due to setuptools' lack of API stability, no promises...), so...

... Installing setuptools as a build dependency should work as a quick hack to run Meson on the alphas. Obviously, this is a quick hack and Meson is not going to rely on this for the final release of python 3.12. ;)

[tacaswell]

With the hint to install setuptools > 60 I was able to get numpy to compile and install

pip install -v . --no-build-isolation
meson setup build --prefix=/home/tcaswell/.virtualenvs/bleeding/
ninja -C build
meson install -C build

from the meson branch.

[tacaswell]

The above commands does not install the dist-info, copying that directory (and accepting the lies) seems to get the rest of the stack building again.

[tacaswell]

I'm also getting numpy reporting a version of '0+unknown' which is breaking scipy's build, will have to look into why that is ...

[rgommers]

Should be fixed in the meson branch now, I hooked up versioneer.

[nilskattenbeck-bosch]

What is the intended plan for the usage of distutils inside numpy.ctypeslib.load_library?

[rgommers]

I'll close this issue, since we are basically done with this transition. NumPy 1.26.x defaults to Meson as the new build system, and numpy.distutils is deprecated and remains available for Python version up to 3.11.x (it is no longer installed for >=3.12).

No distutils code is used anymore on 3.12; there may be a handful of usages left (in f2py) and in the docs. We'll clean those up as we go.