Python 3.10 manylinux wheels

[hrnciar]

Hello.

Could you please release CPython 3.10 manylinux wheels on PyPI?

Python 3.10.0 release candidate 1 was released this week: https://www.python.org/downloads/release/python-3100rc1/

It should be quite safe to release wheels for release candidate. ABI for Python 3.10 should be frozen from now on, even for bugfixes. From release notes:

There will be no ABI changes from this point forward in the 3.10 series and the goal is that there will be as few code changes as possible.

Thank You.

[tylerjereddy]

Matching issue in SciPy is cross-linked above--I'll keep an eye on what Chuck/the NumPy team decides to do.

[charris]

To build wheels for 3.10 currently requires that it be available in manylinux2014 (linux), and azure (windows and mac). It was about three weeks after 3.9 was released before we could build wheels for all the platforms.

[charris]

Note that rc1 did break a couple of NumPy tests.

[charris]

Checking manylinux2014, likely the first out the door, they currently support 3.10.0b2. I imagine that will be updated in the next few weeks, maybe when rc2 comes out, if it comes out.

[charris]

Hmm, it does look like manylinux2014 does support rc1.

3.10.0rc1 is now in the latest images.
ABI is now stable: https://www.python.org/downloads/release/python-3100rc1/

There was no PR for that, it must have just been pushed. I think we can release linux wheels for that.

[charris]

I can build manylinux2014 Python3.10.0rc1 wheels, but they can't be tested until 3.10 is added to the testing docker images. @matthew-brett @mattip Who does that?

[mattip]

Me. The repo is https://github.com/multi-build/docker-images, I will update it.

[mattip]

I updated the docker images. The version available via apt get after adding the deadsnake apt repo is

Python 3.10.0a6 (default, Mar  2 2021, 02:01:08) [GCC 5.4.0 20160609] on linux

I hope that is recent enough? Note these are xenial images, but xenial is EOL. I guess I should update the whole thing to focal (20.04)

[hrnciar]

I updated the docker images. The version available via apt get after adding the deadsnake apt repo is

Python 3.10.0a6 (default, Mar  2 2021, 02:01:08) [GCC 5.4.0 20160609] on linux

I hope that is recent enough? Note these are xenial images, but xenial is EOL. I guess I should update the whole thing to focal (20.04)

This seems quite outdated, the latest released version is Python 3.10.0rc1.

[mattip]

The 3.10rc1 is available in the focal images. But now a new problem: ubuntu has dropped 32-bit i686. How much longer will we ship wheels for that platform?

[hroncok]

Honestly, I would not bother shipping 32-bit i686 wheels for recent Python versions. Ubuntu has dropped it, so did Fedora...

[charris]

@mattip, I'm trying to recall why we use different images for building and testing, something to do with versions?

[charris]

New error: undefined symbol: _PyGen_Send

EDIT: Needs a more recent cython .

[charris]

@mattip The 64 bit wheels on Intel now work, but we need multibuild/focal_arm64v8 for ARM64.

[charris]

The Python 3.10 wheels for Intel 64 bit linux should be available from the nightly builds in an hour or so. You can install them like so:

pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy

[hroncok]

I am not sure why this was closed. No 3.10 manylinux wheels are available on PyPI.

[charris]

@hroncok Because we aren't making an official release with 3.10 wheels until 3.10 is released. The wheel is for testing, in particular for downstream projects who test against the prereleases.

[hroncok]

I don't understand why not. Could you please explain this? I think that having the wheels on PyPI before the final Python 3.10 release (instead of introducing an arbitrary delay after is is released) is beneficial to everybody. Note that:

1. The ABI is locked now. No ABI changes will happen before the final release. There is no reason to wait.
2. Even if the above promise ends up broken, wheels have release numbers and can be re-generated for a single version.

IMHO users will not know about pypi.anaconda.org/scipy-wheels-nightly, they will run pip install numpy instead.

[hroncok]

Clarification: I understand the intention to release the wheels for testing first. However let's not close this issue until the wheels are (tested and) released on PyPI?

[hroncok]

I think that having the wheels on PyPI before the final Python 3.10 release (instead of introducing an arbitrary delay after is is released) is beneficial to everybody.

One more thing that come to my mind after I posted this: Other projects that would like to release their 3.10 wheels are waiting for this. Waiting after the final release of Python 3.10 only makes the delay chained. The time window between the rc1 and final is exactly the time to figure all this out, so users can enjoy 100% pain-free experience since the Python 3.10 release day.

[mattip]

... why we use different images for building and testing

I think it had to do with arm64 support and the default images based on trusty (Ubuntu 14.04) or manylinux. It seems to make sense to test on a different system to make sure later glibc versions are not failing tests.

[mattip]

we need multibuild/focal_arm64v8 for ARM64.

It is now available.

[brettcannon]

As CPython core dev I would say having projects have the ability to get a 3.10 wheel built for testing during the RC at least -- if not the betas -- would be beneficial to both Python itself and the general community as having dependencies that one can't get built locally means everyone ends up postponing supporting new Python releases for months as dependencies trickle out support.

[mattip]

@brettcannon could you comment on the policy not to release wheels on PyPI rather to make wheels available through a secondary index, in this case pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy ? What would the PyPA consider as best practice and when should wheels appear on PyPI for a new python version? How should those wheels be labelled to indicate they may not be final, and what are the appropriate command line arguments for wheel building to generate those labels?

[rgommers]

While I'm perfectly happy with uploading 3.10 wheels for NumPy and other packages, I do want to point out a few things:

• We are already asking maintainers of packages that depend on NumPy to use https://pypi.anaconda.org/scipy-wheels-nightly/simple/ in their CI to test against NumPy master. And scipy/pandas/etc. wheels are there as well. So it's not that strange a location.
• @charris had already anticipated supporting Python 3.10 in Numpy 1.21.x and we had CI in place, therefore the python_requires had a <3.11.0 upper bound. But in general this will not always be true. SciPy 1.7.x has <3.10.0, so it is not correct to upload wheels to the current latest release - that will require a new 1.7.2 or 1.8.0 release. (and yes, the metadata is right - all dependencies should have a correct upper bound)

For example, the Python that is installed via the app store in Windows will be updated to 3.10, Python in homebrew will be updated to 3.10, users of Fedora 35 (that is to be released in October as well) will have Python 3.10 by default. Many users will use Python 3.10 since day 0.

This is not a good thing, those distributors are causing problems by doing that on day zero. Python is a little bit special, but not that special. It is just one dependency in a very large dependency graph, so this makes very little sense. This super-agressive upgrading should be discouraged. Because the end result will indeed be "beginners end up being confused".

Any project normally has a choice between:

1. guaranteeing API and ABI stability, so a downstream project building against release x.y will have packages up that also work with release x.y+1 (this is what NumPy does).
2. retaining the freedom to break API or ABI, at which point you pay with delayed uptake and "early adopter" issues.

Python is doing the first half of (2) and is asking for extra work/releases from their downstream packages to avoid paying the price of (2). If it's the only project that does that (and I think that's the case), then that's still a cost that can be paid fairly easily, however if other packages who choose to not guarantee ABI stability (PyTorch and TensorFlow come to mind) would do the same, that cost quickly gets out of hand. It's not a great habit.

[hroncok]

The rc versions of Python do guarantee ABI stability, which was exactly the point why the wheels were requested after rc went out.

[rgommers]

I understand that, but that's only a 2 month period. It's a very nice improvement, don't get me wrong. However, not all projects are going to do a release (at all, or with Python 3.10 support) in those two months. So it is still going to cause problems if distributors of Python cause beginners to end up with 3.10 straight away on release.

[encukou]

I understand that, but that's only a 2 month period.

What other time period would you suggest to use? Your arguments could just as well apply to two years after the freeze.

[rgommers]

Eh no, they don't. Most important Python projects are using a 3-9 month release cycle, with the typical spacing for minor releases being 6 months.

@encukou I think you're missing the main point. I do not expect CPython to suddenly become ABI stable between minor releases, and a 2 month window is already a lot nicer than nothing. The problem is the expectation that it's okay to have tons of users upgrade to a new release and expect it to all work fine. What Fedora, Homebrew and the Windows App Store are doing is the main problem here, especially coupled with the expectation that it's a good idea to use all those Python install methods together with PyPI.

[encukou]

Ah, so the issue is in upgrading? Yes, I guess all Python distributors should either preserve the old version of Python on upgrade, or not make the upgrade automatic. (edit: I filed a Fedora bug)
I thought we're talking more about new installs of Python – if we agree that new users should get 3.10 by default when 3.10 comes out, then we do need wheels for 3.10 soon.

Most important Python projects are using a 3-9 month release cycle

Important projects are also tested with alphas/betas. (Not necessarily by the upstream maintainers.) For rc1 they can either "just" build new wheels, or (in hopefully rare cases) make a point release with a specific fix.

I do not expect CPython to suddenly become ABI stable between minor releases

Not suddenly, but there is a limited API subset that is ABI stable, and there's a bunch of effort ongoing to make it more useful. I don't expect NumPy to use it any time soon – as it's be slower than the less stable API – but for some projects the performance/maintenance cost tradeoff should be worth it.

[rgommers]

I thought we're talking more about new installs of Python – if we agree that new users should get 3.10 by default when 3.10 comes out, then we do need wheels for 3.10 soon.

We are talking about both upgrades and new installs. New users should not get 3.10 by default on day 1. Unless they understand the tradeoffs and make a conscious choice that they want to be early adopters. But for something like the Windows App Store, that's certainly not the case. The default in Windows App Store should be 3.9.x for some time (1-3 months?) after 3.10 release date.

There just aren't enough new/improved things in Python 3.10 that that outweighs some libraries not being installable at all and beginners wasting a lot of time.

For rc1 they can either "just" build new wheels

I'm sure you are well aware of how complex it is to build wheels in general, and that's why you're using quotes here. It takes a lot of work - we have to move the NumPy wheel build machinery from Azure DevOps to GitHub Actions it looks like for example. Projects more complex than NumPy may have other blockers: maintainer bandwidth, QA trajectories for any official binaries, size limit issues on PyPI that take weeks to get a response, etc.

I suggest to make an inventory after the release of what is missing. My prediction is that despite requests like this issue and people spending time on it, there will be a lot of libraries especially for scientific computing and ML/AI, that will not be available for 3.10 by the release date.

[encukou]

The default in Windows App Store should be 3.9.x for some time (1-3 months?) after 3.10 release date.

A Windows question would be for @brettcannon, I believe?

Can/should we determine who new users are and separate them from old users who are better early adopters? (And should we rather use "early adopter" for users of things like Fedora Rawhide prereleases, which had 3.10 since the alphas?)

AFAICS, the current schedule is the RC period (2 months) for the ecosystem to prepare & test, and the final release is for all users. I don't think we can improve that for 3.10. Should we add another release date for App-Store-like releases for 3.11?

I'm sure you are well aware of how complex it is to build wheels in general, and that's why you're using quotes here.

Yes, exactly.
But there are also projects for which this is easier (or zero-effort if they're pure-python), but they're blocked by the more heavy-weight foundational projects. As I see it, NumPy is in a small category of hard+urgent packages.

I suggest to make an inventory after the release of what is missing.

Definiely.
The 3.9 wheels issue was based on a user report; this one was filed proactively based on the "inventory" for 3.9.
If you have any additions to the "inventory", let me know :)

[brettcannon]

Nope, a Windows question is a @zooba question.

[zooba]

Each Python version in the Windows Store is a separate app (much as I dislike that model, unfortunately the CPython release model forces it). People who want 3.10 have to explicitly look it up and install it, otherwise they are stuck on 3.n forever.

Now, the default python.exe stub that ships with Windows gets updated on its own schedule. Right now, that's basically when I feel comfortable with ecosystem status (my current smoke test is "can I pip install jupyter ... and render a chart in a notebook"). If the ecosystem is lagging, it won't be switched - certainly doesn't happen automatically on the first day. However, it does happen instantly when I decide to do it, because Microsoft controls the version it goes to on the server side.

There will be more downloads on day 0 from python.org than through the store stub ;)

[encukou]

my current smoke test is "can I pip install jupyter ... and render a chart in a notebook"

Thanks! That's essentially the same packages we are looking at. It's nice to have a confirmation :)

[rgommers]

Now, the default python.exe stub that ships with Windows gets updated on its own schedule. Right now, that's basically when I feel comfortable with ecosystem status (my current smoke test is "can I pip install jupyter ... and render a chart in a notebook"). If the ecosystem is lagging, it won't be switched - certainly doesn't happen automatically on the first day.

That's great to hear, thanks for the explanation @zooba.

If you have any additions to the "inventory", let me know :)

Beyond what's in https://bugzilla.redhat.com/show_bug.cgi?id=1989081, I'd suggest (packages with compiled code only; pure Python stuff like Dask and Xarray will work already):

• Accelerators: Cython, Numba
• Other scientific computing libraries: CuPy, scikit-learn, scikit-image, statsmodels, spaCy
• Deep learning libraries: PyTorch (I just opened an issue for it, linked above), TensorFlow, and JAX/jaxlib.

[hroncok]

Cython has a pure Python universal wheel available, so the manylinux etc. wheel is not blocking, just faster.

[charris]

Because NumPy wheels are still azure based, the main blocker for supporting new Python versions is their availability on azure for OS X and windows.

[rgommers]

Did anyone look for other projects who run 3.10 Windows/macOS on Azure? The Windows App Store is supposed to have 3.10 final by now.

Probably we should ensure we're migrated to GHA and cibuildwheel before the Python 3.11.0 release?

[hugovk]

Azure Pipelines will have 3.10 next week when new images are released. Tracking issue:

• Add Python 3.10 (release) actions/runner-images#4226 (comment)

GitHub Actions and Azure Pipelines use the same images, but GHA can also install 3.10 on demand, and AZP cannot. Last year, they suggested upvoting this feature request for it to do so.

(So GHA installs will be slightly faster next week when using pre-cached 3.10 instead of install on-demand.)

[hugovk]

Having said that, does Azure Pipelines have 3.10-dev?

Some CIs have -dev tracking nightly builds (Travis IIRC), others have it tracking most recent pre-release/release (GHA).

In either case, would they be good enough for a release? Or at least to start work setting stuff up? Might find some bugs caused by the sudden introduction of a two-digit minor version number :)

[zooba]

In either case, would they be good enough for a release?

These should be RC or later, so if they're not good enough for a release, we'd love to hear why (over on python-dev or https://discuss.python.org, but I'm happy to take if the answer is posted here).

Changes during RC are usually less impactful than between .0 and .1 (and if they're not, everyone will be very thankful we made whatever fix was needed!), so it should be at least as safe as after release.