MAINT: add Python 3.10 wheels

[tupui]

Build wheels for Python 3.10.

xref scipy/scipy#14567 and #124

[tupui]

@tylerjereddy for azure, do I need to do AZURE_PYTHON_VERSION: "3.9" as you had in your PR?

I also saw I did not update Cython everywhere. Will come with the next commit.

[tylerjereddy]

For ARM64 Python 3.10 issue, looks like we need to bump pip version up slightly to pick up this patch: pypa/pip#10252 -- it may be worth trying to bump the version of the multibuild submodule to the latest version to see if that helps.

for azure, do I need to do AZURE_PYTHON_VERSION: "3.9" as you had in your PR?

Not sure, I guess NumPy was/is doing that because 3.10 wasn't available for the base CI image yet, but 3.10 release candidate was available in the docker images used to build the wheels?

[charris]

do I need to do AZURE_PYTHON_VERSION: "3.9"

Python is used in two places: to run the docker image and inside the docker image. Azure doesn't have 3.10 yet, so 3.9 is used to run the docker image, which in turn uses 3.10.0 internally. For 3.9, it was about three weeks after release before azure had it.

[tupui]

do I need to do AZURE_PYTHON_VERSION: "3.9"

Python is used in two places: to run the docker image and inside the docker image. Azure doesn't have 3.10 yet, so 3.9 is used to run the docker image, which in turn uses 3.10.0 internally. For 3.9, it was about three weeks after release before azure had it.

ok so I need to do this right

AZURE_PYTHON_VERSION: "3.9"
MB_PYTHON_VERSION: "3.10.0"
DOCKER_TEST_IMAGE: "multibuild/focal_{PLAT}"  # this as well?

[tupui]

@charris please commit if this is correct 😃

[tupui]

For ARM64 Python 3.10 issue, looks like we need to bump pip version up slightly to pick up this patch: pypa/pip#10252 -- it may be worth trying to bump the version of the multibuild submodule to the latest version to see if that helps.

Sorry @tylerjereddy I don't get how to update the submodule nor the pip version...

[tylerjereddy]

I usually do something like git submodule update --init then manually cd multibuild and pull in the latest devel branch from https://github.com/matthew-brett/multibuild ; then back up to main repo and commit the change to where the submodule is pointing

[charris]

NumPy has also moved to manylinux2014 everywhere. See https://github.com/MacPython/numpy-wheels/blob/main/azure-pipelines.yml.

EDIT: Also https://github.com/MacPython/numpy-wheels/blob/main/.travis.yml

[charris]

Note that 3.10 isn't yet available for OS X.

[tupui]

NumPy has also moved to manylinux2014 everywhere. See https://github.com/MacPython/numpy-wheels/blob/main/azure-pipelines.yml.

EDIT: Also https://github.com/MacPython/numpy-wheels/blob/main/.travis.yml

Ok I think I fixed it.

Note that 3.10 isn't yet available for OS X.

You are talking about this one?

        osx-Py310-universal2:
          AZURE_PYTHON_VERSION: "3.9"
          MB_PYTHON_VERSION: "3.10"
          MB_PYTHON_OSX_VER: "10.9"
          PLAT: universal2

Because the not universal seemed to go.

[tylerjereddy]

heh, the manylinux version bump is what I was trying to avoid b/c there are a few things that will need adjusting I think, but we are probably long overdue on that anyway (as long as the newer pip version required of users is "ok" with us, but if it works for NumPy should be "ok" for us I guess).

[charris]

Heh, the manylinux version bump is what I was trying to avoid

There were enough changes in the 1.22.x wheels that a NEP might have been useful.

• Change to ILP64 OpenBLAS
• Dropping manylinux1 and manylinux2010
• Dropping 32 bit Python 3.10 manylinux wheels
• Dropping Python 3.7

The changes were all discussed in triage meetings or on the mailing list, but that probably didn't help downstream projects.

[rgommers]

I think we should follow the NumPy choices, except let's not do ILP64 OpenBLAS. Happy to drop manylinux2010.

[tupui]

@rgommers If I understood everything, should be it.

Travis is failing for ARM64-Linux-Py39 with ERROR: no supported wheels found. I am not sure to what it refers to. Only difference is the Cython version but it should be there no?

[rgommers]

The log is, as usual with pip, remarkably uninformative. Comparing with the successful build for Python 3.8, this is what it should have looked like:

...
You should consider upgrading via the '/venv/bin/python -m pip install --upgrade pip' command.
Looking in links: https://5cf40426d9f06eb7461d-6fe47d9331aba7cd62fc36c7196769e4.ssl.cf2.rackcdn.com
Processing ./wheelhouse/scipy-1.8.0.dev0+1759.3159565-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Requirement already satisfied: numpy>=1.17.3 in /venv/lib/python3.8/site-packages (from scipy==1.8.0.dev0+1759.3159565) (1.19.2)
Installing collected packages: scipy
Successfully installed scipy-1.8.0.dev0+1759.3159565

It's unclear why the Rackspace URL didn't work - it's hard to access without the password/token. I've restarted the failed job, let's see if it happens again.

[rgommers]

The appveyor problem seems to be the hex float

#define M_PI64 0x1.921fb54442d18p+1

a c99 thing apparently not supported in the MSVC version used for the wheel build

Argh, continuing headache with that one. I hope @WarrenWeckesser knows the right thingy to replace it with.

[charris]

Argh, continuing headache with that one.

Lot of those in NumPy main too, it no longer compiles with visual studio 2015 (the appveyor default). The version can be changed by adding image: Visual Studio 2017 to the appveyor.yml, see https://www.appveyor.com/docs/build-environment/

[rgommers]

Ah okay, that's a better solution. The version matching Python 3.7 is Visual Studio 2017 version 15.7, so that's an improvement.

[tupui]

Argh, continuing headache with that one.

Lot of those in NumPy main too, it no longer compiles with visual studio 2015 (the appveyor default). The version can be changed by adding image: Visual Studio 2017 to the appveyor.yml, see https://www.appveyor.com/docs/build-environment/

@charris looks like this is causing an error. Or I am not using this correctly.

[charris]

Hmm, looks like that example wasn't quite correct, see https://www.appveyor.com/docs/appveyor-yml/. Maybe the capitalization needs to be image: Visual Studio 2017?

[tupui]

Hmm, looks like that example wasn't quite correct, see https://www.appveyor.com/docs/appveyor-yml/. Maybe the capitalization needs to be image: Visual Studio 2017?

mmm no I had this right (just checked and I even have PyCharm complaining if I change the casing).

Not sure about the output, but seems like Microsoft SDKs is not present in C:\Program Files then.

dir "C:/Program Files/Microsoft SDKs/Windows"
120 Volume in drive C has no label.
121 Volume Serial Number is 1CB8-D4AA
122
123 Directory of C:\Program Files\Microsoft SDKs
124
125File Not Found

I am also not sure about the paths C:\Program Files (x86)\Microsoft Visual Studio 15.7. Purely based on what @rgommers said here.

[charris]

I'm kind of lost then. That scipy needs fortran complicates things.

[tupui]

Ok fixed the path thing, let's see now where it fails... 😅

[charris]

Hmm, looks like MSVC 2017 compiler is 14.1. I'm looking at https://wiki.python.org/moin/WindowsCompilers, which isn't very explicit.

[tupui]

Then this should be fine??

- copy "C:\Program Files (x86)\Microsoft Visual Studio\VC\redist\x86\Microsoft.VC141CRT\msvcp141dll" "build\lib.win32-%PYTHON_VERSION%\scipy\.libs\"
- copy "C:\Program Files (x86)\Microsoft Visual Studio\VC\redist\x64\Microsoft.VC141CRT\msvcp141dll" "build\lib.win-amd64-%PYTHON_VERSION%\scipy\.libs\"

But before this, it's not building NumPy's wheel now.

[charris]

I wouldn't worry about Python 3.10 on windows yet, there are no NumPy 3.10 wheels for that platform yet.

[tupui]

Oh ok but then it's ready because Travis and Azure are passing. I will comment the changes on appveyor and it should be green.

[charris]

Appveyor was already failing because of the hex float, so I think the MSVC update is still needed. But it could be another PR.

[charris]

Seems to be working.

[charris]

Thanks @tupui .

[tupui]

Thanks @tupui .

Well thanks to you @charris!

[tylerjereddy]

I went ahead and purged out about 6 months of old SciPy nightly (weekly) wheels uploads to the anaconda site--there shouldn't be anything more than 7-8 days old there now. Thanks for the reminder.