add support for building wheels for windows on arm64 #920

niyas-sait · 2021-11-15T13:32:40Z

This patch enables building python wheels for windows on arm64 targets.

There is no public CI available yet for win/arm64 targets but expecting Azure support soon. This could help to start releasing binary wheels for python once the CI systems are upgraded.

for more information, see https://pre-commit.ci

…el into enable_win_arm64

for more information, see https://pre-commit.ci

…el into enable_win_arm64

henryiii

Seems reasonable enough to me. @joerick?

joerick

Cool! Have you tested this on an arm64 machine @nsait-linaro? It looks good so far, just wanted to check. I can't test it locally, and we don't have a CI runner for this so we'd have to take your word for it for now!

Other things we'd need to merge this-

the build_platforms.toml file is actually generated/modified weekly by bin/update_pythons.py, so to make this change stick, we'd need to make sure that script still works and retains arm64. Could you try that and make sure it does the right thing?
Docs updates. Off the top of my head, we'd need to update the build id table in the CIBW_BUILD section, and the table in the README. For now, in the README, could we note that win-arm64 support is 'experimental', since we can't test this in CI yet.

joerick · 2021-11-17T09:16:51Z

cibuildwheel/windows.py

+    # pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
+    if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"):
+        call(
+            [
+                "python",
+                "-m",
+                "pip",
+                "install",
+                "--force-reinstall",
+                "--upgrade",
+                "pip",
+                *dependency_constraint_flags,
+            ],
+            env=env,
+            cwd=CIBW_INSTALL_PATH,
+        )
+


This might be overcomplicated. If the pip executable is missing, we should already detect that. Could you try looking for whatever you're missing, like-

if python_configuration.arch == "ARM64" and not (installation_path / "Scripts" / "piparm64.exe").exists(): requires_reinstall = True

or if that doesn't work, just this would be okay for now I think-

if python_configuration.arch == "ARM64": requires_reinstall = True

probably better to move this up to after the requires_reinstall set line too.

That's exactly what I tried at first but unfortunately didn't work.

The issue is that pip bundled with python 3.10 builds executables such as pip.exe for x64 platform. The first re-install of pip updates pip module but builds pip.exe using the old pip which still generates x64 executable. But the second re-install uses updated pip and correctly builds pip.exe for the target.

ah, hmm, okay. I think I understand. What is the Version(get_pip_version(env)) < Version("21.3") about? And are you assuming that requires_reinstall is always true on arm64?

I'd expect this is a temporary problem and should be fixed once a newer pip is bundled? That's a guess, though, and it would be nice to verify this really was fixed in 21.3 and is not just a problem with nuget.

Yes, the fix is available from pip release 21.3 so we don't have to do the extra install.

Got it. I've suggested adding this explanation as a comment, which will help let us know when it can be removed.

cibuildwheel/windows.py

niyas-sait · 2021-11-17T11:40:53Z

Cool! Have you tested this on an arm64 machine @nsait-linaro? It looks good so far, just wanted to check. I can't test it locally, and we don't have a CI runner for this so we'd have to take your word for it for now!

Yes, I've tried it on my surface pro and tried building a few packages like pyzmq and it seems to work.

I am working on a patch to get the cross-compilation working but require a setuptools update to help with cross-compilation.

niyas-sait · 2021-11-17T11:42:51Z

Thanks, @joerick and @henryiii for the review. I've addressed the review comments in the latest commits.

niyas-sait · 2021-11-18T07:18:00Z

Failing CI seems like an infrastructure issue. Can someone please rerun the test ?

henryiii · 2021-11-18T13:45:52Z

Yes, CircleCI has been timing out here in the last two days or so. Not sure when it's going to be fixed. I think the latest main build has the timeout failure too.

niyas-sait · 2021-11-19T10:13:54Z

Yes, CircleCI has been timing out here in the last two days or so. Not sure when it's going to be fixed. I think the latest main build has the timeout failure too.

Thanks

joerick · 2021-11-19T17:12:33Z

cibuildwheel/resources/build-platforms.toml

@@ -87,6 +87,8 @@ python_configurations = [
  { identifier = "cp39-win_amd64", version = "3.9.8", arch = "64" },
  { identifier = "cp310-win32", version = "3.10.0", arch = "32" },
  { identifier = "cp310-win_amd64", version = "3.10.0", arch = "64" },
+  { identifier = "cp310-win_arm64", version = "3.9.9", arch = "ARM64" },


This identifier appears to be wrong? should be cp39... i think.

Shouldn't we have two, possibly, then? 3.9.9 and 3.10.0? Not sure what's actually available.

This identifier appears to be wrong? should be cp39... i think.

Ah, copy-paste error. I will fix it.

Shouldn't we have two, possibly, then? 3.9.9 and 3.10.0? Not sure what's actually available.

Yes, both are available. I thought I had it in my patch. Will fix thtat

joerick · 2021-11-19T17:16:26Z

cibuildwheel/windows.py

+    # pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
+    if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"):
+        call(
+            [
+                "python",
+                "-m",
+                "pip",
+                "install",
+                "--force-reinstall",
+                "--upgrade",
+                "pip",
+                *dependency_constraint_flags,
+            ],
+            env=env,
+            cwd=CIBW_INSTALL_PATH,
+        )
+


ah, hmm, okay. I think I understand. What is the Version(get_pip_version(env)) < Version("21.3") about? And are you assuming that requires_reinstall is always true on arm64?

henryiii

Python identifier needs to match version. Then I'm happy with it. ;)

…el into enable_win_arm64

niyas-sait · 2021-11-22T06:56:49Z

Python identifier needs to match version. Then I'm happy with it. ;)

This has been fixed in the latest commit. Thanks

joerick

Just a couple tweaks needed I think!

joerick · 2021-11-23T09:02:49Z

README.md

+|   | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | Windows Arm64 | manylinux<br/>musllinux x86_64 | manylinux<br/>musllinux i686 | manylinux<br/>musllinux aarch64 | manylinux<br/>musllinux ppc64le | manylinux<br/>musllinux s390x |
 |---------------|----|-----|-----|-----|----|-----|----|-----|-----|
-| CPython 3.6   | ✅ | N/A | ✅  | ✅  | ✅ | ✅  | ✅ | ✅  | ✅  |
-| CPython 3.7   | ✅ | N/A | ✅  | ✅  | ✅ | ✅  | ✅ | ✅  | ✅  |
-| CPython 3.8   | ✅ | ✅  | ✅  | ✅  | ✅ | ✅  | ✅ | ✅  | ✅  |
-| CPython 3.9   | ✅ | ✅  | ✅  | ✅  | ✅ | ✅  | ✅ | ✅  | ✅  |
-| CPython 3.10  | ✅ | ✅  | ✅  | ✅  | ✅ | ✅  | ✅ | ✅  | ✅  |
-| PyPy 3.7 v7.3 | ✅ | N/A | ✅  | N/A | ✅¹ | ✅¹  | ✅¹ | N/A | N/A |
+| CPython 3.6   | ✅ | N/A | ✅  | ✅  | N/A | ✅  | ✅  | ✅ | ✅  | ✅  |
+| CPython 3.7   | ✅ | N/A | ✅  | ✅  | N/A | ✅ |✅  | ✅ | ✅  | ✅  |
+| CPython 3.8   | ✅ | ✅  | ✅  | ✅   | N/A | ✅ | ✅  | ✅ | ✅  | ✅  |
+| CPython 3.9   | ✅ | ✅  | ✅  | ✅  | ✅² | ✅ | ✅  | ✅ | ✅  | ✅  |
+| CPython 3.10  | ✅ | ✅  | ✅  | ✅  | ✅² | ✅ | ✅  | ✅ | ✅  | ✅  |
+| PyPy 3.7 v7.3 | ✅ | N/A | ✅  | N/A | N/A | ✅¹ | ✅¹  | ✅¹ | N/A | N/A |


Looks like this table markdown isn't parsing any more. Do the first two Pythons have too many columns?

The line in-between is wrong.

|---------------|----|-----|-----|-----|----|-----|----|-----|-----|

->

|---------------|----|-----|-----|-----|-----|----|-----|----|-----|-----|

I've tried in a markdown editor and it didn't spot that. Sorry I will fix that.

joerick · 2021-11-23T09:07:12Z

cibuildwheel/windows.py

    if requires_reinstall:
        # maybe pip isn't installed at all. ensurepip resolves that.
        call(["python", "-m", "ensurepip"], env=env, cwd=CIBW_INSTALL_PATH)

+    # pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable


Suggested change

# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable

# pip older than 21.3 builds executables such as pip.exe for x64 platform.

# The first re-install of pip updates pip module but builds pip.exe using

# the old pip which still generates x64 executable. But the second

# re-install uses updated pip and correctly builds pip.exe for the target.

# This can be removed once ARM64 Pythons (currently 3.9 and 3.10) bundle

# pip versions newer than 21.3.

joerick · 2021-11-23T09:08:06Z

cibuildwheel/windows.py

+    # pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
+    if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"):
+        call(
+            [
+                "python",
+                "-m",
+                "pip",
+                "install",
+                "--force-reinstall",
+                "--upgrade",
+                "pip",
+                *dependency_constraint_flags,
+            ],
+            env=env,
+            cwd=CIBW_INSTALL_PATH,
+        )
+


Got it. I've suggested adding this explanation as a comment, which will help let us know when it can be removed.

joerick

Looks good!

henryiii

After this goes in, we should add a third note on the table about musllinux not being supported on Alpine 3.14.

niyas-sait and others added 7 commits November 15, 2021 13:30

add support for building wheels for windows on arm64

325b1c3

[pre-commit.ci] auto fixes from pre-commit.com hooks

e8898c0

for more information, see https://pre-commit.ci

Fix main_platform_test.py

cfeba93

Merge branch 'enable_win_arm64' of github.com:nsait-linaro/cibuildwhe…

76248e2

…el into enable_win_arm64

[pre-commit.ci] auto fixes from pre-commit.com hooks

c91d158

for more information, see https://pre-commit.ci

Remove unrequired explicit nuget source

9bf6b0a

Merge branch 'enable_win_arm64' of github.com:nsait-linaro/cibuildwhe…

454f5d2

…el into enable_win_arm64

henryiii approved these changes Nov 17, 2021

View reviewed changes

joerick reviewed Nov 17, 2021

View reviewed changes

niyas-sait added 2 commits November 17, 2021 11:12

add python arm64 3.9.9 version

dfae980

extend update_pythons.py to add win/arm64 support

aaf23c9

update documentation for new win/arm64 platform

dffac66

style: include ARM64 in literal

4cc5fb0

joerick reviewed Nov 19, 2021

View reviewed changes

henryiii closed this Nov 21, 2021

henryiii reopened this Nov 21, 2021

henryiii requested changes Nov 21, 2021

View reviewed changes

niyas-sait added 2 commits November 22, 2021 06:54

Fix python 3.9 identifier for win_arm64

19e11ff

Merge branch 'enable_win_arm64' of github.com:nsait-linaro/cibuildwhe…

ff6ac4f

…el into enable_win_arm64

joerick reviewed Nov 23, 2021

View reviewed changes

Fix table layout issues in README and add note about arm64 pip issue

05becce

niyas-sait requested a review from henryiii November 23, 2021 17:50

joerick approved these changes Nov 24, 2021

View reviewed changes

henryiii approved these changes Nov 24, 2021

View reviewed changes

henryiii merged commit 594d89f into pypa:main Nov 24, 2021

henryiii mentioned this pull request Dec 9, 2021

Add ARM64 as a new Windows platform scikit-build/scikit-build#612

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for building wheels for windows on arm64 #920

add support for building wheels for windows on arm64 #920

niyas-sait commented Nov 15, 2021

henryiii left a comment

joerick left a comment

joerick Nov 17, 2021 •

edited

niyas-sait Nov 17, 2021

joerick Nov 19, 2021

henryiii Nov 22, 2021

niyas-sait Nov 22, 2021

joerick Nov 23, 2021

niyas-sait commented Nov 17, 2021

niyas-sait commented Nov 17, 2021

niyas-sait commented Nov 18, 2021

henryiii commented Nov 18, 2021

niyas-sait commented Nov 19, 2021

joerick Nov 19, 2021

henryiii Nov 21, 2021

niyas-sait Nov 22, 2021

niyas-sait Nov 22, 2021

joerick Nov 19, 2021

henryiii left a comment

niyas-sait commented Nov 22, 2021

joerick left a comment

joerick Nov 23, 2021

henryiii Nov 23, 2021

henryiii Nov 23, 2021

niyas-sait Nov 23, 2021

joerick Nov 23, 2021

joerick Nov 23, 2021

joerick left a comment

henryiii left a comment

-    # pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
+    # pip older than 21.3 builds executables such as pip.exe for x64 platform.
+    # The first re-install of pip updates pip module but builds pip.exe using
+    # the old pip which still generates x64 executable. But the second
+    # re-install uses updated pip and correctly builds pip.exe for the target.
+    # This can be removed once ARM64 Pythons (currently 3.9 and 3.10) bundle
+    # pip versions newer than 21.3.

add support for building wheels for windows on arm64 #920

add support for building wheels for windows on arm64 #920

Conversation

niyas-sait commented Nov 15, 2021

henryiii left a comment

Choose a reason for hiding this comment

joerick left a comment

Choose a reason for hiding this comment

joerick Nov 17, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niyas-sait commented Nov 17, 2021

niyas-sait commented Nov 17, 2021

niyas-sait commented Nov 18, 2021

henryiii commented Nov 18, 2021

niyas-sait commented Nov 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii left a comment

Choose a reason for hiding this comment

niyas-sait commented Nov 22, 2021

joerick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joerick left a comment

Choose a reason for hiding this comment

henryiii left a comment

Choose a reason for hiding this comment

joerick Nov 17, 2021 •

edited