New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add support for building wheels for windows on arm64 #920
Conversation
for more information, see https://pre-commit.ci
…el into enable_win_arm64
for more information, see https://pre-commit.ci
…el into enable_win_arm64
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems reasonable enough to me. @joerick?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool! Have you tested this on an arm64 machine @nsait-linaro? It looks good so far, just wanted to check. I can't test it locally, and we don't have a CI runner for this so we'd have to take your word for it for now!
Other things we'd need to merge this-
- the build_platforms.toml file is actually generated/modified weekly by
bin/update_pythons.py
, so to make this change stick, we'd need to make sure that script still works and retains arm64. Could you try that and make sure it does the right thing? - Docs updates. Off the top of my head, we'd need to update the build id table in the CIBW_BUILD section, and the table in the README. For now, in the README, could we note that win-arm64 support is 'experimental', since we can't test this in CI yet.
cibuildwheel/windows.py
Outdated
# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable | ||
if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"): | ||
call( | ||
[ | ||
"python", | ||
"-m", | ||
"pip", | ||
"install", | ||
"--force-reinstall", | ||
"--upgrade", | ||
"pip", | ||
*dependency_constraint_flags, | ||
], | ||
env=env, | ||
cwd=CIBW_INSTALL_PATH, | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be overcomplicated. If the pip executable is missing, we should already detect that. Could you try looking for whatever you're missing, like-
if python_configuration.arch == "ARM64" and not (installation_path / "Scripts" / "piparm64.exe").exists():
requires_reinstall = True
or if that doesn't work, just this would be okay for now I think-
if python_configuration.arch == "ARM64":
requires_reinstall = True
probably better to move this up to after the requires_reinstall set line too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's exactly what I tried at first but unfortunately didn't work.
The issue is that pip bundled with python 3.10 builds executables such as pip.exe for x64 platform. The first re-install of pip updates pip module but builds pip.exe using the old pip which still generates x64 executable. But the second re-install uses updated pip and correctly builds pip.exe for the target.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, hmm, okay. I think I understand. What is the Version(get_pip_version(env)) < Version("21.3")
about? And are you assuming that requires_reinstall is always true on arm64?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd expect this is a temporary problem and should be fixed once a newer pip is bundled? That's a guess, though, and it would be nice to verify this really was fixed in 21.3 and is not just a problem with nuget.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the fix is available from pip release 21.3 so we don't have to do the extra install.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. I've suggested adding this explanation as a comment, which will help let us know when it can be removed.
Yes, I've tried it on my surface pro and tried building a few packages like I am working on a patch to get the cross-compilation working but require a setuptools update to help with cross-compilation. |
Failing CI seems like an infrastructure issue. Can someone please rerun the test ? |
Yes, CircleCI has been timing out here in the last two days or so. Not sure when it's going to be fixed. I think the latest main build has the timeout failure too. |
Thanks |
@@ -87,6 +87,8 @@ python_configurations = [ | |||
{ identifier = "cp39-win_amd64", version = "3.9.8", arch = "64" }, | |||
{ identifier = "cp310-win32", version = "3.10.0", arch = "32" }, | |||
{ identifier = "cp310-win_amd64", version = "3.10.0", arch = "64" }, | |||
{ identifier = "cp310-win_arm64", version = "3.9.9", arch = "ARM64" }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This identifier appears to be wrong? should be cp39...
i think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we have two, possibly, then? 3.9.9 and 3.10.0? Not sure what's actually available.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This identifier appears to be wrong? should be
cp39...
i think.
Ah, copy-paste error. I will fix it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we have two, possibly, then? 3.9.9 and 3.10.0? Not sure what's actually available.
Yes, both are available. I thought I had it in my patch. Will fix thtat
cibuildwheel/windows.py
Outdated
# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable | ||
if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"): | ||
call( | ||
[ | ||
"python", | ||
"-m", | ||
"pip", | ||
"install", | ||
"--force-reinstall", | ||
"--upgrade", | ||
"pip", | ||
*dependency_constraint_flags, | ||
], | ||
env=env, | ||
cwd=CIBW_INSTALL_PATH, | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, hmm, okay. I think I understand. What is the Version(get_pip_version(env)) < Version("21.3")
about? And are you assuming that requires_reinstall is always true on arm64?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Python identifier needs to match version. Then I'm happy with it. ;)
This has been fixed in the latest commit. Thanks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a couple tweaks needed I think!
README.md
Outdated
| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | Windows Arm64 | manylinux<br/>musllinux x86_64 | manylinux<br/>musllinux i686 | manylinux<br/>musllinux aarch64 | manylinux<br/>musllinux ppc64le | manylinux<br/>musllinux s390x | | ||
|---------------|----|-----|-----|-----|----|-----|----|-----|-----| | ||
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A | | ||
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | N/A | ✅ |✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this table markdown isn't parsing any more. Do the first two Pythons have too many columns?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The line in-between is wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|---------------|----|-----|-----|-----|----|-----|----|-----|-----|
->
|---------------|----|-----|-----|-----|-----|----|-----|----|-----|-----|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tried in a markdown editor and it didn't spot that. Sorry I will fix that.
cibuildwheel/windows.py
Outdated
if requires_reinstall: | ||
# maybe pip isn't installed at all. ensurepip resolves that. | ||
call(["python", "-m", "ensurepip"], env=env, cwd=CIBW_INSTALL_PATH) | ||
|
||
# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable | |
# pip older than 21.3 builds executables such as pip.exe for x64 platform. | |
# The first re-install of pip updates pip module but builds pip.exe using | |
# the old pip which still generates x64 executable. But the second | |
# re-install uses updated pip and correctly builds pip.exe for the target. | |
# This can be removed once ARM64 Pythons (currently 3.9 and 3.10) bundle | |
# pip versions newer than 21.3. |
cibuildwheel/windows.py
Outdated
# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable | ||
if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"): | ||
call( | ||
[ | ||
"python", | ||
"-m", | ||
"pip", | ||
"install", | ||
"--force-reinstall", | ||
"--upgrade", | ||
"pip", | ||
*dependency_constraint_flags, | ||
], | ||
env=env, | ||
cwd=CIBW_INSTALL_PATH, | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. I've suggested adding this explanation as a comment, which will help let us know when it can be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After this goes in, we should add a third note on the table about musllinux not being supported on Alpine 3.14.
This patch enables building python wheels for windows on arm64 targets.
There is no public CI available yet for win/arm64 targets but expecting Azure support soon. This could help to start releasing binary wheels for python once the CI systems are upgraded.