Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add support for building wheels for windows on arm64 #920

Merged
merged 14 commits into from Nov 24, 2021
Merged

add support for building wheels for windows on arm64 #920

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
325b1c3
add support for building wheels for windows on arm64
niyas-sait Nov 15, 2021
e8898c0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 15, 2021
cfeba93
Fix main_platform_test.py
niyas-sait Nov 15, 2021
76248e2
Merge branch 'enable_win_arm64' of github.com:nsait-linaro/cibuildwhe…
niyas-sait Nov 15, 2021
c91d158
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 15, 2021
9bf6b0a
Remove unrequired explicit nuget source
niyas-sait Nov 15, 2021
454f5d2
Merge branch 'enable_win_arm64' of github.com:nsait-linaro/cibuildwhe…
niyas-sait Nov 15, 2021
dfae980
add python arm64 3.9.9 version
niyas-sait Nov 17, 2021
aaf23c9
extend update_pythons.py to add win/arm64 support
niyas-sait Nov 17, 2021
dffac66
update documentation for new win/arm64 platform
niyas-sait Nov 17, 2021
4cc5fb0
style: include ARM64 in literal
henryiii Nov 17, 2021
19e11ff
Fix python 3.9 identifier for win_arm64
niyas-sait Nov 22, 2021
ff6ac4f
Merge branch 'enable_win_arm64' of github.com:nsait-linaro/cibuildwhe…
niyas-sait Nov 22, 2021
05becce
Fix table layout issues in README and add note about arm64 pip issue
niyas-sait Nov 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion cibuildwheel/architecture.py
View file
Open in desktop
Expand Up @@ -29,6 +29,7 @@ class Architecture(Enum):
# windows archs
x86 = "x86"
AMD64 = "AMD64"
ARM64 = "ARM64"

# Allow this to be sorted
def __lt__(self, other: "Architecture") -> bool:
Expand Down Expand Up @@ -83,7 +84,7 @@ def all_archs(platform: PlatformName) -> "Set[Architecture]":
elif platform == "macos":
return {Architecture.x86_64, Architecture.arm64, Architecture.universal2}
elif platform == "windows":
return {Architecture.x86, Architecture.AMD64}
return {Architecture.x86, Architecture.AMD64, Architecture.ARM64}
else:
assert_never(platform)

Expand Down
1 change: 1 addition & 0 deletions cibuildwheel/logger.py
View file
Open in desktop
Expand Up @@ -27,6 +27,7 @@
"musllinux_s390x": "manylinux s390x",
"win32": "Windows 32bit",
"win_amd64": "Windows 64bit",
"win_arm64": "Windows on ARM 64bit",
"macosx_x86_64": "macOS x86_64",
"macosx_universal2": "macOS Universal 2 - x86_64 and arm64",
"macosx_arm64": "macOS arm64 - Apple Silicon",
Expand Down
1 change: 1 addition & 0 deletions cibuildwheel/resources/build-platforms.toml
View file
Open in desktop
Expand Up @@ -87,6 +87,7 @@ python_configurations = [
{ identifier = "cp39-win_amd64", version = "3.9.8", arch = "64" },
{ identifier = "cp310-win32", version = "3.10.0", arch = "32" },
{ identifier = "cp310-win_amd64", version = "3.10.0", arch = "64" },
{ identifier = "cp310-win_arm64", version = "3.10.0", arch = "ARM64" },
{ identifier = "pp37-win_amd64", version = "3.7", arch = "64", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.7-win64.zip" },
{ identifier = "pp38-win_amd64", version = "3.8", arch = "64", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.7-win64.zip" },
]
30 changes: 23 additions & 7 deletions cibuildwheel/windows.py
View file
Open in desktop
Expand Up @@ -7,6 +7,8 @@
from typing import Dict, List, NamedTuple, Optional, Sequence, Set
from zipfile import ZipFile

from packaging.version import Version

from .architecture import Architecture
from .environment import ParsedEnvironment
from .logger import log
Expand Down Expand Up @@ -43,9 +45,8 @@ def shell(


def get_nuget_args(version: str, arch: str) -> List[str]:
python_name = "python"
if arch == "32":
python_name += "x86"
platform_suffix = {"32": "x86", "64": "", "ARM64": "arm64"}
python_name = "python" + platform_suffix[arch]
return [
python_name,
"-Version",
Expand Down Expand Up @@ -73,10 +74,7 @@ def get_python_configurations(

python_configurations = [PythonConfiguration(**item) for item in full_python_configs]

map_arch = {
"32": Architecture.x86,
"64": Architecture.AMD64,
}
map_arch = {"32": Architecture.x86, "64": Architecture.AMD64, "ARM64": Architecture.ARM64}
joerick marked this conversation as resolved.
Show resolved Hide resolved

# skip builds as required
python_configurations = [
Expand Down Expand Up @@ -189,10 +187,28 @@ def setup_python(
# Install pip

requires_reinstall = not (installation_path / "Scripts" / "pip.exe").exists()

if requires_reinstall:
# maybe pip isn't installed at all. ensurepip resolves that.
call(["python", "-m", "ensurepip"], env=env, cwd=CIBW_INSTALL_PATH)

# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
# pip older than 21.3 builds executables such as pip.exe for x64 platform.
# The first re-install of pip updates pip module but builds pip.exe using
# the old pip which still generates x64 executable. But the second
# re-install uses updated pip and correctly builds pip.exe for the target.
# This can be removed once ARM64 Pythons (currently 3.9 and 3.10) bundle
# pip versions newer than 21.3.

if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"):
call(
[
"python",
"-m",
"pip",
"install",
"--force-reinstall",
"--upgrade",
"pip",
*dependency_constraint_flags,
],
env=env,
cwd=CIBW_INSTALL_PATH,
)

Copy link
Contributor

@joerick joerick Nov 17, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be overcomplicated. If the pip executable is missing, we should already detect that. Could you try looking for whatever you're missing, like-

if python_configuration.arch == "ARM64" and not (installation_path / "Scripts" / "piparm64.exe").exists():
    requires_reinstall = True

or if that doesn't work, just this would be okay for now I think-

if python_configuration.arch == "ARM64":
    requires_reinstall = True

probably better to move this up to after the requires_reinstall set line too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's exactly what I tried at first but unfortunately didn't work.

The issue is that pip bundled with python 3.10 builds executables such as pip.exe for x64 platform. The first re-install of pip updates pip module but builds pip.exe using the old pip which still generates x64 executable. But the second re-install uses updated pip and correctly builds pip.exe for the target.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, hmm, okay. I think I understand. What is the Version(get_pip_version(env)) < Version("21.3") about? And are you assuming that requires_reinstall is always true on arm64?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd expect this is a temporary problem and should be fixed once a newer pip is bundled? That's a guess, though, and it would be nice to verify this really was fixed in 21.3 and is not just a problem with nuget.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the fix is available from pip release 21.3 so we don't have to do the extra install.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I've suggested adding this explanation as a comment, which will help let us know when it can be removed.

# upgrade pip to the version matching our constraints
# if necessary, reinstall it to ensure that it's available on PATH as 'pip.exe'
call(
Expand Down
6 changes: 5 additions & 1 deletion unit_test/main_tests/main_platform_test.py
View file
Open in desktop
Expand Up @@ -178,7 +178,11 @@ def test_archs_platform_all(platform, intercepted_build_args, monkeypatch):
Architecture.s390x,
}
elif platform == "windows":
assert options.globals.architectures == {Architecture.x86, Architecture.AMD64}
assert options.globals.architectures == {
Architecture.x86,
Architecture.AMD64,
Architecture.ARM64,
}
elif platform == "macos":
assert options.globals.architectures == {
Architecture.x86_64,
Expand Down