Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for building wheels for windows on arm64 #920

Merged
merged 14 commits into from Nov 24, 2021
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 8 additions & 7 deletions README.md
Expand Up @@ -22,16 +22,17 @@ Python wheels are great. Building them across **Mac, Linux, Windows**, on **mult
What does it do?
----------------

| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | manylinux<br/>musllinux x86_64 | manylinux<br/>musllinux i686 | manylinux<br/>musllinux aarch64 | manylinux<br/>musllinux ppc64le | manylinux<br/>musllinux s390x |
| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | Windows Arm64 | manylinux<br/>musllinux x86_64 | manylinux<br/>musllinux i686 | manylinux<br/>musllinux aarch64 | manylinux<br/>musllinux ppc64le | manylinux<br/>musllinux s390x |
|---------------|----|-----|-----|-----|----|-----|----|-----|-----|
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | | ✅ | ✅ | ✅ | ✅ |
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | | ✅ | ✅ | ✅ | ✅ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A |
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | N/A | ✅ |✅ | ✅ | ✅ | ✅ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ |
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this table markdown isn't parsing any more. Do the first two Pythons have too many columns?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line in-between is wrong.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

|---------------|----|-----|-----|-----|----|-----|----|-----|-----|

->

|---------------|----|-----|-----|-----|-----|----|-----|----|-----|-----|

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried in a markdown editor and it didn't spot that. Sorry I will fix that.


<sup>¹ PyPy is only supported for manylinux wheels.</sup><br>
<sup>² Windows arm64 support is experimental.</sup><br>

- Builds manylinux, musllinux, macOS 10.9+, and Windows wheels for CPython and PyPy
- Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI
Expand Down
10 changes: 7 additions & 3 deletions bin/update_pythons.py
Expand Up @@ -28,7 +28,7 @@
RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources"


ArchStr = Literal["32", "64"]
ArchStr = Literal["32", "64", "ARM64"]


class ConfigWinCP(TypedDict):
Expand Down Expand Up @@ -68,8 +68,8 @@ def __init__(self, arch_str: ArchStr) -> None:
if resource["@type"] == "PackageBaseAddress/3.0.0":
endpoint = resource["@id"]

ARCH_DICT = {"32": "win32", "64": "win_amd64"}
PACKAGE_DICT = {"32": "pythonx86", "64": "python"}
ARCH_DICT = {"32": "win32", "64": "win_amd64", "ARM64": "win_arm64"}
PACKAGE_DICT = {"32": "pythonx86", "64": "python", "ARM64": "pythonarm64"}

self.arch_str = arch_str
self.arch = ARCH_DICT[arch_str]
Expand Down Expand Up @@ -237,6 +237,7 @@ class AllVersions:
def __init__(self) -> None:
self.windows_32 = WindowsVersions("32")
self.windows_64 = WindowsVersions("64")
self.windows_arm64 = WindowsVersions("ARM64")
self.windows_pypy_64 = PyPyVersions("64")

self.macos_cpython = CPythonVersions()
Expand Down Expand Up @@ -264,6 +265,9 @@ def update_config(self, config: dict[str, str]) -> None:
config_update = self.windows_64.update_version_windows(spec)
elif identifier.startswith("pp"):
config_update = self.windows_pypy_64.update_version_windows(spec)
elif "win_arm64" in identifier:
if identifier.startswith("cp"):
config_update = self.windows_arm64.update_version_windows(spec)

assert config_update is not None, f"{identifier} not found!"
config.update(**config_update)
Expand Down
3 changes: 2 additions & 1 deletion cibuildwheel/architecture.py
Expand Up @@ -29,6 +29,7 @@ class Architecture(Enum):
# windows archs
x86 = "x86"
AMD64 = "AMD64"
ARM64 = "ARM64"

# Allow this to be sorted
def __lt__(self, other: "Architecture") -> bool:
Expand Down Expand Up @@ -83,7 +84,7 @@ def all_archs(platform: PlatformName) -> "Set[Architecture]":
elif platform == "macos":
return {Architecture.x86_64, Architecture.arm64, Architecture.universal2}
elif platform == "windows":
return {Architecture.x86, Architecture.AMD64}
return {Architecture.x86, Architecture.AMD64, Architecture.ARM64}
else:
assert_never(platform)

Expand Down
1 change: 1 addition & 0 deletions cibuildwheel/logger.py
Expand Up @@ -27,6 +27,7 @@
"musllinux_s390x": "manylinux s390x",
"win32": "Windows 32bit",
"win_amd64": "Windows 64bit",
"win_arm64": "Windows on ARM 64bit",
"macosx_x86_64": "macOS x86_64",
"macosx_universal2": "macOS Universal 2 - x86_64 and arm64",
"macosx_arm64": "macOS arm64 - Apple Silicon",
Expand Down
2 changes: 2 additions & 0 deletions cibuildwheel/resources/build-platforms.toml
Expand Up @@ -87,6 +87,8 @@ python_configurations = [
{ identifier = "cp39-win_amd64", version = "3.9.8", arch = "64" },
{ identifier = "cp310-win32", version = "3.10.0", arch = "32" },
{ identifier = "cp310-win_amd64", version = "3.10.0", arch = "64" },
{ identifier = "cp310-win_arm64", version = "3.9.9", arch = "ARM64" },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This identifier appears to be wrong? should be cp39... i think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we have two, possibly, then? 3.9.9 and 3.10.0? Not sure what's actually available.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This identifier appears to be wrong? should be cp39... i think.

Ah, copy-paste error. I will fix it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we have two, possibly, then? 3.9.9 and 3.10.0? Not sure what's actually available.

Yes, both are available. I thought I had it in my patch. Will fix thtat

{ identifier = "cp310-win_arm64", version = "3.10.0", arch = "ARM64" },
{ identifier = "pp37-win_amd64", version = "3.7", arch = "64", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.7-win64.zip" },
{ identifier = "pp38-win_amd64", version = "3.8", arch = "64", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.7-win64.zip" },
]
30 changes: 23 additions & 7 deletions cibuildwheel/windows.py
Expand Up @@ -7,6 +7,8 @@
from typing import Dict, List, NamedTuple, Optional, Sequence, Set
from zipfile import ZipFile

from packaging.version import Version

from .architecture import Architecture
from .environment import ParsedEnvironment
from .logger import log
Expand Down Expand Up @@ -43,9 +45,8 @@ def shell(


def get_nuget_args(version: str, arch: str) -> List[str]:
python_name = "python"
if arch == "32":
python_name += "x86"
platform_suffix = {"32": "x86", "64": "", "ARM64": "arm64"}
python_name = "python" + platform_suffix[arch]
return [
python_name,
"-Version",
Expand Down Expand Up @@ -73,10 +74,7 @@ def get_python_configurations(

python_configurations = [PythonConfiguration(**item) for item in full_python_configs]

map_arch = {
"32": Architecture.x86,
"64": Architecture.AMD64,
}
map_arch = {"32": Architecture.x86, "64": Architecture.AMD64, "ARM64": Architecture.ARM64}
joerick marked this conversation as resolved.
Show resolved Hide resolved

# skip builds as required
python_configurations = [
Expand Down Expand Up @@ -189,10 +187,28 @@ def setup_python(
# Install pip

requires_reinstall = not (installation_path / "Scripts" / "pip.exe").exists()

if requires_reinstall:
# maybe pip isn't installed at all. ensurepip resolves that.
call(["python", "-m", "ensurepip"], env=env, cwd=CIBW_INSTALL_PATH)

# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# pip bundled with python 3.10 for arm64 lacks windows launcher and requires an extra pip force-reinstall to get pip executable
# pip older than 21.3 builds executables such as pip.exe for x64 platform.
# The first re-install of pip updates pip module but builds pip.exe using
# the old pip which still generates x64 executable. But the second
# re-install uses updated pip and correctly builds pip.exe for the target.
# This can be removed once ARM64 Pythons (currently 3.9 and 3.10) bundle
# pip versions newer than 21.3.

if python_configuration.arch == "ARM64" and Version(get_pip_version(env)) < Version("21.3"):
call(
[
"python",
"-m",
"pip",
"install",
"--force-reinstall",
"--upgrade",
"pip",
*dependency_constraint_flags,
],
env=env,
cwd=CIBW_INSTALL_PATH,
)

Copy link
Contributor

@joerick joerick Nov 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be overcomplicated. If the pip executable is missing, we should already detect that. Could you try looking for whatever you're missing, like-

if python_configuration.arch == "ARM64" and not (installation_path / "Scripts" / "piparm64.exe").exists():
    requires_reinstall = True

or if that doesn't work, just this would be okay for now I think-

if python_configuration.arch == "ARM64":
    requires_reinstall = True

probably better to move this up to after the requires_reinstall set line too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's exactly what I tried at first but unfortunately didn't work.

The issue is that pip bundled with python 3.10 builds executables such as pip.exe for x64 platform. The first re-install of pip updates pip module but builds pip.exe using the old pip which still generates x64 executable. But the second re-install uses updated pip and correctly builds pip.exe for the target.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, hmm, okay. I think I understand. What is the Version(get_pip_version(env)) < Version("21.3") about? And are you assuming that requires_reinstall is always true on arm64?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd expect this is a temporary problem and should be fixed once a newer pip is bundled? That's a guess, though, and it would be nice to verify this really was fixed in 21.3 and is not just a problem with nuget.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the fix is available from pip release 21.3 so we don't have to do the extra install.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I've suggested adding this explanation as a comment, which will help let us know when it can be removed.

# upgrade pip to the version matching our constraints
# if necessary, reinstall it to ensure that it's available on PATH as 'pip.exe'
call(
Expand Down
6 changes: 4 additions & 2 deletions docs/options.md
Expand Up @@ -206,8 +206,8 @@ When setting the options, you can use shell-style globbing syntax, as per [fnmat
| Python 3.6 | cp36-macosx_x86_64 | cp36-win_amd64<br/>cp36-win32 | cp36-manylinux_x86_64<br/>cp36-manylinux_i686<br/>cp36-musllinux_x86_64<br/>cp36-musllinux_i686 | cp36-manylinux_aarch64<br/>cp36-manylinux_ppc64le<br/>cp36-manylinux_s390x<br/>cp36-musllinux_aarch64<br/>cp36-musllinux_ppc64le<br/>cp36-musllinux_s390x |
| Python 3.7 | cp37-macosx_x86_64 | cp37-win_amd64<br/>cp37-win32 | cp37-manylinux_x86_64<br/>cp37-manylinux_i686<br/>cp37-musllinux_x86_64<br/>cp37-musllinux_i686 | cp37-manylinux_aarch64<br/>cp37-manylinux_ppc64le<br/>cp37-manylinux_s390x<br/>cp37-musllinux_aarch64<br/>cp37-musllinux_ppc64le<br/>cp37-musllinux_s390x |
| Python 3.8 | cp38-macosx_x86_64<br/>cp38-macosx_universal2<br/>cp38-macosx_arm64 | cp38-win_amd64<br/>cp38-win32 | cp38-manylinux_x86_64<br/>cp38-manylinux_i686<br/>cp38-musllinux_x86_64<br/>cp38-musllinux_i686 | cp38-manylinux_aarch64<br/>cp38-manylinux_ppc64le<br/>cp38-manylinux_s390x<br/>cp38-musllinux_aarch64<br/>cp38-musllinux_ppc64le<br/>cp38-musllinux_s390x |
| Python 3.9 | cp39-macosx_x86_64<br/>cp39-macosx_universal2<br/>cp39-macosx_arm64 | cp39-win_amd64<br/>cp39-win32 | cp39-manylinux_x86_64<br/>cp39-manylinux_i686<br/>cp39-musllinux_x86_64<br/>cp39-musllinux_i686 | cp39-manylinux_aarch64<br/>cp39-manylinux_ppc64le<br/>cp39-manylinux_s390x<br/>cp39-musllinux_aarch64<br/>cp39-musllinux_ppc64le<br/>cp39-musllinux_s390x |
| Python 3.10 | cp310-macosx_x86_64<br/>cp310-macosx_universal2<br/>cp310-macosx_arm64 | cp310-win_amd64<br/>cp310-win32 | cp310-manylinux_x86_64<br/>cp310-manylinux_i686<br/>cp310-musllinux_x86_64<br/>cp310-musllinux_i686 | cp310-manylinux_aarch64<br/>cp310-manylinux_ppc64le<br/>cp310-manylinux_s390x<br/>cp310-musllinux_aarch64<br/>cp310-musllinux_ppc64le<br/>cp310-musllinux_s390x |
| Python 3.9 | cp39-macosx_x86_64<br/>cp39-macosx_universal2<br/>cp39-macosx_arm64 | cp39-win_amd64<br/>cp39-win32<br/>cp39-win_arm64 | cp39-manylinux_x86_64<br/>cp39-manylinux_i686<br/>cp39-musllinux_x86_64<br/>cp39-musllinux_i686 | cp39-manylinux_aarch64<br/>cp39-manylinux_ppc64le<br/>cp39-manylinux_s390x<br/>cp39-musllinux_aarch64<br/>cp39-musllinux_ppc64le<br/>cp39-musllinux_s390x |
| Python 3.10 | cp310-macosx_x86_64<br/>cp310-macosx_universal2<br/>cp310-macosx_arm64 | cp310-win_amd64<br/>cp310-win32<br/>cp310-win_arm64 | cp310-manylinux_x86_64<br/>cp310-manylinux_i686<br/>cp310-musllinux_x86_64<br/>cp310-musllinux_i686 | cp310-manylinux_aarch64<br/>cp310-manylinux_ppc64le<br/>cp310-manylinux_s390x<br/>cp310-musllinux_aarch64<br/>cp310-musllinux_ppc64le<br/>cp310-musllinux_s390x |
| PyPy3.7 v7.3 | pp37-macosx_x86_64 | pp37-win_amd64 | pp37-manylinux_x86_64<br/>pp37-manylinux_i686 | pp37-manylinux_aarch64 |
| PyPy3.8 v7.3 | pp38-macosx_x86_64 | pp38-win_amd64 | pp38-manylinux_x86_64<br/>pp38-manylinux_i686 | pp38-manylinux_aarch64 |

Expand All @@ -216,6 +216,8 @@ The format is `python_tag-platform_tag`, with tags similar to those in [PEP 425]

For CPython, the minimally supported macOS version is 10.9; for PyPy 3.7, macOS 10.13 or higher is required.

Windows arm64 platform support is experimental.

See the [cibuildwheel 1 documentation](https://cibuildwheel.readthedocs.io/en/1.x/) for past end of life versions of Python, and PyPy2.7.

#### Examples
Expand Down
6 changes: 5 additions & 1 deletion unit_test/main_tests/main_platform_test.py
Expand Up @@ -178,7 +178,11 @@ def test_archs_platform_all(platform, intercepted_build_args, monkeypatch):
Architecture.s390x,
}
elif platform == "windows":
assert options.globals.architectures == {Architecture.x86, Architecture.AMD64}
assert options.globals.architectures == {
Architecture.x86,
Architecture.AMD64,
Architecture.ARM64,
}
elif platform == "macos":
assert options.globals.architectures == {
Architecture.x86_64,
Expand Down