Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add Pyodide support #1456

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
+843 −70
Open

feat: add Pyodide support #1456

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
d562681
feat: add Pyodide support
henryiii May 18, 2024
653d766
Try to fix xbuildenv path
hoodmane May 19, 2024
02374e6
fix: remove pinning on pyodide
henryiii May 20, 2024
a9a7f2d
Update for Pyodide 0.26.0a5
hoodmane May 21, 2024
2c77714
Install pyodide-build from pypi
hoodmane May 21, 2024
fc72b26
Update docs/options.md
hoodmane May 24, 2024
4c1180f
Unxfail things that look like they were just a version mismatch
hoodmane May 24, 2024
553636d
refactor: add constraints for pyodide
henryiii May 24, 2024
32d335a
chore: minor cleanup
henryiii May 24, 2024
f0261ef
Apply suggestions from code review
henryiii May 24, 2024
3438192
Apply suggestion from code review
mayeut May 25, 2024
8bee9cb
Apply suggestion from code review
mayeut May 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/test.yml
View file
Open in desktop
Expand Up @@ -148,3 +148,37 @@ jobs:
- name: Run the emulation tests
run: |
pytest --run-emulation test/test_emulation.py
test-pyodide:
name: Test cibuildwheel building pyodide wheels
needs: lint
runs-on: ubuntu-24.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to tests other platforms (mostly to ensure some local builds could work) ?
Maybe just a reduced set on other platforms than linux.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building on windows absolutely does not work and will not work until Python gets proper cross builds. Building on macs generally sort of works but we don't test it in any way upstream and cannot test it because mac runners are too expensive. So I think we should just mark it as unsupported. Mac builds would be possible to support if someone decided to fund CI resources for it. But it's likely that nobody will ever care enough.

Or were you thinking that we'd use a docker image or something on these platforms? It's not worth it I don't think. We could check for win32 and make a hard error saying it won't work, and check for Darwin and warn saying we don't actually test this configuration and it is known to be buggy but it sometimes works.

timeout-minutes: 180
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
name: Install Python 3.12
with:
python-version: '3.12'

henryiii marked this conversation as resolved.
Show resolved Hide resolved
- name: Install dependencies
run: |
python -m pip install ".[test]"
- name: Generate a sample project
run: |
python -m test.test_projects test.test_0_basic.basic_project sample_proj
- name: Run a sample build (GitHub Action)
uses: ./
with:
package-dir: sample_proj
output-dir: wheelhouse
env:
CIBW_PLATFORM: pyodide

- name: Run tests with 'CIBW_PLATFORM' set to 'pyodide'
run: |
python ./bin/run_tests.py
env:
CIBW_PLATFORM: pyodide
31 changes: 16 additions & 15 deletions README.md
View file
Open in desktop
Expand Up @@ -22,24 +22,25 @@ Python wheels are great. Building them across **Mac, Linux, Windows**, on **mult
What does it do?
----------------

| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | Windows Arm64 | manylinux<br/>musllinux x86_64 | manylinux<br/>musllinux i686 | manylinux<br/>musllinux aarch64 | manylinux<br/>musllinux ppc64le | manylinux<br/>musllinux s390x |
|----------------|----|-----|-----|-----|-----|----|-----|----|-----|-----|
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.11 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.12 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.13³ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ |
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A |
| PyPy 3.8 v7.3 | ✅ | ✅ | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A |
| PyPy 3.9 v7.3 | ✅ | ✅ | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A |
| PyPy 3.10 v7.3 | ✅ | ✅ | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A |
| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | Windows Arm64 | manylinux<br/>musllinux x86_64 | manylinux<br/>musllinux i686 | manylinux<br/>musllinux aarch64 | manylinux<br/>musllinux ppc64le | manylinux<br/>musllinux s390x | Pyodide |
|----------------|----|-----|-----|-----|-----|----|-----|----|-----|-----|-----|
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
| CPython 3.11 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
| CPython 3.12 | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ | ✅⁴ |
| CPython 3.13³ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅ | ✅ | ✅ | ✅ | ✅ | N/A |
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A | N/A |
| PyPy 3.8 v7.3 | ✅ | ✅ | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A | N/A |
| PyPy 3.9 v7.3 | ✅ | ✅ | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A | N/A |
| PyPy 3.10 v7.3 | ✅ | ✅ | ✅ | N/A | N/A | ✅¹ | ✅¹ | ✅¹ | N/A | N/A | N/A |

<sup>¹ PyPy is only supported for manylinux wheels.</sup><br>
<sup>² Windows arm64 support is experimental.</sup><br>
<sup>³ CPython 3.13 is available using the [CIBW_PRERELEASE_PYTHONS](https://cibuildwheel.pypa.io/en/stable/options/#prerelease-pythons) option.</sup><br>
<sup>³ CPython 3.13 is available using the [`CIBW_PRERELEASE_PYTHONS`](https://cibuildwheel.pypa.io/en/stable/options/#prerelease-pythons) option. Free-threaded mode requires opt-in, not yet available on macOS.</sup><br>
<sup>⁴ Experimental, not yet supported on PyPI, but can be used directly in web deployment. Use `--platform pyodide` to build.</sup><br>

- Builds manylinux, musllinux, macOS 10.9+, and Windows wheels for CPython and PyPy
- Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, GitLab CI, and Cirrus CI
Expand Down
1 change: 1 addition & 0 deletions bin/generate_schema.py
View file
Open in desktop
Expand Up @@ -280,6 +280,7 @@ def as_object(d: dict[str, Any]) -> dict[str, Any]:
"linux": as_object(non_global_options),
"windows": as_object(not_linux),
"macos": as_object(not_linux),
"pyodide": as_object(not_linux),
}

oses["linux"]["properties"]["repair-wheel-command"] = {
Expand Down
7 changes: 6 additions & 1 deletion cibuildwheel/__main__.py
View file
Open in desktop
Expand Up @@ -15,6 +15,7 @@
import cibuildwheel
import cibuildwheel.linux
import cibuildwheel.macos
import cibuildwheel.pyodide
import cibuildwheel.util
import cibuildwheel.windows
from cibuildwheel._compat.typing import assert_never
Expand Down Expand Up @@ -45,7 +46,7 @@ def main() -> None:

parser.add_argument(
"--platform",
choices=["auto", "linux", "macos", "windows"],
choices=["auto", "linux", "macos", "windows", "pyodide"],
hoodmane marked this conversation as resolved.
Show resolved Hide resolved
default=None,
help="""
Platform to build for. Use this option to override the
Expand Down Expand Up @@ -176,6 +177,8 @@ def _compute_platform_only(only: str) -> PlatformName:
return "macos"
if "win_" in only or "win32" in only:
return "windows"
if "pyodide_" in only:
return "pyodide"
print(
f"Invalid --only='{only}', must be a build selector with a known platform",
file=sys.stderr,
Expand Down Expand Up @@ -246,6 +249,8 @@ def get_platform_module(platform: PlatformName) -> PlatformModule:
return cibuildwheel.windows
if platform == "macos":
return cibuildwheel.macos
if platform == "pyodide":
return cibuildwheel.pyodide
assert_never(platform)


Expand Down
8 changes: 8 additions & 0 deletions cibuildwheel/architecture.py
View file
Open in desktop
Expand Up @@ -15,6 +15,7 @@
"linux": "Linux",
"macos": "macOS",
"windows": "Windows",
"pyodide": "Pyodide",
}

ARCH_SYNONYMS: Final[list[dict[PlatformName, str | None]]] = [
Expand Down Expand Up @@ -46,6 +47,9 @@ class Architecture(Enum):
AMD64 = "AMD64"
ARM64 = "ARM64"

# WebAssembly
wasm32 = "wasm32"

# Allow this to be sorted
def __lt__(self, other: Architecture) -> bool:
return self.value < other.value
Expand Down Expand Up @@ -75,6 +79,9 @@ def parse_config(config: str, platform: PlatformName) -> set[Architecture]:
def auto_archs(platform: PlatformName) -> set[Architecture]:
native_machine = platform_module.machine()

if platform == "pyodide":
return {Architecture.wasm32}

# Cross-platform support. Used for --print-build-identifiers or docker builds.
host_platform: PlatformName = (
"windows"
Expand Down Expand Up @@ -120,6 +127,7 @@ def all_archs(platform: PlatformName) -> set[Architecture]:
},
"macos": {Architecture.x86_64, Architecture.arm64, Architecture.universal2},
"windows": {Architecture.x86, Architecture.AMD64, Architecture.ARM64},
"pyodide": {Architecture.wasm32},
}
return all_archs_map[platform]

Expand Down
1 change: 1 addition & 0 deletions cibuildwheel/logger.py
View file
Open in desktop
Expand Up @@ -34,6 +34,7 @@
"macosx_x86_64": "macOS x86_64",
"macosx_universal2": "macOS Universal 2 - x86_64 and arm64",
"macosx_arm64": "macOS arm64 - Apple Silicon",
"pyodide_wasm32": "Pyodide",
}


Expand Down