Add support for Cirrus CI #1191

Jackenmen · 2022-07-23T23:49:53Z

Resolves #145 and resolves #1240

For macOS arm64, there was a need for some changes to the test suite:

some of the tests were failing because they tried to build for CPython 3.6 (cp36-*) and on macOS arm64, you can only use CPython 3.8 and above (4bc6bd3, b39dc5b)
due to an issue with wheel 0.36.1 and earlier, generated wheel filenames had a wrong minimum macOS version (3c246c7)
due to Fix cp38-macosx universal2 and arm64 Python pkg URL #1169, cibuildwheel generated cp38-macosx_arm64 wheels without testing them which caused it to not delete the wheel when the test expected it to (2b35f3c)

Link to a Cirrus CI test suite build: https://cirrus-ci.com/build/6463822541094912
Link to a Cirrus CI minimal example build: https://cirrus-ci.com/build/6073962235953152
Link to a Cirrus CI macOS Intel example build: https://cirrus-ci.com/build/6736767930859520

joerick

Exciting! I'd love to have native macos-arm64 CI, and another native linux-aarch64 is no bad thing either!

I've tried to add the service to this repo, but it seems that it requires a pypa admin. @pradyunsg would you mind adding the free tier of CirrusCI to this repo? https://github.com/marketplace/cirrus-ci

.cirrus.yml

test/test_dependency_versions.py

test/test_testing.py

unit_test/environment_test.py

test/test_subdir_package.py

henryiii · 2022-08-11T02:53:20Z

Is this ready to review/go? Might be nice to get it in for the upcoming release?

Jackenmen · 2022-08-11T09:31:32Z

Is this ready to review/go? Might be nice to get it in for the upcoming release?

Sadly as I mentioned in the Discord server, my account on Cirrus CI got suspended, most likely by an automated system, so I can't really do anything until their support responds. I emailed them about this 2 days ago, but no response yet. It's possible that if Pradyun added the Cirrus CI app to the PyPA org, it would be possible to see the build output but it's also equally if not more likely that it just wouldn't run the build when it sees me as the one triggering it.

joerick · 2022-08-16T21:36:29Z

@pradyunsg has added cirrus ci to this repo, so let's try again... perhaps try pushing an empty commit and see if cirrus picks it up?

Jackenmen · 2022-08-16T21:42:17Z

My account has actually been unsuspended earlier today after I re-asked on their GitHub rather than their support email so their CI should just work now for me again. I'm going to merge main into this branch first and I'll update this PR to not make changes that are no longer necessary.

Note that Linux arm64 build still doesn't work due to cirruslabs/cirrus-ci-docs#1034, I think it might make sense to leave that work for a future PR.

pypa/wheel - issue 387 and PR 390

Co-authored-by: Joe Rickerby <joerick@mac.com>

Jackenmen · 2022-08-17T16:30:36Z

Alright, all review comments should be addressed and the tests pass. I dropped Linux arm64 support - I don't think it's worth blocking over this and once this PR is merged, a new issue should be created for that with explanation what is a potential blocker for it.

I updated the PR description appropriately and included links to successful builds for example files. This PR should now be ready for review.

mayeut · 2022-08-21T11:35:37Z

Note that Linux arm64 build still doesn't work due to cirruslabs/cirrus-ci-docs#1034, I think it might make sense to leave that work for a future PR.

Linux arm64 might also be solved by cirruslabs/cirrus-ci-docs#1027, whichever comes first. It makes to leave that work for a future PR.

Jackenmen · 2022-08-21T17:12:58Z

I was hoping to re-trigger CircleCI check due to https://status.circleci.com/incidents/3d31fnpthj7y but looks like that didn't work... Gonna try an empty commit.

Jackenmen · 2022-08-21T17:15:27Z

Well, that definitely made the failing check disappear, I guess that's solved then 🤔

joerick

This all looks good to me!

Jackenmen · 2022-08-21T21:17:01Z

I went ahead and made #1240 to track Linux arm64 support on Cirrus CI.

henryiii

I guess the intel Mac example is separate from the rest of the example because the arm64 one cross-compiles Intel wheels? At least for now, I'd guess the ideal workflow for most projects on Cirrus would be to have native wheels & run both runners - cross compiling is harder than native compiling. Eventually, as Intel runners become hard to find, then this workflow would be better.

Just a comment, looks great!

Jackenmen · 2022-08-22T13:16:19Z

I guess the intel Mac example is separate from the rest of the example because the arm64 one cross-compiles Intel wheels?

The cirrus-ci-minimal example file does not do any cross-compilation since the default arch on Apple Silicon is only arm64 (which was done in cibuildwheel 2.9.0).

I'd guess the ideal workflow for most projects on Cirrus would be to have native wheels & run both runners - cross compiling is harder than native compiling.

Cirrus CI does not have an Intel runner. And in case of Apple Silicon, it's not really cross-compilation process per se since it uses Rosetta2 emulation and so it doesn't involve a typical cross-compilation toolchain.

Jackenmen · 2022-08-23T15:45:14Z

README.md


 <sup>¹ [Requires emulation](https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation), distributed separately. Other services may also support Linux ARM through emulation or third-party build hosts, but these are not tested in our CI.</sup><br>
+<sup>² [Uses cross-compilation](https://cibuildwheel.readthedocs.io/en/stable/faq/#universal2). It is not possible to test `arm64` and the `arm64` part of a `universal2` wheel on this CI platform.</sup><br>
+<sup>³ [Uses cross-compilation](https://cibuildwheel.readthedocs.io/en/stable/faq/#universal2). Thanks to Rosetta 2 emulation, it is possible to test `x86_64` and both parts of a `universal2` wheel on this CI platform.</sup><br>


As I said in my previous comment, I'm not sure cross-compilation is the right term here, could anyone confirm whether "emulation" makes more sense here?

Suggested change

³ [Uses cross-compilation](https://cibuildwheel.readthedocs.io/en/stable/faq/#universal2). Thanks to Rosetta 2 emulation, it is possible to test `x86_64` and both parts of a `universal2` wheel on this CI platform. 

³ [Uses Rosetta 2 emulation](https://cibuildwheel.readthedocs.io/en/stable/faq/#universal2). Thanks to the emulation, it is possible to test `x86_64` and both parts of a `universal2` wheel on this CI platform.

Well, technically it's both! Cross-compilation is right, the wheel is built using cross-compilation, but we can still test the wheel using Rosetta 2 emulation.

I think the existing copy is fine as-is.

Jackenmen · 2022-08-23T15:46:59Z

Linux arm64 might also be solved by cirruslabs/cirrus-ci-docs#1027, whichever comes first. It makes to leave that work for a future PR.

Good news, the linked issue got resolved and I've already made a configuration that so far appears to just work, I'm waiting for the tests on my fork to finish before pushing it here.

Jackenmen · 2022-08-23T17:07:07Z

All tests ran in Cirrus CI succeeded, I've also updated the PR description with the new link to a build from examples/cirrus-ci-minimal.yml which also works fine.

It seems that one of the GitHub workflows failed due to an intermittent pip/PyPI issue but would have worked otherwise. I can't re-run it myself.

joerick

Great! The Cirrus aarch64 builder is is 1.5x faster than the Travis one, and 5x faster than emulation on GHA!

examples/cirrus-ci-minimal.yml

.cirrus.yml

Co-authored-by: Joe Rickerby <joerick@mac.com>

joerick · 2022-08-30T21:09:27Z

PR merged. 🎉

Stellar work on this @jack1142! Thank you.

Jackenmen changed the title ~~Skip Python 3.6 test on macOS arm64~~ Add support for Cirrus CI Jul 24, 2022

joerick reviewed Jul 24, 2022

View reviewed changes

.cirrus.yml Outdated Show resolved Hide resolved

test/test_dependency_versions.py Show resolved Hide resolved

test/test_testing.py Outdated Show resolved Hide resolved

unit_test/environment_test.py Outdated Show resolved Hide resolved

This was referenced Jul 24, 2022

When testing x86_64 on macOS arm64, CIBW_TEST_COMMAND isn't ran under Rosetta when using pipes #1193

Closed

Remove dependency on Unix tools in Windows test suite #1196

Merged

mayeut reviewed Jul 31, 2022

View reviewed changes

test/test_subdir_package.py Outdated Show resolved Hide resolved

mayeut reviewed Jul 31, 2022

View reviewed changes

test/test_subdir_package.py Outdated Show resolved Hide resolved

joerick mentioned this pull request Jul 31, 2022

Should arm64 runners build both universal2 and arm64 wheels by default? #1204

Closed

mayeut mentioned this pull request Aug 6, 2022

fix: don't build universal2 as a default on macOS arm64 runners #1217

Merged

Jackenmen and others added 14 commits August 17, 2022 15:12

Skip Python 3.6 test on macOS arm64

4bc6bd3

Bump wheel version to fix deployment target for universal2

2bf4c14

pypa/wheel - issue 387 and PR 390

Test doesn't depend on python version but macOS arm64 can't build 3.6

b39dc5b

Skip cp38-macosx_arm64 in the test testing failing tests

2b35f3c

Add .cirrus.yml running tests

623b39e

Add Cirrus CI to CIProvider enum

6707139

Add a minimal example

2812f8b

List Cirrus CI in README and update relevant scripts

ce8e3fe

Update CI.md

ffeccfe

Add more leeway to the 55 minutes task

dce8c22

Co-authored-by: Joe Rickerby <joerick@mac.com>

Elaborate on why arm64 platform is skipped for failing test test

8d71eca

Co-authored-by: Joe Rickerby <joerick@mac.com>

Use CPython 3.10 instead of 3.8 that was used instead of 3.6

ccd29d8

Show how to build for Intel macOS on Cirrus CI

c50d72c

Remove Linux arm64 build

bd512e9

Jackenmen marked this pull request as ready for review August 17, 2022 16:30

Jackenmen closed this Aug 21, 2022

Jackenmen reopened this Aug 21, 2022

Re-trigger Circle CI

2723bba

joerick approved these changes Aug 21, 2022

View reviewed changes

Jackenmen mentioned this pull request Aug 21, 2022

Add support for native Linux arm64 builds on Cirrus CI #1240

Closed

Czaki approved these changes Aug 22, 2022

View reviewed changes

Czaki mentioned this pull request Aug 22, 2022

CircleCI + native aarch64 linux buils work, how to document this? #1241

Closed

henryiii approved these changes Aug 22, 2022

View reviewed changes

Jackenmen commented Aug 23, 2022

View reviewed changes

Add Linux arm64 back

f0e9518

joerick reviewed Aug 23, 2022

View reviewed changes

examples/cirrus-ci-minimal.yml Outdated Show resolved Hide resolved

.cirrus.yml Outdated Show resolved Hide resolved

Jackenmen and others added 2 commits August 23, 2022 19:39

Apply suggestions from code review

d22703a

Co-authored-by: Joe Rickerby <joerick@mac.com>

Merge branch 'main' into add_cirrus_ci_support

93986db

henryiii approved these changes Aug 24, 2022

View reviewed changes

mayeut approved these changes Aug 27, 2022

View reviewed changes

joerick merged commit 0117165 into pypa:main Aug 30, 2022

Jackenmen deleted the add_cirrus_ci_support branch August 30, 2022 21:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Cirrus CI #1191

Add support for Cirrus CI #1191

Jackenmen commented Jul 23, 2022 •

edited

joerick left a comment

henryiii commented Aug 11, 2022

Jackenmen commented Aug 11, 2022

joerick commented Aug 16, 2022

Jackenmen commented Aug 16, 2022

Jackenmen commented Aug 17, 2022 •

edited

mayeut commented Aug 21, 2022

Jackenmen commented Aug 21, 2022

Jackenmen commented Aug 21, 2022

joerick left a comment

Jackenmen commented Aug 21, 2022

henryiii left a comment

Jackenmen commented Aug 22, 2022

Jackenmen Aug 23, 2022

joerick Aug 23, 2022

Jackenmen commented Aug 23, 2022

Jackenmen commented Aug 23, 2022

joerick left a comment

joerick commented Aug 30, 2022

	<sup>³ [Uses cross-compilation](https://cibuildwheel.readthedocs.io/en/stable/faq/#universal2). Thanks to Rosetta 2 emulation, it is possible to test `x86_64` and both parts of a `universal2` wheel on this CI platform.</sup><br>
	<sup>³ [Uses Rosetta 2 emulation](https://cibuildwheel.readthedocs.io/en/stable/faq/#universal2). Thanks to the emulation, it is possible to test `x86_64` and both parts of a `universal2` wheel on this CI platform.</sup><br>

Add support for Cirrus CI #1191

Add support for Cirrus CI #1191

Conversation

Jackenmen commented Jul 23, 2022 • edited

joerick left a comment

Choose a reason for hiding this comment

henryiii commented Aug 11, 2022

Jackenmen commented Aug 11, 2022

joerick commented Aug 16, 2022

Jackenmen commented Aug 16, 2022

Jackenmen commented Aug 17, 2022 • edited

mayeut commented Aug 21, 2022

Jackenmen commented Aug 21, 2022

Jackenmen commented Aug 21, 2022

joerick left a comment

Choose a reason for hiding this comment

Jackenmen commented Aug 21, 2022

henryiii left a comment

Choose a reason for hiding this comment

Jackenmen commented Aug 22, 2022

Jackenmen Aug 23, 2022

Choose a reason for hiding this comment

joerick Aug 23, 2022

Choose a reason for hiding this comment

Jackenmen commented Aug 23, 2022

Jackenmen commented Aug 23, 2022

joerick left a comment

Choose a reason for hiding this comment

joerick commented Aug 30, 2022

Jackenmen commented Jul 23, 2022 •

edited

Jackenmen commented Aug 17, 2022 •

edited