Fix cp38-macosx universal2 and arm64 Python pkg URL #1169

hombit · 2022-07-07T08:30:33Z

Previous URLs pointed to x86_64 version

Previous URL pointed to x86_64 version

joerick · 2022-07-07T09:20:41Z

Unfortunately we don't want to upgrade the bundled Python because it would increase the minimum MacOS version for the x86_64 arch from 10.9 to 11.0, which is a pretty massive jump.

hombit · 2022-07-07T11:20:03Z

Sorry, could you please explain how arm64 Python URL affects x86_64 minimum macOS version?

Czaki · 2022-07-07T11:37:00Z

cibuildwheel/resources/build-platforms.toml

-  { identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
-  { identifier = "cp38-macosx_universal2", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
+  { identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg" },
+  { identifier = "cp38-macosx_universal2", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg" },


here you update also python to build universal2 wheels. universal2 wheels are wheels that contain code for both x86 and arm. This python version does not allow to build wheels against x86 for macos 10.9.

Should I update arm64 URL only?

I still don't understand, sorry. For Python 3.9 and 3.10 both arm64 and universal2 URLs points to universal2 packages, while all python 3.8 URLs are for the x86_64 package. Why is it different for different Python versions?

I do not play too much with ARM on macOS so let's wait for what @joerick will tell.

joerick · 2022-07-08T08:50:22Z

Ah, I understand what's going on. We still have no arm64 CI, so we've never actually had a chance to test running all this on arm64. The version of 3.8 that we install only runs on x86_64. So when you run it on arm64, it's emulating x86_64. The wheel that you built is fine, but you're not able to test it.

This PR seems to only change the universal2/arm64 build identifiers. I don't think that will change anything, as on macOS you can only have one minor version of Python installed at once, and the first one that cibuildwheel finds, it will install. So if we were going to make this change we'd have to change all three 3.8 URLs.

The Python 3.8 arm64 support was added in #704. I don't have time right now but perhaps there are some clues in there why we're not using the macos11 versions.

henryiii · 2022-07-08T11:37:05Z

We tried very hard to use this when it came out, but it doesn’t work. CPython only has partial support for 3.8 on ARM - it was squeezed in at the last minute in the last binary release. IIRC you can’t produce 10.9-10.15 compatible wheels for Intel from the ARM version since Python didn’t include a 10.9 compatible Intel version in the binary, like they normally do (and do in 3.9+). You can look at the PR where 3.8 was added, I think.

I think to make this “work” we’d need to special case when running on ARM and have the (big) caveat that you can’t build for macOS 10 from AS (but only for 3.8). Is 3.8 still the version Apple ships? I’m traveling and won’t be my AS machine till the 27th.

henryiii · 2022-07-08T11:47:05Z

Read here:

I think we should see if we can build from AS too, but we haven’t had CI runners for it and it will take some changes and should error out of macOS version is set <11.

I’m pretty sure we use the AS version from this table when building pure AS wheels from Intel.

mayeut · 2022-07-09T11:58:36Z

IIRC you can’t produce 10.9-10.15 compatible wheels for Intel from the ARM version since Python didn’t include a 10.9 compatible Intel version in the binary, like they normally do (and do in 3.9+).

Now that MACOSX_DEPLOYMENT_TARGET is always set, I don't think there's a big issue on what you can target (i.e. will still be 10.9 for x86_64) unless I'm missing something. However, using a macOS 10.x runner to do the build won't be possible anymore with this installer which requires macOS 11+.

henryiii · 2022-07-09T22:57:22Z

From what I understand, you can't set MACOSX_DEPLOYMENT_TARGET lower than whatever CPython was prepared with, otherwise we'd not need to download the official binaries at all. I could be wrong, but I'm pretty sure MACOSX_DEPLOYMENT_TARGET can only be increased.

Czaki · 2022-07-09T23:08:20Z

yes. here is a wheel code fragment with a warning about this:

https://github.com/pypa/wheel/blob/9d4264d2f23ea44dd85639a27fa6b284261921cd/src/wheel/macosx_libfile.py#L411-L419

joerick · 2022-07-10T16:07:20Z

Thanks all for the informative discussion. I guess the upshot is that we keep the macos10.9 installer, and we can't test 3.8 arm64 wheels, even when running on arm64.

However, we should change the behaviour when running on Apple Silicon, to not try to test the arm64 3.8 wheel, and include an informative message explaining why.

I've added a warning for users, so they know that it doesn't get tested. See discussion in #1169 for the reason why. Fixes #1168, fixes #1169

Per discussion in pypa#1169, the default installer used for cp38 is an Intel installer. It makes sense to skip testing arm64 wheels in this case. However, if the user choose to manually install the universal2 CPython version, then, tests shall be run on arm64. This allows users that either target 11.0+ on intel, universal2 or only build for arm64 to get the arm64 wheel tested on AppleSilicon.

- not testable due to cpython38 constraints (pypa/cibuildwheel#1169)

joerick · 2024-02-02T12:26:00Z

Just a note for anyone finding this now that arm64 runners are more widely available. If it's important to you to test the cp38 arm64 wheels, there's a workaround.

You can choose to install a different version of Python 3.8 on your runner before invoking cibuildwheel:

curl -fsSLO https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg
sudo installer -pkg python-3.8.10-macos11.pkg -target /
# ... invoke cibuildwheel ...

Just be aware that if you use this Python to build x86_64 wheels too, this increases the minimum supported macOS version on x86_64 to 11.0.

joerick · 2024-02-02T15:57:25Z

For reference, this gives an idea of which Python 3.8 users are using which OS.

That's the last 30 days, from our stats spreadsheet

henryiii · 2024-02-02T16:33:20Z

Do you have your old python versions the last month plot? That was interesting to see the fraction of 3.8 ARM users. I also don't remember what the default Python version was for those versions. I'm on the latest macOS and I see 3.9 in my /usr/bin, so the've finally updated (a little).

joerick · 2024-02-02T16:56:03Z

Though I have no idea what's going on with the Python 2.7 arm64 users!? I guess that Homebrew offers a build of that?

henryiii · 2024-02-02T16:59:35Z

I don't think they do, I don't think there's an open source 2.7 for AS, and homebrew doesn't offer 2.7 anymore. But Apple provided their own patched version in macOS 11 - 12.3.

See pypa/cibuildwheel#1169

hombit added 2 commits July 7, 2022 10:29

Fix cp38-macosx_universal2 Python pkg URL

c188a4a

Previous URL pointed to x86_64 version

Update build-platforms.toml

8dc65be

hombit changed the title ~~Fix cp38-macosx_universal2 Python pkg URL~~ Fix cp38-macosx universal2 and arm64 Python pkg URL Jul 7, 2022

Czaki reviewed Jul 7, 2022

View reviewed changes

joerick added a commit that referenced this pull request Jul 11, 2022

Don't attempt to test CPython 3.8 wheels on Apple Silicon.

3524a5a

I've added a warning for users, so they know that it doesn't get tested. See discussion in #1169 for the reason why. Fixes #1168, fixes #1169

joerick mentioned this pull request Jul 11, 2022

Don't attempt to test CPython 3.8 wheels on Apple Silicon. #1171

Merged

mayeut closed this in #1171 Jul 14, 2022

mayeut pushed a commit that referenced this pull request Jul 14, 2022

Don't attempt to test CPython 3.8 wheels on Apple Silicon. (#1171)

5780092

I've added a warning for users, so they know that it doesn't get tested. See discussion in #1169 for the reason why. Fixes #1168, fixes #1169

Jackenmen mentioned this pull request Jul 23, 2022

Add support for Cirrus CI #1191

Merged

mayeut mentioned this pull request Sep 25, 2022

feature: allow testing cp38-macosx_arm64 wheel #1283

Merged

ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Jan 16, 2023

skip testing for cp38-macosx_arm64 (pypa/cibuildwheel#1169)

40e51a8

ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Jan 22, 2023

skip testing for cp38-macosx_arm64 (pypa/cibuildwheel#1169)

63115b2

ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Dec 22, 2023

drop py38 build on macosx_arm64

cef6dc8

- not testable due to cpython38 constraints (pypa/cibuildwheel#1169)

ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Dec 24, 2023

drop py38 build on macosx_arm64

52ce8f5

- not testable due to cpython38 constraints (pypa/cibuildwheel#1169)

jorisvandenbossche mentioned this pull request Feb 13, 2024

RLS/BLD: use native Apple Silicon macOS for arm64 wheels; remove universal2 shapely/shapely#1990

Merged

tttapa mentioned this pull request Feb 27, 2024

Compiling on macos-14 (arm64) cp39 result in Unable to determine extension suffix. Try manually setting PY_BUILD_EXT_SUFFIX. tttapa/py-build-cmake#24

Closed

laggykiller mentioned this pull request Feb 28, 2024

Testing universal2 wheel on arm64 macOS cp38 runner result in not a supported wheel on this platform #1767

Closed

tlambert03 mentioned this pull request Mar 6, 2024

build arm64 on macos-14 pyapp-kit/psygnal#290

Closed

skirpichev added a commit to skirpichev/gmpy that referenced this pull request Mar 9, 2024

Filter out cibuildwheel warning for testing MacOS wheels on 3.8

df7277d

See pypa/cibuildwheel#1169

cdce8p mentioned this pull request Apr 4, 2024

Update cibuildwheel to 2.17.0 mypyc/mypy_mypyc-wheels#78

Merged

dnanuti mentioned this pull request Apr 10, 2024

Build macOS wheels and integration tests awslabs/s3-connector-for-pytorch#197

Merged

marcelm added a commit to marcelm/dnaio that referenced this pull request Apr 23, 2024

Do not test CPython 3.8 arm64 wheels on macOS

1034dde

See pypa/cibuildwheel#1169

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix cp38-macosx universal2 and arm64 Python pkg URL #1169

Fix cp38-macosx universal2 and arm64 Python pkg URL #1169

hombit commented Jul 7, 2022 •

edited

joerick commented Jul 7, 2022

hombit commented Jul 7, 2022

Czaki Jul 7, 2022

hombit Jul 7, 2022

hombit Jul 7, 2022 •

edited

Czaki Jul 7, 2022

joerick commented Jul 8, 2022 •

edited

henryiii commented Jul 8, 2022

henryiii commented Jul 8, 2022 •

edited

mayeut commented Jul 9, 2022

henryiii commented Jul 9, 2022

Czaki commented Jul 9, 2022

joerick commented Jul 10, 2022

joerick commented Feb 2, 2024

joerick commented Feb 2, 2024 •

edited

henryiii commented Feb 2, 2024

joerick commented Feb 2, 2024

henryiii commented Feb 2, 2024

Fix cp38-macosx universal2 and arm64 Python pkg URL #1169

Fix cp38-macosx universal2 and arm64 Python pkg URL #1169

Conversation

hombit commented Jul 7, 2022 • edited

joerick commented Jul 7, 2022

hombit commented Jul 7, 2022

Czaki Jul 7, 2022

Choose a reason for hiding this comment

hombit Jul 7, 2022

Choose a reason for hiding this comment

hombit Jul 7, 2022 • edited

Choose a reason for hiding this comment

Czaki Jul 7, 2022

Choose a reason for hiding this comment

joerick commented Jul 8, 2022 • edited

henryiii commented Jul 8, 2022

henryiii commented Jul 8, 2022 • edited

mayeut commented Jul 9, 2022

henryiii commented Jul 9, 2022

Czaki commented Jul 9, 2022

joerick commented Jul 10, 2022

joerick commented Feb 2, 2024

joerick commented Feb 2, 2024 • edited

henryiii commented Feb 2, 2024

joerick commented Feb 2, 2024

henryiii commented Feb 2, 2024

hombit commented Jul 7, 2022 •

edited

hombit Jul 7, 2022 •

edited

joerick commented Jul 8, 2022 •

edited

henryiii commented Jul 8, 2022 •

edited

joerick commented Feb 2, 2024 •

edited