macos/py38/arm: wheels tagged 10_16 for MACOSX_DEPLOYMENT_TARGET=12

[stefanv]

Description

While building scikit-image 0.20rc4, we noticed that (only) our Python 3.8 MacOS wheel gets tagged incorrectly as 10_16 instead of as 12_0.

The cibuildwheel history shows that this version sometimes comes up when using macos 11 (although our build image is 12).

Example build logs can be seen here:

• MACOSX_DEPLOYMENT_TARGET=12.0: https://github.com/scikit-image/scikit-image/actions/runs/4088341657/jobs/7049918179
• MACOSX_DEPLOYMENT_TARGET=12: https://github.com/scikit-image/scikit-image/actions/runs/4110960293/jobs/7094258353

For an example of a successful build, see e.g. the one for py39 at https://github.com/scikit-image/scikit-image/actions/runs/4110960293/jobs/7094258456:

  scikit_image-0.20.0rc5.dev0-cp39-cp39-macosx_12_0_arm64.whl   

Any ideas of why py38 would behave differently?

Build log

https://pipelines.actions.githubusercontent.com/serviceHosts/40f46268-6c19-4e51-9405-da082fc91a36/_apis/pipelines/1/runs/10844/signedlogcontent/19?urlExpires=2023-02-07T06%3A47%3A18.4042231Z&urlSigningMethod=HMACV1&urlSignature=xwWdIpUSdnztqH4bZ1eTjr2X5eA3QBYlhlVYz0a%2Fl48%3D

CI config

https://github.com/scikit-image/scikit-image/blob/main/.github/workflows/wheel_tests_and_release.yml

[joerick]

Weird. This might be the same issue as #1410. Are you able to test with an older version of cibuildwheel? e.g. might be worth trying v2.11.4, since this issue has only popped up in the last week or so.

[stefanv]

Also broken under v2.11.4.

https://github.com/scikit-image/scikit-image/actions/runs/4117077160/jobs/7107935610

[stefanv]

Same problem on v2.10.2.

https://github.com/scikit-image/scikit-image/actions/runs/4117218052/jobs/7108250063

[stefanv]

Also not working with latest HEAD, fwiw:

https://github.com/scikit-image/scikit-image/actions/runs/4118133692/jobs/7110290899

[mayeut]

does exporting environment variable SYSTEM_VERSION_COMPAT=0 help ?

[stefanv]

That does fix it! 🎉

https://github.com/scikit-image/scikit-image/actions/runs/4121828219/jobs/7117902113

So, what is this black magic, and is it safe to use it? I don't see it anywhere in the docs.

[mayeut]

So, what is this black magic ?

The black magic is explained in there (& mostly the related issue): pypa/packaging#513

is it safe to use it?

As mentioned in the PR linked, there are absolutely no docs from Apple for this (at least at that time) so other possible side effects are unknown...

The safest would be to track down which package in the build system tries to label this wheel as 10.16 & ask them to either respect MACOSX_DEPLOYMENT_TARGET and/or read the macOS version with SYSTEM_VERSION_COMPAT=0 set (as done in packaging) when seeing 10.16+

[mayeut]

given your build only targets arm64 macOS 12+ & runs on macOS-12, maybe installing manually cpython 3.8 universal2 would lead to the same outcome without the need to export SYSTEM_VERSION_COMPAT or track down which package is labelling the wheel 10.16 in the first place.

c.f. https://cibuildwheel.readthedocs.io/en/stable/faq/#macos-building-cpython-38-wheels-on-arm64

[stefanv]

From https://eclecticlight.co/2020/08/13/macos-version-numbering-isnt-so-simple/

In scripted languages which run within a shell environment, there’s an environmental variable which controls the version number given. Set SYSTEM_VERSION_COMPAT=1 and Big Sur returns 10.16; leave that variable unset, or SYSTEM_VERSION_COMPAT=0, and it returns 11.0.

So, what I'd expect is for the default to be correct, and to have to invoke SYSTEM_VERSION_COMPAT to get the old 10.x version out; here we see the opposite. So, where is SYSTEM_VERSION_COMPAT being set to 1?

Either way, building without emulation is preferable, so I'm trying out the other route as well.

[stefanv]

Excellent. Build time went from 12 to 8 minutes, and the wheel is correctly tagged. Thanks very much @mayeut!