DISCUSS: Raise MacOS minimum target from 10.9 to 10.13?

[h-vetinari]

current todo list

• make an announcement (add announcement for dropping MacOS<10.13 #1993)
• decide how we want to enforce the __osx constraint
  • going to use recipe_append.yaml generated by conda-forge-ci-setup (add recipe_append.yaml for osx with __osx >=10.13 in run conda-forge-ci-setup-feedstock#260)
  • The append doesn't work for recipes with outputs, so we'll have to use a run export on the compilers for osx.
• Do META: {{ stdlib("c") }} migration #2102
• Decide if we will allow folks to set a min version less the global minimum.
  • If yes, do nothing. stdlib jinja mechanism will allow this
  • If no, then we need a minimigrator to adjust / remove any custom minimum pins below 10.13
• clean up recipes with outdated pins in CBC (that don't explicitly want/need it)
• figure out if there are any finicky builds on osx that need adjusting? I recall some but maybe that is out of date by now.
• figure out remaining(?) places in the infrastructure that involve assumptions about the sysroot and fix them up
• patch old builds with 10.9 __osx constraint?
• bump c_stdlib_version to 10.13 for osx-64 in global pinnings

issue text from start

When azure deprecated the macOS-10.15 images, it turned out macOS-11 still supports targets all the way back to our baseline 10.9, and so it was decided to separate the discussion of the image upgrade from the default MACOSX_DEPLOYMENT_TARGET.

However our baseline target 10.9 is now EOL since almost 6(!) years, so I guess that discussion should be had at some point. I don't think there's a really urgent need (what with shipping our own libcxx, plus the _LIBCPP_DISABLE_AVAILABILITY mechanism¹), but this topic recently came up in a numpy discussion, so I though I'd open an issue.

Originally, I thought our hand would be forced once macOS-11 images are deprecated, but it turns out that even in the macOS-12 images, there's still an SDK with Xcode 13.1, which in turn still supports targets back to 10.9. Only once we're forced to use Xcode 14+ would we have to bump the minimum target to 10.13.

July 2023 update

Out of curiosity, I wanted to check when the last bump of the MACOSX_DEPLOYMENT_TARGET happened, and, as it turns out, it's been at 10.9 since the initial commit of https://github.com/conda-forge/conda-forge-pinning-feedstock 🤯

With a bit of digging (and luck), I found the bump from 10.7 to 10.9 though: conda-forge/toolchain-feedstock@7a470c5 - 7 years ago. At the time, 10.9 was just before its EOL. If we applied the same standard today we should jump directly to 11.0.

Given that most users these days are on much newer versions, and want to use newer features (relevant for our packaging, like support for metal or the new LAPACK implementation), I think it might make sense to stop dragging our feet so much on this. We're (slowly but steadily) moving with the time on linux and windows as well, why should osx need to fall so far behind?

To substantiate this a bit more, I wanted to look at the usage numbers for different MacOS versions. The broadest measure is "everyone who uses a browser", but there are apparently no good usage numbers, because Apple keeps misreporting its OS version in HTTP headers, for some complicated reasons. I did find however that 92% are on 10.15+, which is the version every version after that pretends to be. Notably, all (distinguishable) versions that're EOL have max 1-1.5% usage numbers (<7% cumulated) -- and again, this is all MacOS users, not just those of conda-forge.

Footnotes

1. though there are quite a few feedstocks that just bump the MACOSX_DEPLOYMENT_TARGET because it happens to unbreak CI - not least because the compiler errors point in this direction ("X is unavailable: introduced in macOS Y.Z") - but often not adding __osx >={{ MACOSX_DEPLOYMENT_TARGET|default("10.9") }} as a dependence; such packages might already be broken on old OSX anyway... ↩

[isuruf]

FYI: We are not restricted by the xcode version and our builds are completely independent of the macos build image.
See https://github.com/conda-forge/conda-forge-ci-setup-feedstock/blob/main/recipe/download_osx_sdk.sh

[ngam]

The phracker repo we get the SDKs from is no longer maintained.

our builds are completely independent of the macos build image

Yes, but we also use a hack to expose recent Metal stuff, e.g. in pytorch:

• https://github.com/conda-forge/pytorch-cpu-feedstock/blob/5df993a140f950439191ac1149045b7239eddf68/conda-forge.yml#L1-L5
• https://github.com/conda-forge/pytorch-cpu-feedstock/blob/5df993a140f950439191ac1149045b7239eddf68/recipe/conda_build_config.yaml#L1-L2

Maybe irrelevant to the main discussion here though...

[h-vetinari]

One aspect of this discussion that's flown under the radar for a long time is now starting to come to the fore. While we can patch around the C++ standard library by shipping our own up-to-date libcxx, we're dependent on the sdk for an up-to-date C standard library.

So far, many projects have not relied much on newer C functionality, but certain projects (crucially LLVM, which is our main compiler on osx) now starts to rely on C11 features more and more. I've had a lot of problems with building libcxx 15, and one of the big issues was the lack of C11 support on osx (and linux).

I've now solved this by reverting a commit that removed a lot of workarounds in the LLVM 15 timeframe, but this is not a long-term solution. Of course, we could raise the MACOSX_DEPLOYMENT_TARGET for libcxx, but given its central role in conda-forge, that's likely effectively equivalent to bumping the minimum version everywhere.

I haven't yet managed to google (resp. experiment) which SDK added the required C11 functions. I can do that if people are interested.

[h-vetinari]

libcxx 17 has recently dropped support for macOS<10.13 when building the shared libcxx library (I noted this in conda-forge/libcxx-feedstock#110 when the RFC opened).

As far as I can tell, this is more fundamental than the availability workarounds we do (for building projects on top of libcxx), as it will really become impossible to build (shared) libcxx itself for targets <10.13.

Like for other fundamental toolchain dependencies (like upstream requiring vs2019 vs. vs2017 recently), I don't think it's feasible to try to keep patching back in support for older macos versions, as there will be zero coverage upstream and therefore things will bitrot fast.

Seeing that LLVM is our default toolchain on osx (and assuming I'm not overlooking something), I think we'll have to increase the minimum version to 10.13 at the latest when we switch our default compilers on osx to clang 17 (earliest that could happen would be late 2023; of course we could drag it out for quite a while longer by staying on clang 16 for another year or so afterwards).

PS. It's worth noting that libcxx had been waiting on chrome to bump their required minimum version to 10.13, which was apparently the last major "holdout" (from the POV of LLVM). So I think end of 2023 would actually a very reasonable timeline to bump to 10.13; at that point 10.12 will have been EOL for 4 years.

[h-vetinari]

@h-vetinari: It's worth noting that libcxx had been waiting on chrome to bump their required minimum version to 10.13, which was apparently the last major "holdout" (from the POV of LLVM)

I just noticed that abseil (and google more broadly¹) also raised its minimum to 10.13, and uses as official policy what LLVM did informally, i.e. MacOS support is defined in this document as:

We will support [macOS] back to the oldest macOS target platform needed by Chrome

There's also an overview in this table.

Reminder: To the degree that the lower bound of the SDK depends on C++'s stdlib, we can continue to circumvent this with _LIBCPP_DISABLE_AVAILABILITY, but on the C side there are no workarounds. From the outside, it's hard to say if a project is relying on C's stdlib functionality, but use of C11's aligned_alloc (including std::aligned_alloc on the C++ side) is a sure sign, for example.

Footnotes

1. notably protobuf, but also google-cloud-cpp, gtest, and probably several others ↩

[mfansler]

Just want to chime in that several feedstocks are requiring the -D_LIBCPP_DISABLE_AVAILABILITY flag to get passing on osx-64 for the r-base=4.3 migration builds. This is specific to R 4.3 builds, which default to C++17 standard.

It's not widespread, but it is a front where the continued support for 10.9 adds to maintainer workload.

[potiuk]

I redirected all the reports of ours we had - we also had some people on Slack having similar problems I will also direct them there.

[h-vetinari]

Out of curiosity, I wanted to check when the last bump of the MACOSX_DEPLOYMENT_TARGET happened, and, as it turns out, it's been at 10.9 since the initial commit of https://github.com/conda-forge/conda-forge-pinning-feedstock 🤯

With a bit of digging (and luck), I found the bump from 10.7 to 10.9 though: conda-forge/toolchain-feedstock@7a470c5 - 7 years ago. At the time, 10.9 was just before its EOL. If we applied the same standard today we should jump directly to 11.0 (see here).

Given that most users these days are on much newer versions, and want to use newer features (relevant for our packaging, like support for metal or the new LAPACK implementation), I think it might make sense to stop dragging our feet so much on this. We're (slowly but steadily) moving with the time on linux and windows as well, why should osx need to fall so far behind?

To substantiate this a bit more, I wanted to look at the usage numbers for different MacOS versions. The broadest measure is "everyone who uses a browser", but there are apparently no good usage numbers, because Apple keeps misreporting its OS version in HTTP headers, for some complicated reasons. I did find however that 92% are on 10.15+, which is the version every version after that pretends to be. Notably, all (distinguishable) versions that're EOL have max. 1-1.5% usage numbers (<7% cumulated) -- and again, this is all MacOS users, not just those of conda-forge.

[h-vetinari]

Newest abseil requires MacOS 10.13 (and fails with 10.9). I'd like to migrate this together with the new libgrpc & libprotobuf, but seeing how widely abseil is used, that's effectively a conda-forge-wide decision (at least as far as C++ deps are concerned).

I saw that @chenghlee had put up the idea of dropping osx-64 support entirely (in the meeting notes of the last core call, though apparently it wasn't discussed yet). Given all of the above datapoints (libcxx, clang, abseil, protobuf, C11's aligned_alloc, how long ago the EOL was, the comparison to the last bump, etc.), is there any serious opposition to just bumping to 10.13?

[beckermr]

@chrisburr should chime in. He grabbed a bunch of stats from pypi on what to move to next.

I don’t want this to move any faster than it has to. There’s no reason to leave users on older systems behind if we don’t have to.

[chenghlee]

...put up the idea of dropping osx-64 support entirely.

This was mostly to spur discussion of when/if we should start considering that, mostly because I expect that at some point, we will no longer have to (native) osx-64 CI resources. That said, based on the PPC to x86 transition, I'm guessing Apple will at least support for macOS on x86_64 for at least another 2-3 years, so that's a discussion we can continue to punt for a while.

[ngam]

In my opinion, I think we should start considering dropping osx-64 when Apple drops Rosetta 2 support from osx-arm64 systems. Of course, if we stop having osx-64 CI, then we have no choice but to drop it…

I personally totally support increasing the minimum for macOS. I would even advocate to go to 11 or even 12 directly. I completely sympathize with the need to support older linux systems (believe me, I have access to many of these gov systems myself) but if I understand things correctly, that simply doesn’t apply the Apple ecosystem (Apple is aggressive in pushing people to upgrade). If people have other experiences with Apple systems, please weigh in so that your voice is heard!

[beckermr]

Agreed @h-vetinari that this is pretty straight forward for the bot. Getting that migrator to appear on the status page might be a bit more involved.

[h-vetinari]

Google raised its "fundamental" baseline to 10.15 as of google/oss-policies-info@9a9bfe8, which will very likely show up in abseil/grpc/protobuf in the next couple of months.

Also, libcxx took chrome as a lower bound for deciding to bump to 10.13 last year, so it's possible they'll follow soonish as well.

[h-vetinari]

@isuruf, regarding the migration strategy for this you noted in the core call (minutes):

Think we should have a mini-migrator (piggyback) so we don't have to rebuild all C/C++ packages; only rebuild when we really have to.

Do you have something in mind where to piggy-back onto? Something like boost or protobuf?

[isuruf]

I was thinking all migrations.

[h-vetinari]

I saw that @chenghlee had put up the idea of dropping osx-64 support entirely (in the meeting notes of the last core call, though apparently it wasn't discussed yet).

Speaking of which, pytorch 2.2 just deprecated x64 support, to be removed in 2.3. Not proposing this for conda-forge, just noting the signs of the times here in this issue a bit. We'll likely still have a couple years left w.r.t. the constraints of our own infrastructure, though it could actually end up being the case that the thing that breaks us the fastest is GHA/AP removing the last x64 agents for macOS.

[isuruf]

thing that breaks us the fastest is GHA/AP removing the last x64 agents for macOS.

AP would likely provide arm64 agents like GHA is doing now which we can use to cross compile for x64 and run tests with rosetta 2 emulation.

[h-vetinari]

@h-vetinari: Do you have something in mind where to piggy-back onto? Something like boost or protobuf?

@isuruf: I was thinking all migrations.

That's a good idea (after testing it on one migration first, I'd say). I've started writing a piggyback migrator, and wanted to check something else:

what (if anything) are we going to set for c_stdlib{,_version} on windows? ucrt? Because if we don't set up a corresponding package on windows, we'd have to add # [unix] selectors for every {{ stdlib('c') }} line, and that would get messy fast if the compiler itself already has a selector.

[h-vetinari]

I've started writing a piggyback migrator

Got a first draft, please let me know if you have some feedstocks you'd like it tested on. CC @conda-forge/core

[xhochy]

Happy to have it tested on some rust feedstocks, but I migrated most of them manually already. You could still run it over e.g. polars and see what kind of changes it will do.

[isuruf]

An example feedstock with selectors on compilers and things like compiler('m2w64_c') would be good to check.

[h-vetinari]

Heads-up that there's now a PR to bump the c_stdlib_version in the global pinning to 10.13: conda-forge/conda-forge-pinning-feedstock#5829. We agreed in today's core call that things are ready to merge.

Looking at the remaining points in the OP:

• clean up recipes with outdated pins in CBC (that don't explicitly want/need it)

If c_stdlib_version and MACOSX_DEPLOYMENT_TARGET do not match, we always take the higher one. We also warn if someone sets a mismatched version in their local CBC, so this should be handled IMO.

• figure out if there are any finicky builds on osx that need adjusting? I recall some but maybe that is out of date by now.

Nothing that comes to my mind.

• figure out remaining(?) places in the infrastructure that involve assumptions about the sysroot and fix them up

I think we caught most things in the context of #2102, but that's probably going to be an ongoing process for any remaining stragglers. This is also why we're doing the 10.13 bump separately from the glibc bump. so we can learn from the former (to the extent possible) for the latter.

• patch old builds with 10.9 __osx constraint?

That shouldn't be necessary AFAICT.

Please comment if I'm missing something. Otherwise I'll close this issue with conda-forge/conda-forge-pinning-feedstock#5829.

[h-vetinari]

I just noticed that is a recipe uses

c_stdlib_version:          # [osx and x86_64]
  - "10.12"                # [osx and x86_64]
MACOSX_SDK_VERSION:        # [osx and x86_64]
  - "10.12"                # [osx and x86_64]

the merge logic of smithy will not be applied to MACOSX_SDK_VERSION, which will end up being incorrectly populated as 10.12 and then cause a failure at build time (apparently also the ci-setup doesn't take the maximum correctly here). I've opened conda-forge/conda-smithy#1927