dlopen(myext.bundle, 0x0009): symbol not found in flat namespace (_some_symbol) (RuntimeError) on XCode 14.2

[eregon]

From ruby/zlib#75
Affects: TruffleRuby >= 24.0

There is a bug in XCode 14.2 that even though TruffleRuby passes RTLD_LAZY to dlopen() and -Wl,-undefined,dynamic_lookup when linking,
the new linker in XCode 14.2 ignores those and resolves symbols eagerly.
Which seems nothing less than a violation of the POSIX semantics of dlopen() with RTLD_LAZY and yet another breaking change for macOS's native toolchain.
And the macOS man page for dlopen doesn't even bother mentioning this issue.

This is thankfully fixed in XCode 14.3+, so it's really only an issue with XCode 14.2.

This caused similar issues in CRuby and CPython, links at ruby/zlib#75 (comment), notably https://bugs.ruby-lang.org/issues/18912 and https://bugs.ruby-lang.org/issues/19005.

Possible workarounds:

• Using -no_fixup_chains but this flag seems to not exist on older versions of XCode so it seems inconvenient. BTW getting the XCode version seems a bit messy.
• Using MACOSX_DEPLOYMENT_TARGET=11.0 this likely disables the fixup chains stuff and 11.0=Big Sur the currently minimum supported version. But it might disable using newer macOS stuff, so it seems not ideal to set that when installing extensions.
• Use -bundle_loader which seems to be what CRuby did Link ext bundles with bundle loader option for newer ld64 of Ventura ruby/ruby#6193. But macOS linker warnings in macOS ventura python/cpython#97524 makes it sound like all symbols need to be resolved and that's not the case.
• Define dummy functions which raise for all C API functions, this is not OK because it breaks have_func "foo" and so prevents extensions to detect if a function is available. Maybe it could be worked around by using a different native lib for have_func but it sounds hacky.

So my current plan instead is to rely on users using a non-broken version of XCode, i.e., not XCode 14.2.

It's not ideal because GitHub Actions macos-latest = macos-12 currently actually uses XCode 14.2 :/
Either macos-13 or macos-11 works fine.

[eregon]

FWIW it seems CRuby still uses -Wl,-undefined,dynamic_lookup:
https://github.com/eregon/actions-shell/actions/runs/7045968254/job/19176775307#step:4:60

 "DLDFLAGS"=>
  "-L/Users/runner/hostedtoolcache/Ruby/3.2.2/x64/lib  -L/usr/local/opt/gmp/lib -Wl,-multiply_defined,suppress -Wl,-undefined,dynamic_lookup",

And from https://github.com/ruby/ruby/pull/6193/files
it seems the EXTDLDFLAGS='$(LIBRUBYARG_SHARED)' case, i.e. no -bundle_loader flag:
"EXTDLDFLAGS"=>"-lruby.3.2",

We could try to link extensions to @rpath/libtruffleruby.dylib but I'm not sure if that would help, because libtruffleruby.dylib does not define all symbols, some symbols are truly undefined and need to stay like that until they are actually required at runtime.

[eregon]

From ruby/zlib#75 (comment), MACOSX_DEPLOYMENT_TARGET=11.0 seems the easiest workaround for the cases of using XCode/Command line tools 14.2.
But unfortunately checking the version of Command line tools is quite slow: 70ms to 188ms (first call).
That doesn't seem a nice overhead to add to rbconfig.rb or mkmf.rb.
Also it's not obvious how to inject this into the generated Makefile it would likely need to be in rbconfig.rb and that means require 'rbconfig' would be that much slower which seems not worth it for such an edge case of broken XCode linker.
So I'm thinking to workaround this in setup-ruby for macos-12, and otherwise rely on people not using that broken XCode 14.2 linker (or manually setting MACOSX_DEPLOYMENT_TARGET=11.0).

[eregon]

Another idea was maybe it's worth asking GitHub to provide XCode 14.3 on macos-12.
But I found actions/runner-images#8320 (comment) so that seems not possible. Confirmed here