Problem producing an importable extension on macOS ARM64

[lelit]

Hi, I'm seeking some cue about a linkage problem I do not understand.

I'm the author of pglast: it basically builds some facilities on top of a static library containing the PostgreSQL parser.

Recently I added silicon macs to the build matrix. Since I do not own that kind of machine, and that I disabled CI tests for that combo, it went unnoticed that, although the process completed successfully, the wheel contains a broken shared object.

A friend lent me an hour of his M3 machine, and I found that, apparently, the ARM64 pglast/parser.cpython-311-darwin.so was not properly linked against the static libpg_query.a, as even the size suggests: the Intel one is much bigger, and inspecting them with nm it actually contains all the expected _pg_query__XYZ concrete symbols, missing in the ARM64 one.

I looked around and tried a different approach, but even that does not seem to solve the problem.

I wonder if some kind soul around here, more expert than me on that architecture, may shed some light on the problem?

Thanks in advance for any hint!

[joerick]

Hi there

Recently I added silicon macs to the build matrix.

You might already know this, but this config is not specifying an arm64 machine. This runner is on x86_64, Github hasn't released arm64 runners yet(!). So this is cross-compiling arm64 from x86_64.

I looked around and tried a different approach, but even that does not seem to solve the problem.

An issue here is that you're not detecting the right case in setup.py. Because we're running this build on x86_64, the platform.mac_ver() is still x86_64. Take a look at sysconfig.get_platform(), that will return the correct result when cross-compiling.

But I suspect the issue is that your dependency, libpg_query, isn't cross-compiling well. cibuildwheel does set the ARCHFLAGS variable, but there might need to be some more options fiddling to get it to work.

But it's going to be easier to do it native. CirrusCI does offer (limited) free arm64 CI for open source, perhaps you can try that.

[lelit]

Thanks a lot, I will insist in my investigation, and read about CirrusCI.
Maybe you happen to know how easy/possible is it to simulate what is happening on GA on a local Intel Mac?

[joerick]

On an Intel Mac, you can do cibuildwheel --platform macos --archs arm64 to try a build locally. As long as you have Command-line Tools For Xcode installed, that will be very similar to the GHA build.

[lelit]

Much appreciated, thank you!

[lelit]

An update before closing this, for posterity.

First thing I tried has been changing the way the extension gets compiled: instead of building the libpg_query static library I explicitly listed all its sources, and that worked, to my surprise, producing a loadable binary also on Silicon macs.

I eventually switched to the new macos-14 GH Actions runner, re-enabling tests, and as expected that worked too.

Thanks again Joe for your hints!