Add aarch64 support for macOS/darwin targets

[maxfierke]

Resolves #10066

Work from @RomainFranceschini and I, this PR adds the libc bindings for aarch64-apple-darwin (thanks Romain!) and implements proper handling for the arm64-apple-darwin triple (which is a synonym, and presented by default via LLVM on Apple Silicon)

Compiling requires a working x86_64 crystal compiler (via Brew for e.g.) that can be run via Rosetta 2. For working on this, we've installed ARM homebrew to the /opt/homebrew prefix and Intel Homebrew to its usual /usr/local prefix. More info on that here.

As the Intel homebrew is installed to a default lookup path, x86_64 libraries will be read first, so we need to do some overriding of environment variables to get the linker to look up arm64/aarch64 libraries correctly.

Cross-compilation instructions for the compiler:

Environment

export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
alias ibrew="arch -x86_64 /usr/local/bin/brew"
export INTEL_BREW_PREFIX="$(ibrew --prefix)"
export INTEL_LLVM_ROOT="$INTEL_BREW_PREFIX/opt/llvm"
export INTEL_LLVM_CONFIG="$INTEL_LLVM_ROOT/bin/llvm-config"

export ARM_BREW_PREFIX="$(brew --prefix)"
export ARM_LLVM_ROOT="$ARM_BREW_PREFIX/opt/llvm"
export ARM_LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config"

Compiling libcrystal.a

cc -c -o sigfault.o src/ext/sigfault.c
ar -rcs libcrystal.a sigfault.o

Compiling bdw-gc

$ git clone https://github.com/ivmai/bdwgc.git
$ cd bdwgc
$ git clone https://github.com/ivmai/libatomic_ops.git
$ autoreconf -vif
$ ./configure --enable-static --disable-shared
$ make -j
$ make check

Then copy .libs/libgc.a into the Crystal source root

(You can also brew install --HEAD bdw-gc, which has the MT patches too, and is effectively the same as doing the above.)

Compiling Crystal for ARM

$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \
   LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \
   CPPFLAGS="-I$INTEL_LLVM_ROOT/include" \
   CC="$INTEL_LLVM_ROOT/bin/clang" \
   AR="$INTEL_LLVM_ROOT/bin/llvm-ar" \
   arch -x86_64 make

$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \
   LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \
   CRYSTAL_CONFIG_TARGET=aarch64-apple-darwin \
   CRYSTAL_CONFIG_LIBRARY_PATH="$ARM_BREW_PREFIX/lib" \
   arch -x86_64 ./bin/crystal build src/compiler/crystal.cr --cross-compile --target aarch64-apple-darwin -Dwithout_playground

$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   $ARM_LLVM_ROOT/bin/clang -I$ARM_LLVM_ROOT/include -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc

$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   $ARM_LLVM_ROOT/bin/clang crystal.o -o .build/crystal  -rdynamic -L$ARM_BREW_PREFIX/lib src/llvm/ext/llvm_ext.o `"$ARM_LLVM_ROOT/bin/llvm-config" --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre libgc.a -lpthread libcrystal.a -L$(brew --prefix libevent)/lib -levent -liconv -ldl

$ file .build/crystal
.build/crystal: Mach-O 64-bit executable arm64

Compiling and running the compiler specs

$ CRYSTAL_PATH=$(pwd)/src \
   CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
   CC="$ARM_LLVM_ROOT/bin/clang" \
   LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig" \
   ./bin/crystal build -o spec/compiler_spec spec/compiler_spec.cr --link-flags "$PWD/libcrystal.a" -Di_know_what_im_doing

$ CRYSTAL_PATH=$(pwd)/src \
   CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
   CC="$ARM_LLVM_ROOT/bin/clang" \
   LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib $(pwd)/libgc.a $(pwd)/libcrystal.a" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig" \
   spec/compiler_spec

Compiling and running the std library specs

$ CRYSTAL_PATH=$(pwd)/src \
  CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
  CC="$ARM_LLVM_ROOT/bin/clang" \
  LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
  LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
  CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
  PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig" \
  ./bin/crystal build -o spec/std_spec spec/std_spec.cr --target aarch64-apple-darwin --link-flags "$PWD/libcrystal.a" -Di_know_what_im_doing

$ CRYSTAL_PATH=$(pwd)/src \
   CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
   CC="$ARM_LLVM_ROOT/bin/clang" \
   LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib $(pwd)/libgc.a $(pwd)/libcrystal.a" \
   CFLAGS="--target aarch-apple-darwin" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig"
   spec/std_spec

Both sets of specs have been validated as passing, but please also try on your own hardware too if you cross-compile to validate.

The cross-compilation experience leaves a bit to be desired, unfortunately. I'm working on a separate PR to make it a bit nicer to cross-compile this all on one machine to avoid having to do so many gymnastics to compile for multiple targets.

Many thanks to @RomainFranceschini for kicking off this work and doing all the discovery, and @alin23 for validating it on their hardware as well

[maxfierke]

FWIW Linux CI / test_alpine (push) seems to be failing on master as well. This PR shouldn't have any had any effect on that.

[bcardiff]

Is there something that should be changed regarding #10066 (comment) ?

If I read things right src/fiber/context/aarch64.cr is in good shape regarding apple platform divergences, and the rest I guess are handled by llvm. Am I right?

[maxfierke]

Is there something that should be changed regarding #10066 (comment) ?

If I read things right src/fiber/context/aarch64.cr is in good shape regarding apple platform divergences, and the rest I guess are handled by llvm. Am I right?

Yeah, I think fiber switching is in good shape (we don't touch x18 and we always populate x29). I would expect that LLVM handles much of the rest, though I'm not sure if there's anything we might need to do for the argument passing subtleties. I'll try to take a closer look tonight. FWIW all the tests pass, including for ABI, so I would expect that what's there is mostly correct, at least for Crystal's purposes.

[maxfierke]

Looked into the details a bit more, and I think we're okay on the ABI: #10066 (comment)

Made one alteration to the type of LibC::VaList on aarch64-darwin, according to the Apple documentation

[jhass]

Looks good to me, we don't have to have first tier support on the first try :)

Anyone already found a CI solution? Maybe there's a Linux distribution compiling against the Apple ABI that we can use for now on our AArch64 box or something like that?

[Sija]

Lil' bit more on the topic via GHA issue actions/runner#805

[bcardiff]

👍