How to setup for cross compilation in embed mode

[Sytten]

Hi! I am trying to embed the python interpreter in our rust application. We cross compile from linux x86_64 to aarch64, armhf and windows. We use the cross-rs project for that and their pre-built images. I get that we will need to build our own images now to include the python binary, but I am wondering where do people usually get the binaries for the targeted system?
I was thinking we could copy from another docker image using a multi-stage build. Just not sure what is needed exactly, the documentation on it is a bit light. Any idea is appreciated!

[messense]

You can download from https://github.com/indygreg/python-build-standalone/releases

[Sytten]

Thanks!

[Sytten]

I built a small script:

#!/usr/bin/env bash

echo "[*] Install host python"
apt-get update && apt-get -y install python3.8

echo "[*] Prepare sysroot"
mkdir -p /opt/sysroot/usr/lib

echo "[*] Install target python"
curl -L https://github.com/indygreg/python-build-standalone/releases/download/20230116/cpython-3.8.16+20230116-aarch64-unknown-linux-gnu-install_only.tar.gz --output distribution.tar.gz
tar -xvf distribution.tar.gz && rm distribution.tar.gz
mv ./python/lib/* /opt/sysroot/usr/lib/ && rm -rf ./python

and setup cross

[build]
pre-build = "./scripts/cross.sh"
[build.env]
passthrough = [
  "PYO3_PYTHON=/usr/bin/python3.8",
  "PYO3_CROSS=1",
  "PYO3_CROSS_PYTHON_VERSION=3.8",
  "PYO3_CROSS_LIB_DIR=/opt/sysroot/usr/lib",
]

[target.aarch64-unknown-linux-gnu]
image = "ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main"

But I get a linking error: /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: cannot find -lpython3.8
Looks like the lib path is not passed to the linker.

[Sytten]

By forcing the RUSTFLAGS=-L /opt/sysroot/usr/lib, I got further but I hit another issue:

note: /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: warning: libcrypt.so.1, needed by /opt/sysroot/usr/lib/libpython3.8.so, not found (try using -rpath or -rpath-link)
          /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /opt/sysroot/usr/lib/libpython3.8.so: undefined reference to `crypt_r@GLIBC_2.17'

That seems to be known here indygreg/python-build-standalone#113

[Sytten]

I used a different approach that seems to be working. I decided to use the python docker image instead and a multistage build:

FROM --platform=linux/arm64/v8 python:3.8.16 AS python

FROM ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main

RUN apt-get update && apt-get -y install python3.8

COPY --from=python /usr/local/lib/ /opt/sysroot/usr/lib/

I had to keep the RUSTFLAGS though.

[Sytten]

For windows though that method doesn't work since docker won't let you pull window images.
I use the standalone builds there:

FROM ghcr.io/cross-rs/x86_64-pc-windows-gnu:main

RUN apt-get update && apt-get -y install python3.8
RUN curl -L https://github.com/indygreg/python-build-standalone/releases/download/20230116/cpython-3.8.16+20230116-x86_64-pc-windows-msvc-shared-install_only.tar.gz --output distribution.tar.gz && \
    tar -xvf distribution.tar.gz && \
    rm distribution.tar.gz && \
    mkdir -p /opt/sysroot/usr/ && \
    mv ./python /opt/sysroot/usr/lib