Rust support #27

tonistiigi · 2021-08-30T04:48:58Z

Allow building rust projects. Support installing rust via rustup (alpine/debian) and alpine packages (rust/rust-stdlib). Haven't tested official image. Debian package-based installs will probably not be supported as packages don't seem to be properly separated.

Currently have tested with these Dockerfile stages:

ARG RUST_ALG=alpine-rustup

FROM --platform=$BUILDPLATFORM alpine AS rust-alpine-rustup
COPY --from=xx / /
RUN apk add clang lld musl-dev gcc git curl
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
ENV XX_VENDOR=unknown
RUN xx-clang --setup-target-triple
RUN rustup target add $(xx-info)
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM alpine:edge AS rust-apk
COPY --from=xx / /
RUN apk add clang lld musl-dev gcc git rust cargo
WORKDIR /work
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc rust-stdlib
RUN xx-clang --setup-target-triple
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="-C target-feature=+crt-static" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake
# alpine rust disables static linking? https://github.com/alpinelinux/aports/commit/69851bdae1177246337f51a35734e93f1fd7e3d3#diff-92eb9787919d42fb458bdae3211d4ee3d18385106441d644ec49b10ec3dfaa8c

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-rustup
COPY --from=xx / /
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
ENV XX_VENDOR=unknown
RUN xx-clang --setup-target-triple
RUN rustup target add $(xx-info)
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="-C target-feature=+crt-static" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM rust-${RUST_ALG} as rust-build

FROM scratch AS b3sum
COPY --from=rust-build /out/b3sum /

The main complexity is that the binary releases use triple with unknown vendor instead of alpine and as C compiler always gets called with cc --target=wrong-triple it doesn't really matter that the CC is wrapped. I've added a way for cc-clang to work with any vendor by creating symlinks to the official ones. An alternative would be to try to rename the --target flag in the wrapper.

I think it makes sense to add xx-cargo script that would set the 3 env automatically. xx-cargo --setup-target could make the correct calls to rustup target add or xx-apk add rust-stdlib (and set up xx-clang for correct triple).

Risc-V doesn't currently work because the gnu triple is riscv64gc-* not riscv64. Might need some special override for that. In alpine:edge there is no riscv package for rust-stdlib yet, and no musl build in rustup either.

add xx-cargo
support riscv64gc-gnu triple
CI testing

@crazy-max

tonistiigi · 2021-09-01T06:53:43Z

Added xx-cargo and riscv support. Works with both Alpine and Debian with either rustup or package based installs. RiscV only works on debian, musl support is blocked in rust-lang/libc#1994

Test stages with xx-cargo

ARG RUST_ALG=alpine-rustup

FROM --platform=$BUILDPLATFORM alpine AS blake
RUN apk add git
WORKDIR /src
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git

FROM --platform=$BUILDPLATFORM alpine AS rust-alpine-rustup
RUN apk add clang lld musl-dev gcc git curl
COPY --from=xx / /
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
RUN --mount=from=blake,src=/src/BLAKE3,rw cd b3sum && \
    xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM alpine:edge AS rust-apk
RUN apk add clang lld musl-dev gcc git rust cargo
COPY --from=xx / /
WORKDIR /work
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
RUN xx-cargo --setup-target-triple # optional
RUN --mount=from=blake,src=/src/BLAKE3,rw \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake
# alpine rust disables static linking? https://github.com/alpinelinux/aports/commit/69851bdae1177246337f51a35734e93f1fd7e3d3#diff-92eb9787919d42fb458bdae3211d4ee3d18385106441d644ec49b10ec3dfaa8c

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-rustup
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl
WORKDIR /work
COPY --from=xx / /
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
RUN xx-cargo --setup-target-triple # optional
RUN --mount=from=blake,src=/src/BLAKE3,rw \
    export XX_RUSTFLAGS="-C target-feature=+crt-static" && \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && ls -lR /tmp/blake && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-apt
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl cargo
WORKDIR /work
COPY --from=xx / /
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
RUN --mount=from=blake,src=/src/BLAKE3,rw git checkout 0.3.7 && \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake

FROM rust-${RUST_ALG} as rust-build

FROM scratch AS b3sum
COPY --from=rust-build /out/b3sum /

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

crazy-max · 2022-09-21T20:35:46Z

musl support is blocked in rust-lang/libc#1994

@tonistiigi Looks like it has been merged: rust-lang/libc#2537

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Using xx-cargo is optional. Alternative is to install stdlib with xx-apt/xx-apk or with rustup with XX_VENDOR=unknown xx-info. Then CC, and _LINKER needs to be defined via xx-info. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi · 2022-09-23T18:29:05Z

@crazy-max So I think the last commit needs to be reverted to make CI work. Or do you have additional analysis?

crazy-max · 2022-09-23T18:54:32Z

@tonistiigi I looked across GitHub for XX_VENDOR or xx-info env but didn't find any use out there except here when dealing with xx-go, xx-apt, xx-apk and xx-verify.

I'm a bit worried about this case:

xx/base/xx-apt

Line 38 in 5596639

case "${XX_VENDOR}" in

I think we could just use case "$(xx-info vendor)" in.

tonistiigi · 2022-09-23T18:56:33Z

Do you understand what is causing the CI failure. Maybe there is a solution that would allow keeping that line. Having XX_VENDOR and xx-info vendor return different values doesn't make a lot of sense.

crazy-max · 2022-09-23T20:12:15Z

Do you understand what is causing the CI failure.

Yes it fails with xx-go because triple is wrong when XX_VENDOR is being exported in:

xx/base/xx-go

Line 6 in 5596639

export "${l?}"

tonistiigi · 2022-09-23T21:23:45Z

That file does not use XX_VENDOR afaics. So the issue needs to be already in xx-info not it xx-go

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

tonistiigi mentioned this pull request Aug 30, 2021

Going further denzp/cargo-wharf#34

Open

tonistiigi force-pushed the rust-support branch from 7564e6c to b7528bc Compare August 30, 2021 06:48

tonistiigi added this to the 1.1 milestone Oct 10, 2021

tonistiigi modified the milestones: 1.1, 1.2 Nov 29, 2021

Allow custom vendor to support unknown triple in rust

ada701d

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

crazy-max mentioned this pull request Sep 22, 2022

Rust support rebased #76

Closed

tonistiigi added 3 commits September 22, 2022 13:12

apk: add special detection for rust library installs

50e54fa

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

allow setting riscv64 arch to riscv64gc for rust support

ab1effd

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

add xx-cargo to simplify rust building

3e9dd7f

Using xx-cargo is optional. Alternative is to install stdlib with xx-apt/xx-apk or with rustup with XX_VENDOR=unknown xx-info. Then CC, and _LINKER needs to be defined via xx-info. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

rumpl mentioned this pull request Sep 22, 2022

Add a Dockerfile for cross-compilation second-state/runwasi#1

Merged

crazy-max force-pushed the rust-support branch from 132a063 to 2543538 Compare September 22, 2022 17:17

crazy-max marked this pull request as ready for review September 22, 2022 17:17

crazy-max self-requested a review September 22, 2022 17:17

crazy-max force-pushed the rust-support branch from 8d23e49 to 9e46daa Compare September 23, 2022 19:37

crazy-max force-pushed the rust-support branch from 9e46daa to ea9cf16 Compare September 27, 2022 12:19

xx-info: fix vendor

b8dde93

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max force-pushed the rust-support branch from ea9cf16 to b8dde93 Compare September 27, 2022 12:22

crazy-max approved these changes Sep 30, 2022

View reviewed changes

tonistiigi merged commit dad71a2 into master Sep 30, 2022

crazy-max deleted the rust-support branch September 30, 2022 23:59

crazy-max mentioned this pull request Oct 14, 2022

Add tests and documentation for rust support #79

Closed

crazy-max mentioned this pull request Oct 30, 2022

xx-cargo: add tests and docs #83

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rust support #27

Rust support #27

tonistiigi commented Aug 30, 2021 •

edited

tonistiigi commented Sep 1, 2021

crazy-max commented Sep 21, 2022

tonistiigi commented Sep 23, 2022

crazy-max commented Sep 23, 2022

tonistiigi commented Sep 23, 2022

crazy-max commented Sep 23, 2022 •

edited

tonistiigi commented Sep 23, 2022

Rust support #27

Rust support #27

Conversation

tonistiigi commented Aug 30, 2021 • edited

tonistiigi commented Sep 1, 2021

crazy-max commented Sep 21, 2022

tonistiigi commented Sep 23, 2022

crazy-max commented Sep 23, 2022

tonistiigi commented Sep 23, 2022

crazy-max commented Sep 23, 2022 • edited

tonistiigi commented Sep 23, 2022

tonistiigi commented Aug 30, 2021 •

edited

crazy-max commented Sep 23, 2022 •

edited