Static build support (openssl + cross-compile)

[polarathene]

Summary

Static build support appears to be a bit broken?

While troubleshooting a bug encountered from a downstream project, I discovered it was related to hickory-dns when enabling the dnssec-ring feature. During this I noticed hickory-dns fails to succeed at producing a static build on a -gnu build host (including when cross-compiling for a -musl target).

Reproduction

# Builds successfully:
$ RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-gnu

# With dnssec-ring feature fails with errors:
RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-gnu --features dnssec-ring

Compiling hickory-dns v0.24.1 (/tmp/dns/bin)
error: linking with `cc` failed: exit status: 1

# ...

  = note: /usr/bin/ld: cannot find -lssl: No such file or directory
          /usr/bin/ld: cannot find -lcrypto: No such file or directory
          collect2: error: ld returned 1 exit status

# Little bit more informative when additionally built with `OPENSSL_DIR=/usr`:
$ OPENSSL_DIR=/usr RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-gnu --features dnssec-ring

   Compiling hickory-util v0.24.1 (/tmp/dns/util)
error: linking with `cc` failed: exit status: 1

# ...

  = note: /usr/bin/ld: /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d2ef02247056996e.rlib(std-d2ef02247056996e.std.e4dfbc2c3f4b09f1-cgu.0.rcgu.o): in function `<std::sys_common::net::LookupHost as core::convert::TryFrom<(&str,u16)>>::try_from::{{closure}}':
          /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/net.rs:207:(.text._ZN104_$LT$std..sys_common..net..LookupHost$u20$as$u20$core..convert..TryFrom$LT$$LP$$RF$str$C$u16$RP$$GT$$GT$8try_from28_$u7b$$u7b$closure$u7d$$u7d$17h31ae8f29b0ed096aE+0x5b): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
          /usr/bin/ld: /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d2ef02247056996e.rlib(std-d2ef02247056996e.std.e4dfbc2c3f4b09f1-cgu.0.rcgu.o): in function `std::sys::pal::unix::os::home_dir::fallback':
          /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/os.rs:732:(.text._ZN3std3env8home_dir17h13958f3bcb4836b9E+0x109): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
          /usr/bin/ld: /tmp/dns/target/x86_64-unknown-linux-gnu/debug/deps/libhickory_util-e875a01acb16e7d1.rlib(hickory_util-e875a01acb16e7d1.5aac1r0u63bssn38.rcgu.o): undefined reference to symbol '__tls_get_addr@@GLIBC_2.3'
          /usr/bin/ld: /usr/lib64/ld-linux-x86-64.so.2: error adding symbols: DSO missing from command line
          collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-link-lib)

error: could not compile `hickory-util` (bin "dns") due to 1 previous error
error: could not compile `hickory-util` (bin "recurse") due to 1 previous error
error: could not compile `hickory-util` (bin "resolve") due to 1 previous error

This is understandable for a glibc "static" build, but is still the same output when attempting to build for a musl target. This is only when OPENSSL_DIR was specified, so presumably the glibc symbols and related warnings above with the musl target are due to the build attempting to bundle/link the glibc openssl on the build host?

# With zigbuild, I can successfully cross-compile to musl static build without the above failure:
OPENSSL_DIR=/usr RUSTFLAGS="-C target-feature=+crt-static" cargo zigbuild --target x86_64-unknown-linux-musl --features dnssec-ring

# However they're not valid binaries:
$ ldd target/x86_64-unknown-linux-musl/debug/{dns,resolve,recurse,hickory-dns}
target/x86_64-unknown-linux-musl/debug/dns: error while loading shared libraries: /lib64/libc.so: invalid ELF header
target/x86_64-unknown-linux-musl/debug/resolve: error while loading shared libraries: /lib64/libc.so: invalid ELF header
target/x86_64-unknown-linux-musl/debug/recurse: error while loading shared libraries: /lib64/libc.so: invalid ELF header
target/x86_64-unknown-linux-musl/debug/hickory-dns: error while loading shared libraries: /lib64/libc.so: invalid ELF header

With both cargo build and cargo zigbuild (without the OPENSSL_DIR ENV present), the build failure for a musl target is more obvious (note: cargo zigbuild cannot perform -gnu target static builds):

  run pkg_config fail: pkg-config has not been configured to support cross-compilation.

  Install a sysroot for the target platform and configure it via
  PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH, or install a
  cross-compiling wrapper for pkg-config and set it via
  PKG_CONFIG environment variable.

  --- stderr
  thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.102/build/find_normal.rs:190:5:


  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

  $HOST = x86_64-unknown-linux-gnu
  $TARGET = x86_64-unknown-linux-musl
  openssl-sys = 0.9.102

Despite having the development packages required, and using the OPENSSL_DIR or other supported ENVs from the openssl crate documentation, these are not helpful to support cross-compiling to musl target. Possibly due to a long standing issue with pkg-config that may be environment specific? 🤷‍♂️

Reproduction environment (Docker)

If it's helpful, the above can be reproduced with docker and the following commands:

# Create reproduction environment:
docker run --rm -it --workdir /tmp fedora:40 bash

# Prep environment (Fedora):
dnf install -y --setopt=install_weak_deps=0 git-core gcc musl-gcc perl pip rustup
rustup-init -y --profile minimal --target x86_64-unknown-linux-musl --default-toolchain stable
source "$HOME/.cargo/env"

# Optional - Install cargo-zigbuild + zig:
# (NOTE: musl-gcc dep is not needed if cross-compiling via `cargo zigbuild` command)
pip install ziglang
cargo install cargo-zigbuild

# Building will fail as the enabled feature is not compatible with static builds:
git clone https://github.com/hickory-dns/hickory-dns .
RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-musl --features dnssec-ring

# NOTE: No apparent package to provide `libssl.a` like Ubuntu/Alpine have

# Alternative environment (same failure)
docker run --rm -it --workdir /tmp ubuntu:24.04 bash

# Prep environment (Ubuntu):
export DEBIAN_FRONTEND=noninteractive
apt update && apt install -y --no-install-recommends \
    git gcc g++ make musl-tools \
    python3-venv rustup ca-certificates
rustup toolchain install --profile minimal --target x86_64-unknown-linux-musl -- stable

# Optional - Install cargo-zigbuild + zig (ubuntu makes a venv for pip mandatory):
# (NOTE: musl-tools dep is not needed if cross-compiling via `cargo zigbuild` command)
python3 -m venv /opt/.venv
export PATH="/opt/.venv/bin:${PATH}"
pip install ziglang
cargo install cargo-zigbuild

# Building will fail as the enabled feature is not compatible with static builds for a different target triple than the host:
git clone https://github.com/hickory-dns/hickory-dns .
RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-musl --features dnssec-ring

# A static build for the host triple (x86_64-unknown-linux-gnu) is possible with the `libssl-dev` package:
OPENSSL_STATIC=1 OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu OPENSSL_INCLUDE_DIR=/usr/include/openssl \
RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-gnu --features dnssec-ring

Both should be capable of building statically linked musl target.

Whereas on a musl build host to avoid cross-compiling it will be successful as it can use static libs:

# NOTE: `rust:alpine` could alternatively be used (instead of getting rust via rustup):
docker run --rm -it --workdir /tmp alpine ash

# Prep environment (Alpine):
# NOTE: The last two openssl deps allow to avoid building openssl (thus perl is not be required) as they satisfy `OPENSSL_STATIC=1`.
apk add --no-cache git gcc musl-dev perl rustup openssl-dev openssl-libs-static
rustup-init -y --profile minimal && source "$HOME/.cargo/env"

# Building will succeed with the enabled feature when the required openssl packages are present for a static build:
# NOTE: Without those packages it will fail like the other two build environments.
# NOTE: Alpine lacks support for cross-compiling static linked glibc build with openssl.
git clone https://github.com/hickory-dns/hickory-dns .
RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-musl --features dnssec-ring

Expected outcome

It should be possible to build statically, at least for cross-compiling to a musl target?

Details

System:

• OS: Fedora (Docker container running via WSL2 on Windows 11 host)
• Architecture: x86_64
• Version 40
• rustc version: 1.78

Version:
Crate: hickory-util
Version: v0.24.1

[polarathene]

This can be resolved by adjusting the openssl dep to use the vendored feature, which will build OpenSSL from source:

hickory-dns/Cargo.toml

Lines 61 to 63 in cffc3fa

	# ssl
	native-tls = "0.2"
	openssl = "0.10.55"

openssl = { version = "0.10.55", features = ["vendored"] }

The above reproduction environments in all three containers can then successfully build the project.

It is then possible to opt-out via OPENSSL_NO_VENDOR=1. AFAIK, there doesn't appear to be support for opt-in via ENV, hence I don't think there is no nice workaround available? (should be possible via editing a downstream project's Cargo.toml to override this deps openssl features.. but that seems a little brittle?)

As long as something in a downstream project opts in to vendored feature elsewhere, it also seems to have the same effect of building vendored, so it's not a feature you can control with explicit opt-out on a crate I think? Only at build time via the mentioned opt-out ENV?

I assume a potential drawback/side-effect for non-static builds is that vendoring in your own build of openssl into the binary instead of dynamically linking to openssl?

[bluejekyll]

Sounds like this is specific to openssl. Do you need Openssl? We've been trying to direct people to use the ring and rustls defaults.

[polarathene]

Do you need Openssl?

Not personally no. It was just an observation when I was trying to identify why cargo-binstall was failing to resolve DNS while other programs had no issue. During that troubleshooting I tried both openssl and rustls but it didn't seem specific to either AFAIK.

What I did find was their trust-dns / hickory-dns feature (old project name is a feature alias) was the culprit. Specifically when the hickory-dns/dnssec-ring feature is enabled. As reported above, when I tried building this crate with that feature as a static build and with openssl (default) this failed. I looked into it and the only way for static build with openssl to work reliably across environments seems to require vendored to build from source, especially for cross-compilation to other targets.

Despite addressing that concern, the resolve utility was still working successfully like a dynamic linked build. I'm not sure what is different between that CLI and the cargo-binstall project, perhaps I wasn't actually using the feature via the resolve CLI just by enabling it at build 😓

I may try investigate that further to track down if the downstream failure encountered with cargo-binstall is reproducible with resolve CLI and then if that's specific to the ring crate, but I'm not sure if I'll have the time with my inexperience across these crates.

---

We've been trying to direct people to use the ring and rustls defaults.

Perhaps consider switching to those as the defaults in the future? If you do not think this concern with openssl and static build support is worth addressing, no worries, I just wanted to report my findings for others to benefit 👍

Feel free to close this issue if you like :)

[bluejekyll]

Thank you for spending the time on this, I appreciate that.

[PikuZheng]

Since Quinn 0.11 has been released with ring 0.17, I think the factors preventing static compilation have been resolved.

[djc]

Well, only after #2107 (or something like it) has been merged (see also #2206).