Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move links key from pyo3 to pyo3-ffi #2224

Merged
merged 2 commits into from Mar 15, 2022
Merged

Move links key from pyo3 to pyo3-ffi #2224

merged 2 commits into from Mar 15, 2022

Conversation

adamreichold
Copy link
Member

This ensures that its build script has finished before pyo3's build script starts. The extra dependency edge enables reliably passing configuration files like pyo3-cross-compile-config.txt created by pyo3-ffi's build script to the one of pyo3.

Closes #2222

…ipt has finished before pyo3's build script starts.
@davidhewitt
Copy link
Member

I don't think this will necessarily help? The problem is that this PR will cause pyo3-ffi will be built twice, once for the host (as a build dependency) and once for the final target.

Even if the build script runs for the host build, it won't get the right information from cargo to detect cross compiles.

@adamreichold
Copy link
Member Author

Even if the build script runs for the host build, it won't get the right information from cargo to detect cross compiles.

I it is indeed built twice but the (useless) version for the host does not write the cross-comilation-specific file, does it?

In any case, I think this appears to fix things because it seems to ensure that both build script builds of pyo3-ffi before the build script for pyo3 starts, but that could also be accidental scheduling by Cargo.

Log of building `pyo3-ffi` and `pyo3` with this change
   Compiling pyo3-ffi v0.16.1 (/home/adam/Projekte/pyo3/pyo3-ffi)
     Running `CARGO=/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME=build_script_build CARGO_MANIFEST_DIR=/home/adam/Projekte/pyo3/pyo3-ffi CARGO_PKG_AUTHORS='PyO3 Project and Contributors <https://github.com/PyO3>' CARGO_PKG_DESCRIPTION='Python-API bindings for the PyO3 ecosystem' CARGO_PKG_HOMEPAGE='https://github.com/pyo3/pyo3' CARGO_PKG_LICENSE=Apache-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=pyo3-ffi CARGO_PKG_REPOSITORY='https://github.com/pyo3/pyo3' CARGO_PKG_VERSION=0.16.1 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=16 CARGO_PKG_VERSION_PATCH=1 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/adam/Projekte/jaccard-agreement/target/debug/deps:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' rustc --crate-name build_script_build --edition=2018 /home/adam/Projekte/pyo3/pyo3-ffi/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="abi3"' --cfg 'feature="abi3-py310"' --cfg 'feature="abi3-py38"' --cfg 'feature="abi3-py39"' --cfg 'feature="default"' --cfg 'feature="extension-module"' -C metadata=91405922c2dfe107 -C extra-filename=-91405922c2dfe107 --out-dir /home/adam/Projekte/jaccard-agreement/target/debug/build/pyo3-ffi-91405922c2dfe107 -C incremental=/home/adam/Projekte/jaccard-agreement/target/debug/incremental -L dependency=/home/adam/Projekte/jaccard-agreement/target/debug/deps --extern pyo3_build_config=/home/adam/Projekte/jaccard-agreement/target/debug/deps/libpyo3_build_config-61d95692dfeb027a.rlib`
     Running `/home/adam/Projekte/jaccard-agreement/target/debug/build/pyo3-ffi-91405922c2dfe107/build-script-build`
     Running `/home/adam/Projekte/jaccard-agreement/target/debug/build/pyo3-ffi-91405922c2dfe107/build-script-build`
[pyo3-ffi 0.16.1] [/home/adam/Projekte/pyo3/pyo3-build-config/src/impl_.rs:582] &host = "x86_64-unknown-linux-gnu"
[pyo3-ffi 0.16.1] [/home/adam/Projekte/pyo3/pyo3-build-config/src/impl_.rs:582] &target = "x86_64-unknown-linux-gnu"
[pyo3-ffi 0.16.1] [/home/adam/Projekte/pyo3/pyo3-build-config/src/impl_.rs:582] &host = "x86_64-unknown-linux-gnu"
[pyo3-ffi 0.16.1] [/home/adam/Projekte/pyo3/pyo3-build-config/src/impl_.rs:582] &target = "x86_64-pc-windows-gnu"
[pyo3-ffi 0.16.1] cargo:rerun-if-env-changed=PYO3_CROSS
[pyo3-ffi 0.16.1] cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
[pyo3-ffi 0.16.1] cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
[pyo3-ffi 0.16.1] cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_3_6
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_3_7
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_3_8
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_LIMITED_API
[pyo3-ffi 0.16.1] cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
[pyo3-ffi 0.16.1] cargo:rustc-link-lib=pythonXY:python3
[pyo3-ffi 0.16.1] cargo:rustc-link-search=native=/home/adam/Projekte/jaccard-agreement/libs
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_3_6
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_3_7
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_3_8
[pyo3-ffi 0.16.1] cargo:rustc-cfg=Py_LIMITED_API
[pyo3-ffi 0.16.1] cargo:rustc-cfg=min_const_generics
[pyo3-ffi 0.16.1] cargo:rustc-cfg=addr_of
     Running `CARGO=/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME=pyo3_ffi CARGO_MANIFEST_DIR=/home/adam/Projekte/pyo3/pyo3-ffi CARGO_PKG_AUTHORS='PyO3 Project and Contributors <https://github.com/PyO3>' CARGO_PKG_DESCRIPTION='Python-API bindings for the PyO3 ecosystem' CARGO_PKG_HOMEPAGE='https://github.com/pyo3/pyo3' CARGO_PKG_LICENSE=Apache-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=pyo3-ffi CARGO_PKG_REPOSITORY='https://github.com/pyo3/pyo3' CARGO_PKG_VERSION=0.16.1 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=16 CARGO_PKG_VERSION_PATCH=1 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/adam/Projekte/jaccard-agreement/target/debug/deps:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' OUT_DIR=/home/adam/Projekte/jaccard-agreement/target/debug/build/pyo3-ffi-2caf4940af618652/out rustc --crate-name pyo3_ffi --edition=2018 /home/adam/Projekte/pyo3/pyo3-ffi/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="abi3"' --cfg 'feature="abi3-py310"' --cfg 'feature="abi3-py38"' --cfg 'feature="abi3-py39"' --cfg 'feature="default"' --cfg 'feature="extension-module"' -C metadata=4fff5a899f77bf02 -C extra-filename=-4fff5a899f77bf02 --out-dir /home/adam/Projekte/jaccard-agreement/target/debug/deps -C incremental=/home/adam/Projekte/jaccard-agreement/target/debug/incremental -L dependency=/home/adam/Projekte/jaccard-agreement/target/debug/deps --extern libc=/home/adam/Projekte/jaccard-agreement/target/debug/deps/liblibc-d017bdaca5bf0cd4.rmeta --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3_8 --cfg Py_LIMITED_API --cfg min_const_generics --cfg addr_of`
[pyo3-ffi 0.16.1] cargo:rustc-cfg=min_const_generics
[pyo3-ffi 0.16.1] cargo:rustc-cfg=addr_of
     Running `CARGO=/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME=pyo3_ffi CARGO_MANIFEST_DIR=/home/adam/Projekte/pyo3/pyo3-ffi CARGO_PKG_AUTHORS='PyO3 Project and Contributors <https://github.com/PyO3>' CARGO_PKG_DESCRIPTION='Python-API bindings for the PyO3 ecosystem' CARGO_PKG_HOMEPAGE='https://github.com/pyo3/pyo3' CARGO_PKG_LICENSE=Apache-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=pyo3-ffi CARGO_PKG_REPOSITORY='https://github.com/pyo3/pyo3' CARGO_PKG_VERSION=0.16.1 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=16 CARGO_PKG_VERSION_PATCH=1 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/adam/Projekte/jaccard-agreement/target/debug/deps:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' OUT_DIR=/home/adam/Projekte/jaccard-agreement/target/x86_64-pc-windows-gnu/debug/build/pyo3-ffi-ec24e07e9c71c12f/out rustc --crate-name pyo3_ffi --edition=2018 /home/adam/Projekte/pyo3/pyo3-ffi/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="abi3"' --cfg 'feature="abi3-py310"' --cfg 'feature="abi3-py38"' --cfg 'feature="abi3-py39"' --cfg 'feature="default"' --cfg 'feature="extension-module"' -C metadata=2669cd6950faacd9 -C extra-filename=-2669cd6950faacd9 --out-dir /home/adam/Projekte/jaccard-agreement/target/x86_64-pc-windows-gnu/debug/deps --target x86_64-pc-windows-gnu -C incremental=/home/adam/Projekte/jaccard-agreement/target/x86_64-pc-windows-gnu/debug/incremental -L dependency=/home/adam/Projekte/jaccard-agreement/target/x86_64-pc-windows-gnu/debug/deps -L dependency=/home/adam/Projekte/jaccard-agreement/target/debug/deps --extern libc=/home/adam/Projekte/jaccard-agreement/target/x86_64-pc-windows-gnu/debug/deps/liblibc-7f8608fbd9b1de2c.rmeta -L native=/home/adam/Projekte/jaccard-agreement/libs --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3_8 --cfg Py_LIMITED_API --cfg min_const_generics --cfg addr_of -l 'pythonXY:python3'`
   Compiling pyo3 v0.16.1 (/home/adam/Projekte/pyo3)
     Running `CARGO=/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME=build_script_build CARGO_MANIFEST_DIR=/home/adam/Projekte/pyo3 CARGO_PKG_AUTHORS='PyO3 Project and Contributors <https://github.com/PyO3>' CARGO_PKG_DESCRIPTION='Bindings to Python interpreter' CARGO_PKG_HOMEPAGE='https://github.com/pyo3/pyo3' CARGO_PKG_LICENSE=Apache-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=pyo3 CARGO_PKG_REPOSITORY='https://github.com/pyo3/pyo3' CARGO_PKG_VERSION=0.16.1 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=16 CARGO_PKG_VERSION_PATCH=1 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/adam/Projekte/jaccard-agreement/target/debug/deps:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/adam/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' rustc --crate-name build_script_build --edition=2018 /home/adam/Projekte/pyo3/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="abi3"' --cfg 'feature="abi3-py310"' --cfg 'feature="abi3-py38"' --cfg 'feature="abi3-py39"' --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="indoc"' --cfg 'feature="macros"' --cfg 'feature="pyo3-macros"' --cfg 'feature="pyproto"' --cfg 'feature="unindent"' -C metadata=09d5c0e5572ec4c5 -C extra-filename=-09d5c0e5572ec4c5 --out-dir /home/adam/Projekte/jaccard-agreement/target/debug/build/pyo3-09d5c0e5572ec4c5 -C incremental=/home/adam/Projekte/jaccard-agreement/target/debug/incremental -L dependency=/home/adam/Projekte/jaccard-agreement/target/debug/deps --extern pyo3_build_config=/home/adam/Projekte/jaccard-agreement/target/debug/deps/libpyo3_build_config-61d95692dfeb027a.rlib --extern pyo3_ffi=/home/adam/Projekte/jaccard-agreement/target/debug/deps/libpyo3_ffi-4fff5a899f77bf02.rlib`
     Running `/home/adam/Projekte/jaccard-agreement/target/debug/build/pyo3-09d5c0e5572ec4c5/build-script-build`
[pyo3 0.16.1] cargo:rustc-cfg=Py_3_6
[pyo3 0.16.1] cargo:rustc-cfg=Py_3_7
[pyo3 0.16.1] cargo:rustc-cfg=Py_3_8
[pyo3 0.16.1] cargo:rustc-cfg=Py_LIMITED_API

@adamreichold
Copy link
Member Author

So the CI failure show that this is a bad idea in any case as the host build of pyo3-ffi might fail for unrelated reasons. Sorry for the noise...

Instead of a fake build dependency, move the links key from pyo3 to pyo3-ffi
which according to [1] is a prerequisite for build scripts to emit meta-data
that can be used by other build scripts which would incidentally ensure the
ordering between the build script of pyo3 and pyo3-ffi that we need.

[1] https://doc.rust-lang.org/cargo/reference/build-script-examples.html#using-another-sys-crate
@adamreichold
Copy link
Member Author

adamreichold commented Mar 15, 2022

Another try: Instead of a fake build dependency, move the links key from pyo3 to pyo3-ffi which according to the Cargo book is a prerequisite for build scripts to emit meta-data that can be used by other build scripts which would incidentally ensure the ordering between the build script of pyo3 and pyo3-ffi that we need.

(I think the links key would actually belong to pyo3-ffi in principle as well since that is our equivalent of a *-sys crate.)

@davidhewitt
Copy link
Member

That sounds likely to be a correct fix to me!

@adamreichold

This comment was marked as resolved.

@adamreichold

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@adamreichold

This comment was marked as resolved.

@adamreichold adamreichold changed the title Make pyo3-ffi a build dependency of pyo3 Move links key from pyo3 to pyo3-ffi Mar 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cross-compilation to Windows randomly fails after upgrade to 0.16
3 participants