pyo3-build-config: Add `python3-dll-a` crate support #2282

ravenexp · 2022-04-07T12:51:25Z

Automatically generate python3.dll import libraries
for Windows compile targets in the build script.

Adds a new PyO3 crate feature auto-abi3-import-lib enabling
automatic import library generation.

Closes #2231

WIP: The documentation is still a TODO

Cargo.toml

pyo3-build-config/src/util.rs

guide/src/building_and_distribution.md

adamreichold · 2022-04-08T09:43:52Z

guide/src/building_and_distribution.md

@@ -230,6 +234,14 @@ When cross-compiling, PyO3's build script cannot execute the target Python inter
 * `PYO3_CROSS_PYTHON_VERSION`: Major and minor version (e.g. 3.9) of the target Python installation. This variable is only needed if PyO3 cannot determine the version to target from `abi3-py3*` features, or if `PYO3_CROSS_LIB_DIR` is not set, or if there are multiple versions of Python present in `PYO3_CROSS_LIB_DIR`.
 * `PYO3_CROSS_PYTHON_IMPLEMENTATION`: Python implementation name ("CPython" or "PyPy") of the target Python installation. CPython is assumed by default when this variable is not set, unless `PYO3_CROSS_LIB_DIR` is set for a Unix-like target and PyO3 can get the interpreter configuration from `_sysconfigdata*.py`.

+An experimental `pyo3` crate feature `generate-abi3-import-lib` enables the user to cross compile
+"abi3" extension modules for Windows targets without setting `PYO3_CROSS_LIB_DIR` environment
+variable or providing any Windows Python library files. It uses external `python3-dll-a` crate


A link to the crate homepage might be a nice courtesy?

I'll add a link to docs.rs page. As for the GitHub project page, I actually hoped to transfer that project to the PyO3 organization after this feature is done. Unfortunately, there's a chance I might lose access to my GitHub or Gmail accounts because of the circumstances, and I don't want PyO3 to end up with an unmaintainable dependency.

I've added links to the docs.rs page to both the user guide and Architecture.md.

The changes look fine to me. Leaving this unresolved due to the question of long-term maintenance though.

I don't want to be misunderstood here. I do intend to maintain it and to contribute to PyO3 in the future, but I also want to safeguard against the situation where I'd be unable to do so. GitHub has blocked accounts of users from sanctioned countries before, and if that happens I don't want it to become a problem for the projects that depend on mine. That's why I want to transfer the GitHub project and the crates.io package ownership to an organization as a backup measure.

davidhewitt

Sorry for the delay on my part, thanks @adamreichold for reviewing.

Last thing that would be nice to see IMO would be a check for this in CI. I think it could probably be done similarly to the other cross compilation check -

pyo3/.github/workflows/ci.yml

Line 245 in 2a54641

- name: Test cross compilation

ravenexp · 2022-04-10T09:58:38Z

Last thing that would be nice to see IMO would be a check for this in CI. I think it could probably be done similarly to the other cross compilation check -

I'll try to figure out how to do it, but it'll probably take some time. I've only used GitLab CI before, and GitHub Actions are a complete mystery to me. Everybody just calls somebody else's actions, and those actions call somebody else's actions, and it's difficult for me to figure out where the resulting shell commands even come from...

davidhewitt · 2022-04-10T10:04:55Z

I think it'll be something like this:

      - name: Test windows python3-dll-a cross compilation
        if: ${{ matrix.platform.os == 'ubuntu-latest' && matrix.python-version == '3.9' }}
        uses: messense/maturin-action@v1
        with:
          target: x86_64-pc-windows-msvc
          args: --release -i python3.9 --no-sdist -m examples/maturin-starter/Cargo.toml --cargo-extra-args="--features=pyo3/generate-abi3-import-lib --features=pyo3/abi3"

(might need env: PYO3_NO_PYTHON too? Not sure if maturin will set it.)

ravenexp · 2022-04-10T10:12:37Z

(might need env: PYO3_NO_PYTHON too? Not sure if maturin will set it.)

That's the issue with using maturin: it has its own cross-compilation logic and abi3 bindings handling logic, which get special treatment. It used to set PYO3_NO_PYTHON and alter RUSTFLAGS accordingly, but apparently it does not do that anymore after PyO3/maturin#851 was merged.

I'd rather start with plain cargo build --target ... and proceed from there, if that's possible.

adamreichold · 2022-04-10T11:21:57Z

I'd rather start with plain cargo build --target ... and proceed from there, if that's possible.

While I understand aiming for simplicity to be able to follow things and understand errors, I also think that a CI job should be as close as possible to what actual usage will look like. Which I expect to go through Maturin most of the time (definitely, in the case where I would like to use this: building Windows wheels using MinGW in GitLab CI which I currently do by checking in the import libraries into source control). Or putting it differently, if Maturin's abi3 handling logic interferes with using this features, this might become a problem for the intended use cases which a Maturin-based CI job would only surface early (which seems a good thing to me).

davidhewitt · 2022-04-10T12:21:04Z

cc @messense - I'd be interested in your opinion on how we might be able to test this (if you get a chance to see this considering the situation in Shanghai, no worries if you are unable to comment atm)

ravenexp · 2022-04-10T12:39:41Z

While I understand aiming for simplicity to be able to follow things and understand errors, I also think that a CI job should be as close as possible to what actual usage will look like. Which I expect to go through Maturin most of the time (definitely, in the case where I would like to use this: building Windows wheels using MinGW in GitLab CI which I currently do by checking in the import libraries into source control).

Well, in my tests both cargo build and maturin build commands work (for the current version of maturin =0.12.12). I wouldn't submit this if they didn't. But, maturin's cross-compilation approach changes from version to version, and it is currently in the process of eliminating its internal PYO3_NO_PYTHON use, which will hopefully simplify things a lot.

Definitely, in the case where I would like to use this: building Windows wheels using MinGW in GitLab CI which I currently do by checking in the import libraries into source control).

This is my main use case as well, and this works with just maturin build --target x86_64-pc-windows-gnu.

My original point was that, preferably, both commands should be tested, because I've encountered the situations where one of them works and the other one doesn't, like PyO3/maturin#824 (comment). Solving issues like that is hard when no additional information is available.

adamreichold · 2022-04-10T12:50:35Z

My original point was that, preferably, both commands should be tested

This certainly sounds like a good idea if even more work to implement.

messense · 2022-04-10T12:53:19Z

I'll try to figure out how to do it, but it'll probably take some time. I've only used GitLab CI before, and GitHub Actions are a complete mystery to me.

I can work on adding tests to CI by pushing to your branch once this PR is rebased.

Automatically generate `python3.dll` import libraries for Windows compile targets in the build script. Adds a new PyO3 crate feature `generate-abi3-import-lib` enabling automatic import library generation. Closes PyO3#2231

Update the user guide to describe its applicability to the native and cross-compilation build scenarios.

ravenexp · 2022-04-10T13:58:02Z

I'll try to figure out how to do it, but it'll probably take some time. I've only used GitLab CI before, and GitHub Actions are a complete mystery to me.

I can work on adding tests to CI by pushing to your branch once this PR is rebased.

I've rebased the PR branch on the current main. This would be a huge help, thanks!

Cross and native compilation to *-pc-windows-gnu targets should work out of the box, and cross-compilation to *-pc-windows-msvc I've only tested with rust-lld as the linker. I don't know how to run link.exe on Linux.

*-msvc targets also need LLVM binutils for llvm-dlltool and the MSVC runtime import libraries to link, as usual.
I don't use the native MSVC toolchain myself, so I can't help much here.

feature is enabled

Switch from Python 3.10 to Python 3.8 because I don't want to test with Rust 1.48.0

.github/workflows/ci.yml

davidhewitt · 2022-04-11T05:44:08Z

guide/src/building_and_distribution.md

+On Unix-like systems this works unconditionally; on Windows you must also set the `RUSTFLAGS` environment variable
+to contain `-L native=/path/to/python/libs` so that the linker can find `python3.lib`.
+
+If the `python3.dll` import library is not available, an experimental `generate-abi3-import-lib` crate


Just noticed it would also be great to mention this in features.md

Added in 4347624

Update the feature list section of the user guide to include `generate-abi3-import-lib` description.

davidhewitt

LGTM! Thanks all!

ravenexp marked this pull request as draft April 7, 2022 12:56

adamreichold reviewed Apr 7, 2022

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

adamreichold reviewed Apr 7, 2022

View reviewed changes

pyo3-build-config/src/util.rs Outdated Show resolved Hide resolved

adamreichold reviewed Apr 7, 2022

View reviewed changes

pyo3-build-config/src/util.rs Outdated Show resolved Hide resolved

ravenexp force-pushed the python3-dll-a branch from 9f6ad96 to 4707c33 Compare April 8, 2022 07:10

ravenexp marked this pull request as ready for review April 8, 2022 09:30

ravenexp requested a review from adamreichold April 8, 2022 09:30

adamreichold reviewed Apr 8, 2022

View reviewed changes

guide/src/building_and_distribution.md Outdated Show resolved Hide resolved

adamreichold reviewed Apr 8, 2022

View reviewed changes

adamreichold approved these changes Apr 8, 2022

View reviewed changes

ravenexp force-pushed the python3-dll-a branch from 863390d to f7fe905 Compare April 8, 2022 11:07

davidhewitt approved these changes Apr 10, 2022

View reviewed changes

ravenexp added 3 commits April 10, 2022 16:45

pyo3-build-config: Add python3-dll-a crate support

479fe67

Automatically generate `python3.dll` import libraries for Windows compile targets in the build script. Adds a new PyO3 crate feature `generate-abi3-import-lib` enabling automatic import library generation. Closes PyO3#2231

Document generate-abi3-import-lib crate feature

a7c0de3

Update the user guide to describe its applicability to the native and cross-compilation build scenarios.

Add a ChangeLog entry for generate-abi3-import-lib

c84cecc

ravenexp force-pushed the python3-dll-a branch from 438fd74 to c84cecc Compare April 10, 2022 13:47

messense added 4 commits April 10, 2022 22:05

Add Windows abi3 wheel cross comiplation test

c5ca6d0

Enable generate-abi3-import-lib feature for maturin-starter when abi3

eb6b645

feature is enabled

maturin still requires a Python interpreter on Windows for now

8368c62

Switch from Python 3.10 to Python 3.8 because I don't want to test with Rust 1.48.0

Add a windows msvc cross compilation test

4ba11f7

messense force-pushed the python3-dll-a branch from ee9e6aa to 4ba11f7 Compare April 10, 2022 15:17

Use cargo-xwin to test windows msvc cross compilation

75ea171

messense reviewed Apr 11, 2022

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

messense reviewed Apr 11, 2022

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

davidhewitt reviewed Apr 11, 2022

View reviewed changes

Mention generate-abi3-import-lib in the features guide

4347624

Update the feature list section of the user guide to include `generate-abi3-import-lib` description.

davidhewitt approved these changes Apr 11, 2022

View reviewed changes

messense mentioned this pull request Apr 11, 2022

pyo3-build-config: Build "abi3" extensions without an interpreter #2293

Merged

davidhewitt merged commit 3b45170 into PyO3:main Apr 11, 2022

ravenexp deleted the python3-dll-a branch April 12, 2022 06:30

messense mentioned this pull request Apr 17, 2022

Better support for cross compiling to Windows PyO3/maturin#830

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyo3-build-config: Add `python3-dll-a` crate support #2282

pyo3-build-config: Add `python3-dll-a` crate support #2282

ravenexp commented Apr 7, 2022

adamreichold Apr 8, 2022

ravenexp Apr 8, 2022

ravenexp Apr 8, 2022

adamreichold Apr 8, 2022

ravenexp Apr 8, 2022

davidhewitt left a comment

ravenexp commented Apr 10, 2022

davidhewitt commented Apr 10, 2022 •

edited

ravenexp commented Apr 10, 2022

adamreichold commented Apr 10, 2022

davidhewitt commented Apr 10, 2022

ravenexp commented Apr 10, 2022

adamreichold commented Apr 10, 2022

messense commented Apr 10, 2022

ravenexp commented Apr 10, 2022

davidhewitt Apr 11, 2022

ravenexp Apr 11, 2022

davidhewitt left a comment

pyo3-build-config: Add python3-dll-a crate support #2282

pyo3-build-config: Add python3-dll-a crate support #2282

Conversation

ravenexp commented Apr 7, 2022

adamreichold Apr 8, 2022

Choose a reason for hiding this comment

ravenexp Apr 8, 2022

Choose a reason for hiding this comment

ravenexp Apr 8, 2022

Choose a reason for hiding this comment

adamreichold Apr 8, 2022

Choose a reason for hiding this comment

ravenexp Apr 8, 2022

Choose a reason for hiding this comment

davidhewitt left a comment

Choose a reason for hiding this comment

ravenexp commented Apr 10, 2022

davidhewitt commented Apr 10, 2022 • edited

ravenexp commented Apr 10, 2022

adamreichold commented Apr 10, 2022

davidhewitt commented Apr 10, 2022

ravenexp commented Apr 10, 2022

adamreichold commented Apr 10, 2022

messense commented Apr 10, 2022

ravenexp commented Apr 10, 2022

davidhewitt Apr 11, 2022

Choose a reason for hiding this comment

ravenexp Apr 11, 2022

Choose a reason for hiding this comment

davidhewitt left a comment

Choose a reason for hiding this comment

pyo3-build-config: Add `python3-dll-a` crate support #2282

pyo3-build-config: Add `python3-dll-a` crate support #2282

davidhewitt commented Apr 10, 2022 •

edited