pyo3-build-config: Make `PYO3_CROSS_LIB_DIR` optional #2241

ravenexp · 2022-03-22T14:26:45Z

Try to support most common extension module cross compilation targets
without requiring any of the PYO3_CROSS_* environment variables.

This is a work in progress tangentially related to #2231
I'm creating this mostly to get some initial feedback from the pyo3-build-config maintainers.

pyo3-build-config/src/impl_.rs

pyo3-build-config/src/lib.rs

davidhewitt

Thanks; to check I understand the idea, it's that when PYO3_CROSS_LIB_DIR is not set then (for unix extension modules at least) we can infer all we need from the abi3 and version settings?

Also, a fair number of functions have changed to be pub fn. Was that necessary?

.gitignore

pyo3-build-config/src/impl_.rs

ravenexp · 2022-03-22T16:33:00Z

Thanks; to check I understand the idea, it's that when PYO3_CROSS_LIB_DIR is not set then (for unix extension modules at least) we can infer all we need from the abi3 and version settings?

Yes, that's the general idea. I needed to make PYO3_CROSS_LIB_DIR optional to allow python3-dll-a to substitute its own library directory located somewhere inside OUT_DIR. In the process of doing that, I've realized it can also work for Unix-to-Unix cross compilation in many cases.

It's not entirely viable yet though, because of Android as you've mentioned above and also projects embedding a Python interpreter on Unix need to link to a real libpython3*.so somehow. This can be solved for Windows and abi3, but not in the general case.

This is a work in progress, and I need to add a clear error message for every currently accepted but broken configuration.

Also, a fair number of functions have changed to be pub fn. Was that necessary?

I've noticed that several PyO3 build scripts contained literal re-implementations of these functions, so I intended to unify them in later commits. This commit alone is getting way too big, perhaps I should further split it into the refactoring and the breaking change parts for easier review... Also the bug fix part needs to be split out.

davidhewitt · 2022-03-22T16:54:38Z

👍 makes sense, yes I think several commits (or even PRs) will be needed to get to the final goal for this one. These build scripts are quite fiddly, there's so many ways to distribute Python leading to lots of edge cases to handle!

ravenexp · 2022-03-22T17:28:34Z

My current plan is to submit a new PR with ABI3_MAX_MINOR bug fix only, and convert the current PR into 2+ commits on top of it. I've added a lot of new code that doesn't have test coverage, so it needs to be addressed too...

It it OK to rebase and force push pull requests for this project, or do you prefer that I use merges and fixup commits, so that you can squash them yourself before the final merge?

davidhewitt · 2022-03-22T17:43:28Z

Rebase & force push is fine! Thanks 👍

ravenexp · 2022-03-23T14:32:54Z

I'm resubmitting PR this as a set of smaller patches to get continuous feedback from the CI.

It's still a work in progress, but feel free to review the commits I've added so far. I'm highly confident in these, so if changes are requested the sooner I learn about them the better.

pyo3-build-config/src/impl_.rs

messense · 2022-03-24T06:26:08Z

pyo3-build-config/src/impl_.rs

+        self.os == "windows"
+    }
+
+    fn is_windows_mingw(&self) -> bool {


Why not is_windows_gnu?

That's the convention the existing code uses:

pyo3/pyo3-build-config/src/impl_.rs

Lines 209 to 211 in 9f60b67

print("calcsize_pointer", struct.calcsize("P"))

print("mingw", get_platform().startswith("mingw"))

"#;

pyo3/pyo3-build-config/src/impl_.rs

Lines 1370 to 1380 in 9f60b67

version: PythonVersion,

implementation: PythonImplementation,

abi3: bool,

mingw: bool,

) -> String {

if abi3 && !implementation.is_pypy() {

WINDOWS_ABI3_LIB_NAME.to_owned()

} else if mingw {

// https://packages.msys2.org/base/mingw-w64-python

format!("python{}.{}", version.major, version.minor)

} else {

ravenexp · 2022-03-24T12:16:35Z

I've just pushed version 2 of this PR, split into 5 smaller commits this time.
User documentation and the ChangeLog entry are still missing,
but I'll add them once all remaining code issues are solved.

Unresolved questions from the previous rounds of review (shortlist):

Replacing ad-hoc TargetInfo with target_lexicon::Triple. If the answer is yes, seems easier to address it in this PR.
Adding PYO3_CROSS_PYTHON_IMPLEMENTATION environment variable. This can be addressed in a different PR because it involves writing new user documentation and adding a ChangeLog entry.
Cross-compilation mode detection. See pyo3-build-config: Make PYO3_CROSS_LIB_DIR optional #2241 (comment) and the later discussion. I think I've solved it in the current revision, but it's better reviewed once again.

adamreichold · 2022-03-24T12:42:03Z

Replacing ad-hoc TargetInfo with target_lexicon::Triple. If the answer is yes, seems easier to address it in this PR.

In my experience, build scripts tend to become choke points in parallel builds which is why I would be wary of adding more dependencies to build scripts. That said, target-lexicon seems lightweight enough and seems to have good provenance with the Bytecode Alliance.

adamreichold · 2022-03-24T12:42:44Z

Adding PYO3_CROSS_PYTHON_IMPLEMENTATION environment variable. This can be addressed in a different PR because it involves writing new user documentation and adding a ChangeLog entry.

More smaller PR are preferable IMHO.

davidhewitt

Yes, I think this revision looks ok to me.

I had been wondering whether we would need to use target_lexicon, it would definitely be worth having a go in a separate commit so we can see how much it helps 👍

ravenexp · 2022-03-24T17:48:00Z

Thanks!

I had been wondering whether we would need to use target_lexicon, it would definitely be worth having a go in a separate commit so we can see how much it helps +1

I'll add this commit on top to evaluate the extent of the required changes, but I'd like to fold it in the first commit for the final revision to reduce useless code churn within the same PR.

ravenexp · 2022-03-24T18:07:37Z

In my experience, build scripts tend to become choke points in parallel builds which is why I would be wary of adding more dependencies to build scripts. That said, target-lexicon seems lightweight enough and seems to have good provenance with the Bytecode Alliance.

The issue with target-lexicon is that it itself has a quite extensive build script, which among other things tries to run rustc to detect its version. Unfortunately, it can't be disabled via crate features.

davidhewitt · 2022-03-24T23:09:11Z

Right, yes I think that would make it a lot less lightweight than we were hoping. I guess you decided not to try using it as a result?

Anything else to consider before merging this?

ravenexp · 2022-03-25T03:43:00Z

Right, yes I think that would make it a lot less lightweight than we were hoping. I guess you decided not to try using it as a result?

I can still try to add it as an experiment. I haven't actually checked if it has any measurable impact on the build times.

Anything else to consider before merging this?

Yes, updating user documentation and adding the changelog entry are still a TODO.

ravenexp · 2022-03-25T09:30:31Z

Right, yes I think that would make it a lot less lightweight than we were hoping. I guess you decided not to try using it as a result?

I can still try to add it as an experiment. I haven't actually checked if it has any measurable impact on the build times.

I've added d787d2a on top of the current branch to evaluate the target_lexicon::Triple integration impact.

I think there's a demonstrable benefit from the code quality standpoint.

The downsides:

Build times: A simple PyO3 extension module build time increased (on average) from 15.05s to 15.25s (+0.2s) because of the new target-lexicon dependency and its build script.
I had to break the public crate API in pyo3_build_config::cross_compile() function because there's just not enough information to make sound decisions based on the existing function arguments.

In my opinion the build time cost is acceptable, and as for cross_compile() I'm not sure if it has any external users.
PyO3 and maturin themselves do not use it. Maybe it's better to deprecate it altogether.

messense · 2022-03-25T09:43:17Z

and as for cross_compile() I'm not sure if it has any external users.
PyO3 and maturin themselves do not use it. Maybe it's better to deprecate it altogether.

It was supposed to be used to replace the duplicated code in maturin, see PyO3/maturin#718

ravenexp · 2022-03-25T10:24:41Z

and as for cross_compile() I'm not sure if it has any external users.
PyO3 and maturin themselves do not use it. Maybe it's better to deprecate it altogether.

It was supposed to be used to replace the duplicated code in maturin, see PyO3/maturin#718

Can this code be adapted to the new cross_compile() signature?
As I can see maturin already uses Triple internally.
This would delay the feature merge until the next major PyO3 release, though...

messense · 2022-03-25T10:30:23Z

Can this code be adapted to the new cross_compile() signature?

I think it's ok.

This would delay the feature merge until the next major PyO3 release, though...

I think it's fine to break that api since it was tailored for maturin anyway.

davidhewitt · 2022-03-25T10:34:24Z

I think it's fine to break that api since it was tailored for maturin anyway.

Even if someone else using it is unlikely, I would still prefer not change cross_compiling signature until the next breaking release. I think anyone who is using it would have good reason to be upset if we broke them in a semver-noncompliant way.

I'm thinking to put out a 0.16.3 release in the next few days (just one last thing I'd like to add to it), and then I'd be happy to start merging breaking changes because I've got ideas anyway 😄

ravenexp · 2022-03-25T10:38:32Z

Even if someone else using it is unlikely, I would still prefer not change cross_compiling signature until the next breaking release. I think anyone who is using it would have good reason to be upset if we broke them in a semver-noncompliant way.

I can try to keep this function as is and try to create a somewhat correct implementation for it until it's deprecated/removed in v0.17.
As a substitute, I'll add a new public API function that uses Triple arguments.

ravenexp · 2022-03-25T11:46:08Z

I've pushed the target-lexicon patch version 2 in fe9c160

Here I'm trying to keep the existing cross_compile() implementation afloat and introduce a new public function cross_compile_from_to() that uses Triple arguments directly.

If everyone is OK with moving on with target_lexicon::Triple, I'd rather split it off in a separate PR with its own ChangeLog entry and then rebase the current PR on top of it.

davidhewitt · 2022-03-25T14:46:30Z

Sounds good to me 👍

ravenexp · 2022-03-29T13:24:48Z

Rebased on top of #2253

Change the `CrossCompileConfig` structure definition and make the public `lib_dir` field optional to support more flexible cross-compilation configuration in the future. FIXME: This change breaks the public `pyo3-build-config` crate API. Update the sysconfigdata extraction functions to fall through when `lib_dir` field is not set. WIP: Add `unwrap()` stubs to the main cross compile switch.

Try to generalize `windows_hardcoded_cross_compile()` to all supported target platforms (when possible). Rename it to `default_cross_compile()` and add some unit tests. Rewrite `load_cross_compile_config()` to fall back to the default interpreter configuration when no other config information sources are available.

Rename `$OUT_DIR/pyo3-cross-compile-config.txt` to `$OUT_DIR/<triple>/pyo3-build-config.txt` to exclude the possibility of using stale build configuration data when the build target changes. Use the presence of the corresponding build configuration file in the `pyo3-build-config` build script output directory to detect whether we are cross compiling or not. This patch enables cross compilation without using any of `PYO3_CROSS_*` env variables in many cases.

Update Architecture.md to reflect the current cross compilation support state.

ravenexp · 2022-04-01T08:42:59Z

Rebased on main and reopened.

davidhewitt

Thanks, looks good to me. Thanks all for reviewing on this; I didn't manage to find any time to look for a number of days.

adamreichold reviewed Mar 22, 2022

View reviewed changes

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

adamreichold reviewed Mar 22, 2022

View reviewed changes

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

adamreichold reviewed Mar 22, 2022

View reviewed changes

pyo3-build-config/src/lib.rs Outdated Show resolved Hide resolved

davidhewitt reviewed Mar 22, 2022

View reviewed changes

ravenexp mentioned this pull request Mar 23, 2022

Fix abi3-py310 feature implementation: use Python 3.10 ABI when available. #2242

Merged

ravenexp force-pushed the cross-compile branch 3 times, most recently from 05e7a12 to 7e39565 Compare March 23, 2022 11:23

messense reviewed Mar 24, 2022

View reviewed changes

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

messense reviewed Mar 24, 2022

View reviewed changes

ravenexp requested review from davidhewitt and messense March 24, 2022 13:33

davidhewitt approved these changes Mar 24, 2022

View reviewed changes

messense approved these changes Mar 25, 2022

View reviewed changes

ravenexp force-pushed the cross-compile branch from 65b382e to d787d2a Compare March 25, 2022 09:32

ravenexp force-pushed the cross-compile branch from d787d2a to fe9c160 Compare March 25, 2022 11:40

ravenexp marked this pull request as draft March 25, 2022 15:34

ravenexp force-pushed the cross-compile branch from fe9c160 to ed73313 Compare March 29, 2022 13:23

adamreichold mentioned this pull request Mar 31, 2022

pyo3-build-config: Replace TargetInfo with target_lexicon::Triple #2253

Merged

ravenexp added 4 commits April 1, 2022 11:32

Update the user guide and add a ChangeLog entry

2d2b9f5

Update Architecture.md to reflect the current cross compilation support state.

ravenexp force-pushed the cross-compile branch from ed73313 to 2d2b9f5 Compare April 1, 2022 08:42

ravenexp marked this pull request as ready for review April 1, 2022 08:43

ravenexp requested review from davidhewitt, messense and adamreichold April 1, 2022 09:04

messense approved these changes Apr 2, 2022

View reviewed changes

davidhewitt approved these changes Apr 2, 2022

View reviewed changes

davidhewitt merged commit 040ce86 into PyO3:main Apr 2, 2022

ravenexp deleted the cross-compile branch April 2, 2022 20:49

ravenexp mentioned this pull request Apr 4, 2022

release: 0.16.3 #2267

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyo3-build-config: Make `PYO3_CROSS_LIB_DIR` optional #2241

pyo3-build-config: Make `PYO3_CROSS_LIB_DIR` optional #2241

ravenexp commented Mar 22, 2022

davidhewitt left a comment

ravenexp commented Mar 22, 2022

davidhewitt commented Mar 22, 2022

ravenexp commented Mar 22, 2022

davidhewitt commented Mar 22, 2022

ravenexp commented Mar 23, 2022

messense Mar 24, 2022

ravenexp Mar 24, 2022

ravenexp commented Mar 24, 2022

adamreichold commented Mar 24, 2022

adamreichold commented Mar 24, 2022

davidhewitt left a comment

ravenexp commented Mar 24, 2022

ravenexp commented Mar 24, 2022

davidhewitt commented Mar 24, 2022

ravenexp commented Mar 25, 2022

ravenexp commented Mar 25, 2022 •

edited

messense commented Mar 25, 2022

ravenexp commented Mar 25, 2022 •

edited

messense commented Mar 25, 2022

davidhewitt commented Mar 25, 2022

ravenexp commented Mar 25, 2022

ravenexp commented Mar 25, 2022

davidhewitt commented Mar 25, 2022

ravenexp commented Mar 29, 2022

ravenexp commented Apr 1, 2022

davidhewitt left a comment

	print("calcsize_pointer", struct.calcsize("P"))
	print("mingw", get_platform().startswith("mingw"))
	"#;

	version: PythonVersion,
	implementation: PythonImplementation,
	abi3: bool,
	mingw: bool,
	) -> String {
	if abi3 && !implementation.is_pypy() {
	WINDOWS_ABI3_LIB_NAME.to_owned()
	} else if mingw {
	// https://packages.msys2.org/base/mingw-w64-python
	format!("python{}.{}", version.major, version.minor)
	} else {

pyo3-build-config: Make PYO3_CROSS_LIB_DIR optional #2241

pyo3-build-config: Make PYO3_CROSS_LIB_DIR optional #2241

Conversation

ravenexp commented Mar 22, 2022

davidhewitt left a comment

Choose a reason for hiding this comment

ravenexp commented Mar 22, 2022

davidhewitt commented Mar 22, 2022

ravenexp commented Mar 22, 2022

davidhewitt commented Mar 22, 2022

ravenexp commented Mar 23, 2022

messense Mar 24, 2022

Choose a reason for hiding this comment

ravenexp Mar 24, 2022

Choose a reason for hiding this comment

ravenexp commented Mar 24, 2022

adamreichold commented Mar 24, 2022

adamreichold commented Mar 24, 2022

davidhewitt left a comment

Choose a reason for hiding this comment

ravenexp commented Mar 24, 2022

ravenexp commented Mar 24, 2022

davidhewitt commented Mar 24, 2022

ravenexp commented Mar 25, 2022

ravenexp commented Mar 25, 2022 • edited

messense commented Mar 25, 2022

ravenexp commented Mar 25, 2022 • edited

messense commented Mar 25, 2022

davidhewitt commented Mar 25, 2022

ravenexp commented Mar 25, 2022

ravenexp commented Mar 25, 2022

davidhewitt commented Mar 25, 2022

ravenexp commented Mar 29, 2022

ravenexp commented Apr 1, 2022

davidhewitt left a comment

Choose a reason for hiding this comment

pyo3-build-config: Make `PYO3_CROSS_LIB_DIR` optional #2241

pyo3-build-config: Make `PYO3_CROSS_LIB_DIR` optional #2241

ravenexp commented Mar 25, 2022 •

edited

ravenexp commented Mar 25, 2022 •

edited