Expose cross compiling configuration from pyo3-build-config #1996

aganders3 · 2021-11-15T14:50:44Z

These modifications were proposed in #1627.

So far this exposes three functions:

cross_compiling (altered to take arguments, with current behavior moved to cross_compiling_from_cargo_env)
find_sysconfigdata
parse_sysconfigdata

and modifies InterpreterConfig with three new fields:

abi_flags
abi_tag
ext_suffix

The refactored cross_compiling takes host (triple), arch, vendor, and os, but vendor is not currently used separately from what I see in maturin. It might be better to change it to take only the complete "target triple" to simplify things, but splitting os and arch from there for CrossCompileConfig may not be straightforward as the overall format seems a bit complicated (i.e. not always only three parts). Maturin uses another crate (target_lexicon) for this, which could be pulled in here so it could perhaps take a target_lexicon::Triple. This might be the best option for supporting the most platforms.

I'm also not sure I have the correct behavior for the fields added to InterpreterConfig - specifically I'm not sure if the defaults are handled correctly. Also, do they need to change depending on the value of abi3?

It seems there's more in pyo3-build-config that could be exposed so I'm happy to bundle that in this PR or work on the different features separately.

TODO:

an entry in CHANGELOG.md
docs to all new functions and / or detail in the guide (possibly related to Better pyo3-build-config example #1628, docs: pyo3 config files #1976)
tests for all new or changed functions

… fields to InterpreterConfig

davidhewitt

Looking good! Few comments below.

I try to avoid adding dependencies to pyo3 and sub-crates as much as possible. It looks like target_lexicon is itself a simple dependency with no other dependencies, so it's relatively lightweight. Which platforms do you feel we're lacking support for which target_lexicon could help with?

pyo3-build-config/src/impl_.rs

pyo3-build-config/src/lib.rs

aganders3 · 2021-11-15T15:45:40Z

Thanks for the quick feedback! I'll work on these suggestions.

pyo3-build-config/src/impl_.rs

messense · 2021-11-16T02:32:00Z

Thank you for helping with this. Just a reminder, this branch has conflicts that must be resolved.

pyo3-build-config/src/impl_.rs

messense · 2021-11-17T03:58:20Z

FYI, we recently merged PyO3/maturin#677 , it introduced new dependency on sysconfig.get_platform() of Python.

aganders3 · 2021-11-17T19:05:10Z

FYI, we recently merged PyO3/maturin#677 , it introduced new dependency on sysconfig.get_platform() of Python.

Hm, thanks for the heads up. This is already called by InterpreterConfig::from_interpreter but parse_sysconfigdata does not necessarily use the target interpreter.

pyo3-build-config/src/impl_.rs

…nterpreterConfig::from_sysconfigdata.

aganders3 · 2021-11-20T03:53:26Z

I know I definitely still have a couple tests to write, but I'm a bit confused by the codecov results. Is there a good way for me to run this locally?

Otherwise I think this refactor definitely provides a better API now than before, so thanks for the suggestion! I'm not sure if the parse_sysconfigdata is what you had in mind, as this will parse the whole file where before there was a list of keys to keep.

davidhewitt · 2021-11-20T14:08:27Z

If you go into the pyo3-build-config subdirectory you should be able to run cargo test to exercise the tests in these files? Or is there something more specific you want to run.

This is definitely looking great, let me know when you want a re-review. (I think there will only be minor nits left tbh, nice work.)

this will parse the whole file where before there was a list of keys to keep

Yes indeed I didn't state it but I was assuming that we would parse (and keep) the whole file contents 👍

davidhewitt · 2021-11-20T14:09:14Z

(#2011 may be relevant if you're getting weird coverage results.)

…re open.

aganders3 · 2021-11-23T17:53:20Z

I wrote a few more tests to improve coverage (not pushed), but the cases where it is cross compiling all rely on environment variables, which I am having a hard time controlling for tests. I can set/unset in each test, but the default for cargo test is to run in parallel which causes intermittent failures. Do you have any suggestions?

Aside from that I am mostly happy with the interface and would appreciate feedback when you have time to review. I don't have a great feeling for whether cross_compiling should take AsRef<str> or Into<String> instead of &str for some of its args, for example. It seems maybe it should be something like this, because the host can always be borrowed but the target values need to become owned by the CrossCompileConfig if cross-compiling:

pub fn cross_compiling(
    host: AsRef<str>,
    target_arch: Into<String>,
    target_vendor: Into<String>,
    target_os: Into<String>,
) -> Result<Option<CrossCompileConfig>>

Edit: regarding tests/coverage

Thanks - I can run the tests locally (though I could get better at actually doing that before I push), but I wasn't sure how to check for a codecov failure before pushing. I guess codecov is its own thing though even if I can measure coverage locally. I was having trouble viewing the full report but that seems working now.

davidhewitt · 2021-11-23T23:07:38Z

👍 I'll give this a full review in the morning, thanks!

EDIT: sorry it took me an extra day and a bit 😪

messense · 2021-11-24T02:20:10Z

Looks good to me! I'd recommend to actually give it a try on maturin(as a git dependency) to see if it fits.

davidhewitt

Thank you, this is looking excellent!

I have a few minor tidy ups commented below, really just nits though.

I agree also that it's probably worth having a whirl with using this as a git dependency with maturin (and maybe even opening that branch as a PR on the maturin repo) to confirm that we're happy with how this works downstream before we merge.

pyo3-build-config/src/impl_.rs

davidhewitt · 2021-11-30T08:22:37Z

👍 this is looking great to me from the PyO3 side, thanks!

I see you have opened PyO3/maturin#718, so will wait to merge this until you're happy with that one too. Ping me when ready!

aganders3 · 2021-12-03T17:09:40Z

pyo3-build-config/src/impl_.rs

+    if cfg!(feature = "resolve-config") {
+        println!("cargo:rerun-if-env-changed={}", var);
+    }


@davidhewitt let me know if this is problematic or if there's a better way to suppress this output in maturin.

This seems like a sensible way to do it to me!

aganders3 · 2021-12-06T17:47:12Z

Okay PyO3/maturin#718 seems good to go and I think this is ready for another look. I will make a few inline comments for areas I have questions but of course appreciate any additional feedback

Merging probably needs to wait for #2040 but otherwise hopefully we can close this out soon!

aganders3 · 2021-12-06T17:49:07Z

pyo3-build-config/src/impl_.rs

@@ -826,7 +920,7 @@ fn ends_with(entry: &DirEntry, pat: &str) -> bool {
 /// So we must find the file in the following possible locations:
 ///
 /// ```sh
-/// # distribution from package manager, lib_dir should include lib/
+/// # distribution from package manager, (lib_dir may or may not include lib/)


Hopefully this is okay - see implementation below. This was requested in Maturin to support cross-compiling with PyPy, where the _sysconfigdata*.py is not under the lib dir.

messense · 2021-12-13T02:37:09Z

I think we have a use case for the new Sysconfigdata in cryptography, how about exposing the current detected Python interpreter and it's sysconfigdata in pyo3-build-config for downstream consumption?

aganders3 · 2021-12-13T03:02:14Z

I think we have a use case for the new Sysconfigdata in cryptography, how about exposing the current detected Python interpreter and it's sysconfigdata in pyo3-build-config for downstream consumption?

Just as I merge main and get back to green 😆.

I can look into adding this here if you all think it's best to have in the same PR. It doesn't seem like too much work. In any case I'd be happy to take this on as another project.

messense · 2021-12-13T03:08:58Z

I can look into adding this here if you all think it's best to have in the same PR. It doesn't seem like too much work. In any case I'd be happy to take this on as another project.

Thanks! Do it in another PR is fine.

aganders3 · 2021-12-15T02:41:32Z

No rush @davidhewitt and @messense but I'm just waiting for feedback at this point so please let me know if there's anything else needed here! My PR in Maturin is passing tests while pointed to the latest in this branch.

Otherwise I will start looking into the above request to expose more features of pyo3-build-config for use in cryptography.

messense · 2021-12-15T02:46:03Z

Looks good to me. Let's give @davidhewitt a chance to take another look before merging.

davidhewitt

Looks brilliant! Thank you for all the work put into this, and I'm sorry it took me so long to find time to review.

davidhewitt · 2021-12-16T00:11:57Z

pyo3-build-config/src/impl_.rs

+    if cfg!(feature = "resolve-config") {
+        println!("cargo:rerun-if-env-changed={}", var);
+    }


This seems like a sensible way to do it to me!

Move changelog entry for #1996.

Ashley Anderson added 2 commits November 13, 2021 12:43

Initial refactor - expose cross-compiling functions and add necessary…

fbab45c

… fields to InterpreterConfig

Refactor cross_compiling to take arch/vendor/os separately.

1bd27f7

davidhewitt reviewed Nov 15, 2021

View reviewed changes

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

pyo3-build-config/src/lib.rs Outdated Show resolved Hide resolved

Address review comments.

69cd710

aganders3 commented Nov 16, 2021

View reviewed changes

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

Ashley Anderson added 3 commits November 16, 2021 11:33

Merge branch 'main' into expose-cross-compiling

1a326c2

Update changelog with note about pyo3-build-config APIs.

52301f0

Fix panic when parsing ABI tag on Windows.

7ab810a

aganders3 force-pushed the expose-cross-compiling branch from 97439a3 to 7ab810a Compare November 16, 2021 17:57

aganders3 commented Nov 16, 2021

View reviewed changes

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

Update parse_sysconfigdata test to best-guess values for linux.

c8f0785

aganders3 force-pushed the expose-cross-compiling branch from 1f16e1c to c8f0785 Compare November 16, 2021 19:28

messense mentioned this pull request Nov 17, 2021

Add support for cross compiling PyPy wheels PyO3/maturin#687

Merged

davidhewitt reviewed Nov 17, 2021

View reviewed changes

pyo3-build-config/src/impl_.rs Outdated Show resolved Hide resolved

Ashley Anderson added 2 commits November 19, 2021 14:36

Revert added fields in InterpreterConfig.

490fe61

Refactor parse_sysconfigdata to return Sysconfigdata (HashMap). Add I…

4974fc8

…nterpreterConfig::from_sysconfigdata.

aganders3 force-pushed the expose-cross-compiling branch from 664c31d to 4974fc8 Compare November 20, 2021 01:19

Ashley Anderson added 4 commits November 22, 2021 13:04

Update BuildFlags test to use from_sysconfigdata.

c698c2e

Merge branch 'main' into expose-cross-compiling

40fa4be

Add tests for from_sysconfigdata. Refactor Sysconfigdata API to be mo…

011d8fb

…re open.

Add basic tests for not cross compiling. Add some error handling.

79565d7

davidhewitt approved these changes Nov 25, 2021

View reviewed changes

Address review comments.

8b0c391

aganders3 mentioned this pull request Nov 30, 2021

Get cross compiling from pyo3-build-config PyO3/maturin#718

Closed

Ashley Anderson added 5 commits November 30, 2021 12:10

Update search_lib_dir to recurse into lib and pypy dirs.

6b40cb4

Merge branch 'main' into expose-cross-compiling

bc128e1

Look even harder for sysconfigdata.

edb59c3

Add skip-build-config feature.

c0f5ce5

Revert skip-build-config feature.

2986aca

aganders3 force-pushed the expose-cross-compiling branch from 94362d1 to 2986aca Compare December 3, 2021 16:36

Suppress cargo:rerun-if-env-changed without resolve-config feature.

41cc76d

aganders3 commented Dec 3, 2021

View reviewed changes

aganders3 commented Dec 6, 2021

View reviewed changes

Merge branch 'main' into expose-cross-compiling

fa09343

messense approved these changes Dec 13, 2021

View reviewed changes

messense referenced this pull request in pyca/cryptography Dec 14, 2021

Define Py_LIMITED_API

065c2f1

davidhewitt approved these changes Dec 16, 2021

View reviewed changes

davidhewitt merged commit c30ca03 into PyO3:main Dec 16, 2021

davidhewitt mentioned this pull request Dec 16, 2021

Expose pyo3-build-config APIs for maturin #1627

Closed

aganders3 pushed a commit to aganders3/pyo3 that referenced this pull request Dec 16, 2021

Move changelog entry for PyO3#1996.

a6c7016

aganders3 mentioned this pull request Dec 16, 2021

Move changelog entry for #1996. #2054

Merged

davidhewitt added a commit that referenced this pull request Dec 16, 2021

Merge pull request #2054 from aganders3/fix-changelog

777a607

Move changelog entry for #1996.

aganders3 deleted the expose-cross-compiling branch December 16, 2021 21:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose cross compiling configuration from pyo3-build-config #1996

Expose cross compiling configuration from pyo3-build-config #1996

aganders3 commented Nov 15, 2021 •

edited

davidhewitt left a comment

aganders3 commented Nov 15, 2021

messense commented Nov 16, 2021

messense commented Nov 17, 2021

aganders3 commented Nov 17, 2021

aganders3 commented Nov 20, 2021

davidhewitt commented Nov 20, 2021

davidhewitt commented Nov 20, 2021

aganders3 commented Nov 23, 2021 •

edited

davidhewitt commented Nov 23, 2021 •

edited

messense commented Nov 24, 2021

davidhewitt left a comment

davidhewitt commented Nov 30, 2021

aganders3 Dec 3, 2021

davidhewitt Dec 16, 2021

aganders3 commented Dec 6, 2021

aganders3 Dec 6, 2021

messense commented Dec 13, 2021

aganders3 commented Dec 13, 2021

messense commented Dec 13, 2021

aganders3 commented Dec 15, 2021

messense commented Dec 15, 2021

davidhewitt left a comment

davidhewitt Dec 16, 2021

Expose cross compiling configuration from pyo3-build-config #1996

Expose cross compiling configuration from pyo3-build-config #1996

Conversation

aganders3 commented Nov 15, 2021 • edited

davidhewitt left a comment

Choose a reason for hiding this comment

aganders3 commented Nov 15, 2021

messense commented Nov 16, 2021

messense commented Nov 17, 2021

aganders3 commented Nov 17, 2021

aganders3 commented Nov 20, 2021

davidhewitt commented Nov 20, 2021

davidhewitt commented Nov 20, 2021

aganders3 commented Nov 23, 2021 • edited

davidhewitt commented Nov 23, 2021 • edited

messense commented Nov 24, 2021

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt commented Nov 30, 2021

aganders3 Dec 3, 2021

Choose a reason for hiding this comment

davidhewitt Dec 16, 2021

Choose a reason for hiding this comment

aganders3 commented Dec 6, 2021

aganders3 Dec 6, 2021

Choose a reason for hiding this comment

messense commented Dec 13, 2021

aganders3 commented Dec 13, 2021

messense commented Dec 13, 2021

aganders3 commented Dec 15, 2021

messense commented Dec 15, 2021

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt Dec 16, 2021

Choose a reason for hiding this comment

aganders3 commented Nov 15, 2021 •

edited

aganders3 commented Nov 23, 2021 •

edited

davidhewitt commented Nov 23, 2021 •

edited