auto-initialize: new feature to control initializing Python #1347

davidhewitt · 2020-12-28T21:16:16Z

This PR add a new feature auto-initialize to control whether Python::with_gil and Python::acquire_gil call prepare_freethreaded_python automatically.

I wanted to add this for a couple of reasons:

When building an extension-module then this functionality is not needed.
When linking against a static python lib (or PyPy) then this functionality doesn't even work correctly.

The PR checks for the static python and PyPy cases and removes the prepare_freethreaded_python API, and instead emits a compile_error! if the auto-initialize feature is enabled.

Closes #742
Closes #762
Closes #1213

TODO:

It would be nice to make this feature opt-in instead of opt-out, but that's a breaking change so we should delay that until 0.14 release.

I'd like to write some more documentation for this in the guide before merging.

Also needs CHANGELOG entry.

kngwyu · 2020-12-29T03:29:42Z

Thanks!
I really like the idea to raise a compile error before failing to link, but I need some clarifications:

About prepare_freethreaded_python:
- I agree that this should be disabled for PyPy or static linking. Then, how about introducing some new configuration(e.g., Py_SHARED) and mark #[cfg(not(any(PyPy, Py_SHARED)))] fn prepare_freethreaded_python? I'm not sure this should be a feature flag.
About auto-initialize
- Isn't this a not(extension-module)? Why do we need a new feature?

Again, this is a difficult problem, and thank you for tackling it.

davidhewitt · 2020-12-29T05:24:07Z

Thanks for the questions. I'm very happy to change the design if we think alternatives are better. Here's my thinking about your proposed alternatives:

using Py_SHARED flag, I don't think we can fail compilation in build.rs. This means the error message when building will be that prepare_freethtreaded_python is missing. A nicer error than link failures maybe, especially if we add documentation to that function do that users understand what might have happened.

The failure will still be quite hard to diagnose for the user compared to a friendly error message from the build script.
With auto-initialize being not(extension-module) - I suppose in practice yes, this is true. But I think this would then be another case of The extension-module feature fails to be additive #771 because then extension-module feature is taking away functionality again.

Also, this would mean that users who want to run cargo test which relies on auto-initialize would have to always use the --no-default-features workaround to disable extension-module, even after Fix cargo test with extension-module feature. #1123 is stable. It seems more idiomatic Rust to me to be able to control this properly with a separate feature flag.

The feature flag is also more general. For example, in some special cases users might want to embed python and manually initialize it before calling PyO3 code. In those cases they don't need auto-initialize functionality either.

davidhewitt · 2020-12-29T05:50:20Z

... I'm thinking that maybe we can use Py_SHARED plus the auto-initialize feature. We can use compile_error! to produce a friendly error message, so my argument above for the embedding feature now seems weak.

davidhewitt · 2020-12-31T01:18:53Z

Reworked this PR as discussed but I expect CI to continue to fail until #1350 is merged, because one of the simplifications in that PR fixes how Py_SHARED would be set.

davidhewitt · 2021-01-01T08:17:27Z

Rebased and rebuilt this PR as a single commit; hopefully CI is green this time 🤞

kngwyu

LGTM, thanks!
CI failures are because auto-initialise is unintentionally invoked. I'm sorry but I have no idea why this happens.

src/gil.rs

kngwyu · 2021-01-02T07:22:33Z

src/gil.rs

+        cfg_if::cfg_if! {
+            if #[cfg(all(feature = "auto-initialize", Py_SHARED, not(PyPy)))] {
+                prepare_freethreaded_python();
+            } else if #[cfg(all(feature = "auto-initialize", not(feature = "extension-module"), not(Py_SHARED), not(rustdoc)))] {


I think we can make this simpler by else if all(auto-initialize, PyPy) { ... } else if all(auto_initialize, ...)

not(extension-module) is unfortunately necessary to avoid a breaking change - otherwise all extension modules would suddenly start failing to build PyPy wheels unless users changed their Cargo.toml to remove default features. We could make it as you suggest in 0.14.

Ah, thanks. That makes sense.
BTW, do you think that we should keep auto-initialize in default-features?

I think in the long-run it's better to remove auto-initialize from default-features, but that's also a breaking change, so let's remove it in 0.14.

I'll create an issue to track these cleanups and add it to the 0.14 milestone.

kngwyu · 2021-01-02T07:36:11Z

.github/workflows/ci.yml


-      - name: Build without default features
+      - name: Build (no features)
        run: cargo build --no-default-features --verbose --target ${{ matrix.platform.rust-target }}


It looks like that auto-initialize and macros are enabled in this build, though I don't know why 😕

It's because of the Cargo.toml changes - see other comment.

kngwyu · 2021-01-02T07:36:47Z

.github/workflows/ci.yml


      # Run tests (except on PyPy, because no embedding API).
      - if: matrix.python-version != 'pypy-3.6'
        name: Test
-        run: cargo test --features "num-bigint num-complex hashbrown" --target ${{ matrix.platform.rust-target }}
+        run: cargo test --no-default-features --features "macros num-bigint num-complex hashbrown" --target ${{ matrix.platform.rust-target }}


Why not using auto-initialize?

It's automatically included in tests because I put this in [dev-dependencies]:

# features needed to run the PyO3 test suite pyo3 = { path = ".", default-features = false, features = ["macros", "auto-initialize"] }

However, that's also the reason why auto-initialize and macros are included in the cargo build command above. Interestingly cargo is about to stabilize a fix for this as a new resolver = "2" option in Cargo.toml. rust-lang/cargo#8997 (I tested on nightly and the features are no longer included in cargo build.)

Because in the future what I've done here will work correctly in the future, I'm going to keep the code mostly as-is for the moment and introduce a hack to avoid the immediate issue with a TODO to tidy it up once we bump MSRV in the future (probably to Rust 1.52).

davidhewitt · 2021-01-02T16:59:04Z

@kngwyu I pushed a CHANGELOG entry and a new guide page guide/src/features.md to document this functionality.

I also added a tiny "hack" with cfg(__pyo3_ci) to suppress the error message in our CI until we are able to bump MSRV to a Rust version which includes the cargo resolver = "2". Might be nearly a year though - I put TODO on the location in ci.yml and build.rs where I added the hack.

If you are happy with this, I think this is ready to merge! 🚀

kngwyu · 2021-01-04T03:48:39Z

I also added a tiny "hack" with cfg(__pyo3_ci) to suppress the error message in our CI until we are able to bump MSRV to a Rust version which includes the cargo resolver = "2". Might be nearly a year though - I put TODO on the location in ci.yml and build.rs where I added the hack.

Is it much simpler to remove the dev-dependency for now?

davidhewitt · 2021-01-04T07:35:45Z

If we remove the dev-dependency and also remove auto-initialize from default features then to run the PyO3 test suite we'll have to write cargo test --features auto-initialize.

I'm sure I'll forget to add the feature a lot and it's also annoying to type something so long.

I think it's better to leave the very small hack in so that we can make exactly the API we want for users. Especially as we know the fix which removes the need for the hack is going to be stabilised very soon.

(Because we only need the hack to fix the pypy CI jobs, we only need to wait until it's in the most recent stable rust version.)

kngwyu · 2021-01-04T07:48:02Z

If we remove the dev-dependency and also remove auto-initialize from default features

OK, if you are to release 0.14 before the next stable release, let's go in this way 👍🏼

davidhewitt · 2021-01-04T21:08:46Z

(I force-pushed to fix the commit message. Will merge shortly.)

davidhewitt force-pushed the embedding branch from dfa52b2 to 600db5f Compare December 28, 2020 21:23

davidhewitt force-pushed the embedding branch 7 times, most recently from e336bbe to 1543525 Compare December 31, 2020 01:16

davidhewitt mentioned this pull request Jan 1, 2021

Refactor build rs #1350

Merged

davidhewitt force-pushed the embedding branch 3 times, most recently from 9d4a410 to 5eb33f7 Compare January 1, 2021 08:14

davidhewitt changed the title ~~embedding: new feature to enable embedding Python~~ auto-initialize: new feature to control initializing Python Jan 1, 2021

davidhewitt force-pushed the embedding branch from 5eb33f7 to f592a9f Compare January 1, 2021 08:36

kngwyu approved these changes Jan 2, 2021

View reviewed changes

davidhewitt force-pushed the embedding branch 2 times, most recently from 02e27c1 to c2cc186 Compare January 2, 2021 16:05

auto-initialize: new feature to control initializing Python

59707f0

davidhewitt force-pushed the embedding branch from c2cc186 to 59707f0 Compare January 2, 2021 16:08

davidhewitt mentioned this pull request Jan 2, 2021

auto-initialize to become opt-in on PyO3 0.14 #1354

Closed

This was referenced Jan 3, 2021

gil: tidy ups to finalization #1355

Merged

readme/faq: document rlib crate type better #1356

Merged

auto-initialize: add CHANGELOG and docs

e0c35d1

davidhewitt force-pushed the embedding branch from 1b92a8a to e0c35d1 Compare January 4, 2021 21:08

davidhewitt merged commit 8e37d37 into PyO3:master Jan 4, 2021

davidhewitt deleted the embedding branch January 4, 2021 21:31

davidhewitt mentioned this pull request Mar 30, 2021

Add PYO3_CROSS_NO_ENABLE_SHARED variable #1532

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

auto-initialize: new feature to control initializing Python #1347

auto-initialize: new feature to control initializing Python #1347

davidhewitt commented Dec 28, 2020 •

edited

kngwyu commented Dec 29, 2020

davidhewitt commented Dec 29, 2020

davidhewitt commented Dec 29, 2020

davidhewitt commented Dec 31, 2020

davidhewitt commented Jan 1, 2021

kngwyu left a comment •

edited

kngwyu Jan 2, 2021

davidhewitt Jan 2, 2021 •

edited

kngwyu Jan 2, 2021

davidhewitt Jan 2, 2021

kngwyu Jan 4, 2021

kngwyu Jan 2, 2021

davidhewitt Jan 2, 2021

kngwyu Jan 2, 2021

davidhewitt Jan 2, 2021

davidhewitt commented Jan 2, 2021

kngwyu commented Jan 4, 2021

davidhewitt commented Jan 4, 2021

kngwyu commented Jan 4, 2021 •

edited

davidhewitt commented Jan 4, 2021

auto-initialize: new feature to control initializing Python #1347

auto-initialize: new feature to control initializing Python #1347

Conversation

davidhewitt commented Dec 28, 2020 • edited

kngwyu commented Dec 29, 2020

davidhewitt commented Dec 29, 2020

davidhewitt commented Dec 29, 2020

davidhewitt commented Dec 31, 2020

davidhewitt commented Jan 1, 2021

kngwyu left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidhewitt Jan 2, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidhewitt commented Jan 2, 2021

kngwyu commented Jan 4, 2021

davidhewitt commented Jan 4, 2021

kngwyu commented Jan 4, 2021 • edited

davidhewitt commented Jan 4, 2021

davidhewitt commented Dec 28, 2020 •

edited

kngwyu left a comment •

edited

davidhewitt Jan 2, 2021 •

edited

kngwyu commented Jan 4, 2021 •

edited