Skip to content

Commit

Permalink
pyo3-build-config: docs wip
Browse files Browse the repository at this point in the history
  • Loading branch information
davidhewitt committed May 21, 2021
1 parent 79c7149 commit 825ec08
Show file tree
Hide file tree
Showing 6 changed files with 69 additions and 5 deletions.
1 change: 1 addition & 0 deletions guide/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
- [Features Reference](features.md)
- [Advanced Topics](advanced.md)
- [Building and Distribution](building_and_distribution.md)
- [Supporting multiple Python versions](building_and_distribution/multiple_python_versions.md)
- [PyPy support](building_and_distribution/pypy.md)
- [Useful Crates](ecosystem.md)
- [Logging](ecosystem/logging.md)
Expand Down
7 changes: 4 additions & 3 deletions guide/src/building_and_distribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,7 @@ There are two ways to distribute your module as a Python package: [setuptools-ru

By default, Python extension modules can only be used with the same Python version they were compiled against -- if you build an extension module with Python 3.5, you can't import it using Python 3.8. [PEP 384](https://www.python.org/dev/peps/pep-0384/) introduced the idea of the limited Python API, which would have a stable ABI enabling extension modules built with it to be used against multiple Python versions. This is also known as `abi3`.

Note that [maturin] >= 0.9.0 or [setuptools-rust] >= 0.11.4 support `abi3` wheels.
See the [corresponding](https://github.com/PyO3/maturin/pull/353) [PRs](https://github.com/PyO3/setuptools-rust/pull/82) for more.
The advantage of building extension module using the limited Python API is that you only need to build and distribute a single copy (for each OS / architecture), and your users can install it on all Python versions from your [minimum version](#minimum-python-version-for-abi3) and up. The downside of this is that PyO3 can't use optimizations which rely on being compiled against a known exact Python version. It's up to you to decide whether this matters for your extension module. It's also possible to design your extension module such that you can distribute `abi3` wheels but allow users compiling from source to benefit from additional optimizations - see the [support for multiple python versions](./building_and_distribution/multiple_python_versions.html) section of this guide, in particular the `#[cfg(Py_LIMITED_API)]` flag.

There are three steps involved in making use of `abi3` when building Python packages as wheels:

Expand All @@ -55,7 +54,8 @@ There are three steps involved in making use of `abi3` when building Python pack
pyo3 = { {{#PYO3_CRATE_VERSION}}, features = ["abi3"] }
```

2. Ensure that the built shared objects are correctly marked as `abi3`. This is accomplished by telling your build system that you're using the limited API.
2. Ensure that the built shared objects are correctly marked as `abi3`. This is accomplished by telling your build system that you're using the limited API. [maturin] >= 0.9.0 and [setuptools-rust] >= 0.11.4 support `abi3` wheels.
See the [corresponding](https://github.com/PyO3/maturin/pull/353) [PRs](https://github.com/PyO3/setuptools-rust/pull/82) for more.

3. Ensure that the `.whl` is correctly marked as `abi3`. For projects using `setuptools`, this is accomplished by passing `--py-limited-api=cp3x` (where `x` is the minimum Python version supported by the wheel, e.g. `--py-limited-api=cp35` for Python 3.5) to `setup.py bdist_wheel`.

Expand All @@ -76,6 +76,7 @@ not work when compiling for `abi3`. These are:
- `#[text_signature]` does not work on classes until Python 3.10 or greater.
- The `dict` and `weakref` options on classes are not supported until Python 3.9 or greater.
- The buffer API is not supported.
- Optimizations which rely on knowledge of the exact Python version compiled against.

## Cross Compiling

Expand Down
60 changes: 60 additions & 0 deletions guide/src/building_and_distribution/multiple_python_versions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Supporting multiple Python versions

PyO3 supports all actively-supported Python 3 and PyPy versions. As much as possible, this is done internally to PyO3 so that your crate's code does not need to adapt to the differences between each version. However, as Python features grow and change between versions, PyO3 cannot a completely identical API for every Python version. This may require you to add conditional compilation to your crate or runtime checks for the Python version.

This section of the guide first introduces the `pyo3-build-config` crate, which you can use as a `build-dependency` to add additional `#[cfg]` flags which allow you to support multiple Python versions at compile-time.

Second, we'll show how to check the Python version at runtime. This can be useful when building for multiple versions with the `abi3` feature, where the Python API compiled against is not always the same as the one in use.

## Conditional compilation for different Python versions

The `pyo3-build-config` exposes multiple [`#[cfg]` flags](https://doc.rust-lang.org/rust-by-example/attribute/cfg.html) which can be used to conditionally compile code for a given Python version. PyO3 itself depends on this crate, so by using it you can be sure that you are configured correctly for the Python version PyO3 is building against.

This allows us to write code like the following

```rust,ignore
#[cfg(Py_3_7)]
fn function_only_supported_on_python_3_7_and_up() { }
#[cfg(not(Py_3_8))]
fn function_only_supported_before_python_3_8() { }
#[cfg(not(Py_LIMITED_API))]
fn function_incompatible_with_abi3_feature() { }
```

The following sections first show how to add these `#[cfg]` flags to your build process, and then cover some common patterns flags in a little more detail.

To see a full reference of all the `#[cfg]` flags provided, see the [`pyo3-build-cfg` docs](https://docs.rs/pyo3-build-config).

### Using `pyo3-build-config`

You can use the `#[cfg]` flags in just two steps:

1. Add `pyo3-build-config` it to your crate's build dependencies in `Cargo.toml`:

```toml
[build-dependencies]
pyo3-build-config = "{{#PYO3_CRATE_VERSION}}"
```

2. Add a [`build.rs`](https://doc.rust-lang.org/cargo/reference/build-scripts.html) file to your crate with the following contents:

```rust,ignore
fn main() {
// If you have an existing build.rs file, just add this line to it.
pyo3_build_config::use_pyo3_cfgs();
}
```

After these steps you are ready to annotate your code!

### Common usages of `pyo3-build-cfg` flags

The following are some common patterns implemented using these flags:

// TODO

## Checking the Python version at runtime

// TODO
2 changes: 1 addition & 1 deletion guide/src/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Currently, [#341](https://github.com/PyO3/pyo3/issues/341) causes `cargo test` t

```toml
[dependencies.pyo3]
version = "{{#PYO3_VERSION}}"
version = "{{#PYO3_CRATE_VERSION}}"

[features]
extension-module = ["pyo3/extension-module"]
Expand Down
2 changes: 2 additions & 0 deletions pyo3-build-config/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
//!
//! The only public API currently exposed is [`use_pyo3_cfgs`], which is intended to be used in
//! build scripts to add a standard set of `#[cfg]` flags for handling multiple Python versions.
//!
//! TODO: tabulate all the flags here

#[allow(dead_code)] // TODO cover this using tests
mod impl_;
Expand Down
2 changes: 1 addition & 1 deletion pyo3-macros-backend/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,4 @@ default-features = false
features = ["derive", "parsing", "printing", "clone-impls", "full", "extra-traits"]

[build-dependencies]
pyo3-build-config = { path = "../pyo3-build-config", version = "0.14.0-alpha.0" }
pyo3-build-config = { path = "../pyo3-build-config", version = "=0.14.0-alpha.0" }

0 comments on commit 825ec08

Please sign in to comment.