Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release: 0.14.0 #1698

Merged
merged 1 commit into from Jul 3, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
48 changes: 35 additions & 13 deletions Architecture.md
Expand Up @@ -17,20 +17,22 @@ protocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e.,
Since implementing `PyClass` requires lots of boilerplate, we have a proc-macro `#[pyclass]`.

To summarize, there are six main parts to the PyO3 codebase.

1. [Low-level bindings of Python/C API.](#1-low-level-bindings-of-python-capi)
- [`src/ffi`]
- [`src/ffi`]
2. [Bindings to Python objects.](#2-bindings-to-python-objects)
- [`src/instance.rs`] and [`src/types`]
- [`src/instance.rs`] and [`src/types`]
3. [`PyClass` and related functionalities.](#3-pyclass-and-related-functionalities)
- [`src/pycell.rs`], [`src/pyclass.rs`], and more
- [`src/pycell.rs`], [`src/pyclass.rs`], and more
4. [Protocol methods like `__getitem__`.](#4-protocol-methods)
- [`src/class`]
- [`src/class`]
5. [Procedural macros to simplify usage for users.](#5-procedural-macros-to-simplify-usage-for-users)
- [`src/derive_utils.rs`], [`pyo3-macros`] and [`pyo3-macros-backend`]
- [`src/derive_utils.rs`], [`pyo3-macros`] and [`pyo3-macros-backend`]
6. [`build.rs`](#6-buildrs)
- [`build.rs`](https://github.com/PyO3/pyo3/tree/master/build.rs)
- [`build.rs`](https://github.com/PyO3/pyo3/tree/master/build.rs)

## 1. Low-level bindings of Python/C API

[`src/ffi`] contains wrappers of [Python/C API].

We aim to provide straight-forward Rust wrappers resembling the file structure of
Expand All @@ -44,24 +46,27 @@ In the [`src/ffi`] module, there is lots of conditional compilation such as `#[c
`#[cfg(Py_37)]`, and `#[cfg(PyPy)]`.
`Py_LIMITED_API` corresponds to `#define Py_LIMITED_API` macro in Python/C API.
With `Py_LIMITED_API`, we can build a Python-version-agnostic binary called an
[abi3 wheel](https://pyo3.rs/v0.13.2/building_and_distribution.html#py_limited_apiabi3).
[abi3 wheel](https://pyo3.rs/main/building_and_distribution.html#py_limited_apiabi3).
`Py_37` means that the API is available from Python >= 3.7.
There are also `Py_38`, `Py_39`, and so on.
`PyPy` means that the API definition is for PyPy.
Those flags are set in [`build.rs`](#6-buildrs).

## 2. Bindings to Python objects

[`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html)
of Python, such as `dict` and `list`.
For historical reasons, Python's `object` is called `PyAny` in PyO3 and located in [`src/types/any.rs`].
Currently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as:

```rust
#[repr(transparent)]
pub struct PyAny(UnsafeCell<ffi::PyObject>);
```

All built-in types are defined as a C struct.
For example, `dict` is defined as:

```c
typedef struct {
/* Base object */
Expand All @@ -77,6 +82,7 @@ typedef struct {

However, we cannot access such a specific data structure with `#[cfg(Py_LIMITED_API)]` set.
Thus, all builtin objects are implemented as opaque types by wrapping `PyAny`, e.g.,:

```rust
#[repr(transparent)]
pub struct PyDict(PyAny);
Expand All @@ -92,12 +98,14 @@ Since we need lots of boilerplate for implementing common traits for these types
[`src/types/mod.rs`].

## 3. `PyClass` and related functionalities

[`src/pycell.rs`], [`src/pyclass.rs`], and [`src/type_object.rs`] contain types and
traits to make `#[pyclass]` work.
Also, [`src/pyclass_init.rs`] and [`src/pyclass_slots.rs`] have related functionalities.

To realize object-oriented programming in C, all Python objects must have the following two fields
at the beginning.

```rust
#[repr(C)]
pub struct PyObject {
Expand All @@ -106,16 +114,19 @@ pub struct PyObject {
...
}
```

Thanks to this guarantee, casting `*mut A` to `*mut PyObject` is valid if `A` is a Python object.

To ensure this guarantee, we have a wrapper struct `PyCell<T>` in [`src/pycell.rs`] which is roughly:

```rust
#[repr(C)]
pub struct PyCell<T: PyClass> {
object: crate::ffi::PyObject,
inner: T,
}
```

Thus, when copying a Rust struct to a Python object, we first allocate `PyCell` on the Python heap and then
move `T` into it.
Also, `PyCell` provides [RefCell](https://doc.rust-lang.org/std/cell/struct.RefCell.html)-like methods
Expand All @@ -131,6 +142,7 @@ For example, you can see `type({})` shows `dict` and `type(type({}))` shows `typ
`T: PyTypeObject` implies that `T` has a corresponding type object.

## 4. Protocol methods

Python has some built-in special methods called dunder, such as `__iter__`.
They are called [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in
Python/C API.
Expand All @@ -147,40 +159,50 @@ Also, [`src/class/methods.rs`] has utilities for `#[pyfunction]` and [`src/class
some internal tricks for making `#[pyproto]` flexible.

## 5. Procedural macros to simplify usage for users.

[`pyo3-macros`] provides six proc-macro APIs: `pymodule`, `pyproto`, `pyfunction`, `pyclass`,
`pymethods`, and `#[derive(FromPyObject)]`.
[`pyo3-macros-backend`] has the actual implementations of these APIs.
[`src/derive_utils.rs`] contains some utilities used in code generated by these proc-macros,
such as parsing function arguments.

## 6. `build.rs`

PyO3's [`build.rs`](https://github.com/PyO3/pyo3/tree/master/build.rs) is relatively long
(about 900 lines) to support multiple architectures, interpreters, and usages.
Below is a non-exhaustive list of its functionality:

- Cross-compiling support.
- If `TARGET` architecture and `HOST` architecture differ, we find cross compile information
from environment variables (`PYO3_CROSS_LIB_DIR` and `PYO3_CROSS_PYTHON_VERSION`) or system files.
from environment variables (`PYO3_CROSS_LIB_DIR` and `PYO3_CROSS_PYTHON_VERSION`) or system files.
- Find the interpreter for build and detect the Python version.
- We have to set some version flags like `Py_37`.
- If the interpreter is PyPy, we set `PyPy`.
- If `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed
entirely and only abi3 extensions can be built.
entirely and only abi3 extensions can be built.
- Check if we are building a Python extension.
- If we are building an extension (e.g., Python library installable by `pip`),
we don't link `libpython`.
Currently we use the `extension-module` feature for this purpose. This may change in the future.
See [#1123](https://github.com/PyO3/pyo3/pull/1123).
we don't link `libpython`.
Currently we use the `extension-module` feature for this purpose. This may change in the future.
See [#1123](https://github.com/PyO3/pyo3/pull/1123).

<!-- External Links -->
[Python/C API]: https://docs.python.org/3/c-api/

[python/c api]: https://docs.python.org/3/c-api/

<!-- Crates -->

[`pyo3-macros`]: https://github.com/PyO3/pyo3/tree/master/pyo3-macros
[`pyo3-macros-backend`]: https://github.com/PyO3/pyo3/tree/master/pyo3-macros-backend

<!-- Directories -->

[`src/class`]: https://github.com/PyO3/pyo3/tree/master/src/class
[`src/ffi`]: https://github.com/PyO3/pyo3/tree/master/src/ffi
[`src/types`]: https://github.com/PyO3/pyo3/tree/master/src/types

<!-- Files -->

[`src/derive_utils.rs`]: https://github.com/PyO3/pyo3/tree/master/src/derive_utils.rs
[`src/instance.rs`]: https://github.com/PyO3/pyo3/tree/master/src/instance.rs
[`src/pycell.rs`]: https://github.com/PyO3/pyo3/tree/master/src/pycell.rs
Expand Down