How can we make Python's C-API more friendly for Rust?

[encukou]

Hello,
I'm figuring out a strategy for the future of Python's C-API, and as one of the goals I'd like to make it friendly for non-C languages. But since I mostly deal with C & Python, I'd like to ask you for any suggestions, pain points or comments on how to better do that.
Would you like to help?

I started a Google doc to collect notes, please feel free to comment there, or here, or suggest a better place.
Thank you for your time!

[davidhewitt]

Thank you for thinking of us and reaching out! I spent some time this evening trying to share a few of my thoughts, please feel free to ask me to elaborate on anything which you think is not well explained.

[davidhewitt]

@encukou I've been thinking a little about sub-interpreters and PEP 489 compatibility for PyO3. I have the impression this is the way concurrency in Python is planned to evolve. As expressed in that Google doc, I'm concerned this is going to be a challenge for PyO3 to support.

Am I right that at the C-API level, there will be no protections against sharing Python objects between sub-interpreters, and the rule is simply "don't do this, because refcounting (etc.) is not thread-safe"? If so, then this would make associating Python objects to a GIL an unsafe operation in Rust semantics. That would imply PyO3's public API may have to undergo breaking changes to support PEP 489, and may end up with a significant set of unsafe APIs.

(Many Rust projects forbid usage of unsafe APIs and leave that to lower-level projects. For example, one of the key goals of PyO3 is to provide a safe abstraction for Rust projects to interact with Python.)

It may be that PyO3 ends up needing to opt-out of PEP 489 to give users a safe API surface. PyO3 and Rust projects may eventually become an obstacle to wider usage of sub-interpreters in the ecosystem if so.

Some help in thinking through this would be appreciated. One way in which I could see this working, for example, would be to take some ideas of the nogil project and make refcounting and builtin types like dict thread-safe, so that sharing objects between sub-interpreters became safe. I understand that would be a major change to CPython which is unlikely to happen, so alternative ideas and research may be necessary. (Probably on PyO3's side, though if there are other ideas to guard against sharing of objects between sub-interpreters in CPython that would help us solve our problems.)

[adamreichold]

From a first look at this, maybe CPython's object header could contain an immutable sub interpreter ID which could be checked when entering a subinterpreter and then abort the process. This would not be particularly ergonomic, but at least prevent unsafe. Overhead could of course be prohibitive, especially as the branches will never be taken but still weigh down correct code. So maybe, CPython could fill that in as a courtesy and only PyO3 (and other safe bindings) could use it when associating an object with a GIL, so I guess when calling as_ref? (I also suspect that this would mean that our GIL token Python would not be zero-sized any more as it would need to identify the actual subinterpreter? So another source of overhead.)

[encukou]

Hi,
Sorry for the delay -- right after I asked, “life happened” (literally 👶). I'm back now.

I'm not sure if we're talking about the same thing.
Multiple interpreters are already part of Python (see Py_NewInterpreter). There are efforts to make things safer/more isolated, expose a Python API to create them, make sharing objects easier, and even split up the GIL, but all of these are at various stages, so it's not clear what exactly you're referring to. For example, I don't know what “entering a subinterpreter” should mean. (I don't mean to blame -- I understand that it's confusing!)

Sharing Python objects between interpreters is generally a bad idea. You can do it in some cases if you know what you're doing, but it's very convenient to just tell people not to do it. Some issues are:

• anything that references interpreter state, even indirectly (e.g. a function object, which has a ref to its globals and builtins, which might contain e.g. sys.modules), will cause chaos if exposed to a different interpreter
• if (if!) per-interpreter GIL is implemented in the future, sharing mutable objects will be restricted -- but the specifics would depend on the implementation.

Keep in mind that Python also allows running multiple interpreters “in series”: calling Py_Finalize and then Py_Initialize again. If any objects “survive” this, they'll have many of the same issues as when shared between “parallel” interpreters.

Instead of associating objects to a GIL or interpreter, I recommend associating them to a module object.
If you haven't already, I recommend reading my summary PEP 630. Let me know if anything is unclear or questionable.
I don't know much about PyO3, so PEP 630 might not apply at all -- but if so, I'd like to know about that!

[davidhewitt]

Sorry for the delay -- right after I asked, “life happened” (literally 👶). I'm back now.

Congratulations! I'm in the same place - my little one just made it to 3 months old! 👶 Sorry for being slow on my side too! Hope you and your family are getting on ok.

To attempt to describe where I'm coming from: in Rust it's usually goal of libraries like PyO3 to provide a "safe" abstraction so that users who don't know what they're doing can produce sound code.

Sharing Python objects between interpreters is generally a bad idea. You can do it in some cases if you know what you're doing, but it's very convenient to just tell people not to do it.

Translating this statement to Rust semantics, this means that any API which permits sharing of Python objects between interpreters is unsafe.

Continuing that line, this means that PyO3's API should be designed so that sharing objects between interpreters is impossible except by using unsafe APIs. However, here's a snippet of code that's valid with PyO3 today using only safe APIs:

use parking_lot::{Mutex, const_mutex};

pub static GLOBAL: Mutex<Option<PyObject>> = const_mutex(None);

#[pyfunction]
fn read_global() -> PyObject {
    GLOBAL.lock()
        .get_or_insert_with(/* some function to init the global, doesn't really matter what this is */)
        .clone()
}

This snippet stores a PyObject in a static variable. If the read_global #[pyfunction] is called from two separate interpreters, this will result in this object being accessible from both interpreters. Hence, something about this above snippet is unsound; in Rust semantics there should be use of the unsafe keyword somewhere in this snippet, but there isn't, so this must be a flaw in PyO3's API.

Now, I agree that this is already potentially a problem today because multiple parallel interpreters are already a thing. However, they're not widely used, so we have not run into significant problems yet.

I agree that PyO3 should be following the approach in PEP 630. However, it is not a trivial internal refactoring - we likely need to redesign PyO3's external API quite extensively to make snippets like the above impossible. It's not clear that this is achievable.

Keep in mind that Python also allows running multiple interpreters “in series”: calling Py_Finalize and then Py_Initialize again. If any objects “survive” this, they'll have many of the same issues as when shared between “parallel” interpreters.

I am aware of this, and we have unsafe fn with_embedded_python_interpreter which already gives a nod to this using the unsafe keyword.

[encukou]

Hmm... How does Rust ensure this object is deleted before interpreter shutdown? And not accesed before Py_Initialize?

they're not widely used, so we have not run into significant problems yet.

That sounds like C's kind of “safe”, not Rust's </rant>

we have unsafe fn with_embedded_python_interpreter

That's great for software that embeds Python, but doesn't do much for extensions, where it's other code calls Py_Finalize/Py_Initialize.

[davidhewitt]

The TLDR of the above is that I think PyO3 needs one of the two following things to make itself safe in the presence of multiple interpreters:

1. Extensive redesign to make it impossible to store PyObject references in static data, or
2. Have an efficient way to check at runtime which interpreter and object is associated with.

(1) would make PyO3 sound at the type system level.
(2) would make PyO3 sound by allowing it to raise errors when objects are incorrectly crossing between interpreters.

The extensive redesign seems intractable, so I wonder if you think it could be possible to implement a way in the Python C API to verify if an object belongs to the current interpreter?

[encukou]

Sadly, I doubl (2) can be done in CPython without adding extra data on each object, which would meed too much extra memory. Also, where would this be called on the Rust side? On each interaction with the API? Or when retreiving a static object (where it might be possibe to have a interpreter→object map)?

I guess another way is to embrace the current constraints, and make sure PyO3 is only usable in one interpreter per process. If there's an init function, make it fail the second time it's called. Or record the current interpreter at startup and check it's the only one used from Rust.

[davidhewitt]

Sorry for dropping off this thread for such a long time, little one and other threads have been keeping me busy. Hopefully you and your family are also doing ok.

I've opened #2523 to raise ImportError if a PyO3 module is initialised multiple times - thanks for recommending that.

Wrt (2) - agreed that the overhead on every object would be too much. What if the interpreter id was recorded in the type objects? I think that would probably be sufficient to know which interpreter an object should be associated with.

[encukou]

No worries, I also struggle with finding time :)

The problem with types is that some types can be – and are – shared between interpreters. And it looks like the current plan toward per-interpreter GIL is not to have per-interpreter core types, but to make core type objects immortal, skip reference counting on them entirely, and move mutable bits (like __subclasses__) to per-interpreter state. (Most singletons – static pointers` – the API currently exposes are type objects, so keeping them that way avoids lots of API breakage.)

But let me ask again (either the question is important or I'm misunderstanding something): if you had the (2) check, where would you call it? On every interaction with an object? Or only when retrieving PyObject from static storage?
What would you do if the check failed?

[encukou]

There is now a wider effort for mapping out C API issues at: https://github.com/capi-workgroup/problems/issues