Add weakref Python types #3835

SuperJappie08 · 2024-02-14T11:04:18Z

#3134 Add PyWeakRef, PyWeakProxy and PyWeakCallableProxy types, corresponding to Python's weakref.ReferenceType, weakref.ProxyType and weakref.CallableProxyType.

As a result of the checks made about the objectpointers in PyO3, all errors in PyWeakref_GetObject are already caught. Therefor there is no need to check for these errors in the non-ffi layer.

SuperJappie08 · 2024-02-14T11:17:39Z

There are some issues with the new run_bound.
I will resolve these issues now.

codspeed-hq · 2024-02-14T11:21:55Z

CodSpeed Performance Report

Merging #3835 will not alter performance

_{Comparing SuperJappie08:weakref (c7ab375) with main (b11174e)}

Summary

✅ 69 untouched benchmarks

davidhewitt · 2024-02-22T01:00:00Z

Thank you and sorry for the slow review, I will try my best to finally look at this tomorrow...

SuperJappie08 · 2024-02-22T08:46:00Z

Thank you and sorry for the slow review, I will try my best to finally look at this tomorrow...

No problem, completely understandable.

I also ran into one last issue, PyPy does not include all the required bindings for the weakref module in their C-api.
At least the bindings required in the way it is done now.

The type-objects for the reference types are not exported (not bound to in pyo3-ffi and I can't find them in the PyPy header files). Also, the memory layout is missing, since in PyPy weakref is not actually a binary module.
That last one should at least be solvable, but I cannot think of a solution for the missing type objects

davidhewitt

Thank you very much for this!

It looked like a big diff but that's because you've included all docs and tests, thank you 🙏

I've given this a full read; I have a few initial things which need discussing and changing. I think once those are decided upon & done I might need to take another final read through just due to the size of the diff.

.gitignore

src/types/weakref/reference.rs

src/types/weakref/proxy.rs

davidhewitt · 2024-02-23T08:36:17Z

Regarding PyPy, for now I suggest just not offering these types on PyPy.

…unted for

SuperJappie08 · 2024-03-08T18:03:20Z

@davidhewitt, I'm running into an issue where code compiled with abi-py37 does not enable PyClasses to be weakreferencable even if marked as so.

It does work when compiling against Python 3.7 without abi3-py37

Do you have any idea what is going on?

SuperJappie08 · 2024-03-09T17:02:05Z

https://github.com/PyO3/pyo3/blob/main/src%2Fpyclass%2Fcreate_type_object.rs#L311-L370

When compiling abi3 on Python versions lower than 3.9 it does not write any offsets to the type object.

Could this be the origin of the problem?

davidhewitt · 2024-03-09T18:03:52Z

IIRC this is a known limitation, we should probably make the pyclass macro fail in this case. For this PR maybe it's good enough to skip the tests in the configuration.

…and 3.8

davidhewitt

Thanks for pushing this forward, it seems like it's nearly there now. I added some further thoughts. I was a little slow to review as the format checks are failing so I got a bit lots on the status of this.

src/types/weakref/callableproxy.rs

src/types/weakref/reference.rs

davidhewitt · 2024-03-23T23:01:16Z

src/types/weakref/callableproxy.rs

+            assert!(
+                object.is_callable(),
+                "An object to be referenced by a PyWeakCallableProxy should be callable. Use PyWeakProxy instead."
+            );


I wonder, given this footgun here and in PyWeakProxy constructors, should we consider merging these types into just PyWeakProxy? I guess the downside is that we don't match the C API so well, but maybe it works out as more ergonomic? I'm open to hearing your thoughts on this one, there might be very good reasons to keep them separate.

I agree that it is a real footgun. Merging these types wouldn't be that weird, since the Python-side API doesn't really differentiate between them. And the C API provides a check function which globs them together.

My suggestion may be the following:

Rename PyWeakProxy to PyWeakNonCallableProxy

Create a new PyWeakProxy which can hold an enum to abstract over PyWeakNonCallableProxy and PyWeakCallableProxy

This would (hopefully) allow for specific use of one specific type (if desired), while preventing the major footgun (as a result of the Python API design) for most users which would use the PyWeakProxy. (I referred to this as PyWeakProxyGeneric or PyWeakGenericProxy in #3134 (#3134 (comment)))

Looking at the type definitions, it seems that the check for both of these is PyWeakref_CheckProxy. I think this means that it gets hard to distinguish these two types from Rust.

What use cases do you envision for the specific types from the enum? If we can't think of any right now, one option might be to slim this type down a bit and just implement PyTypeCheck instead of PyTypeInfo, i.e. more like PyNone.

(and have one single PyWeakProxy covering both callable and non-callable types.)

That would probably make, the most sense.
I also recently figured out that the weakref type objects are actually part of the internal/private API, so they cannot be expected to be consistent in ABI3 or between patch versions, but it has been quite stable as far as I can see.

So maybe it would be good to move PyWeakRef also to the non-typeobject implementation, at least for the ABI3 target. We could still use the Type object pointer if we are compiling for a specific version, since it appears fairly stable(, while it is not guaranteed).

This would also allow work for PyPy, since they do export the Check functions.

An option I could for see is that we would follow the check functions:

PyWeak... (Maybe PyWeakAnyRef?): Any Weakreference type, using PyWeakref_Check

PyWeakRef: weakref.ReferenceType using PyWeakref_CheckRef for the specific normal reference type

PyWeakProxy: Any weak Proxy type (Callable or Not) using PyWeakref_CheckProxy.

This would align with the C API, and probably with the way these type will be used, since I can imagine that a function could take a weakreference, without caring what the exact reference type would be, since it does not matter (that much) when using it from the C API.
And the bonus point that we wouldn't have to exclude PyPy.

Sorry for the delay here by me yet again. I agree with the suggestion to move them all to the non-typeobject implementation. In general I'm trying hard to make PyO3 not need to depend on CPython private APIs.

We can always expand the implementation later if CPython makes the APIs public.

As for naming, I guess the one that will be used most commonly is "any weak rerference type", so let's call that one PyWeakref maybe? We could then have PyWeakrefReference and PyWeakrefProxy for the specific subtypes.

So:

PyWeakref

PyWeakrefReference

PyWeakrefProxy

How does that sound to you?

The naming makes sense.

However I'm having some trouble doing the type objectless implementation.
You mentioned PyNone and PyNotImplemented as an example implementation however those are more like 'singleton' objects.

There is probably some way to do it and I want to figure it out. But do you have a hint?

(I also will not have much free time in the coming 3 weeks, so it is going to take a bit longer.)

Edit: After reading your previous comments carefully I could give it another go.

I have done it for PyWeakref and PyWeakrefProxy and renamed PyWeakRef to PyWeakrefReference.

I still need to make the changes for PyWeakrefReference; however, I have 2 findings:

PyWeakrefReference is quite a long name, is this very practical to use or might PyWeakrefRef be better. I like the longer name for the clarity, but it feels clunky due to the length.

PyWeakrefReference (or Pythons weakref.ref) is subclassable (The only subclassable weakref type). However, this non-typepointer implementation would not allow for subclassing. The use case of subclassing is to add extra data to the weakreference. Python weakref documentation mentioning this

Changing the implementation of PyWeakrefReference would mean no subclassing from PyO3 (with the cost of using an internal API, which has been available since Python 2.2 (and probably will remain available since they are used in the check macros)), but making the change would allow for alternative interpreter support.

What do you think?

@davidhewitt, The solution I thought pushed now does the following:
Only PyWeakrefReference uses the unstable API, only if you are compiling against a specific CPython version.
This keeps it subclassable, while allowing support for the other interpreters and the Limited API.

src/types/weakref/reference.rs

SuperJappie08 · 2024-04-18T16:04:43Z

I think all the (current) feedback has been processed.

SuperJappie08 · 2024-05-07T11:24:47Z

@davidhewitt, Would you have time to review the PR?
Because I think it is nearly finished.
I'm still happy to make changes if necessary, but I am just afraid this PR is being buried (probably by more important PR's, which is fine) .

SuperJappie08 added 11 commits February 8, 2024 11:54

Add vscode folder to gitignore

cb75d28

Initial work on PyWeakRef (weakref.ReferenceType)

da6fa21

Add documentation for PyWeakRef::upgrade

f84d6d2

Add missing docs for PyWeakRef

b696f33

Add PyWeakProxy

880fab7

Add PyWeakCallableProxy

62616a0

Merge branch 'PyO3:main' into weakref

123077b

Add PyWeakRefMethods::upgrade_exact and prevent unnecessary panicing

0d5667b

Change PyWeakRefMethods to exclude infeasible errors.

5508d3a

As a result of the checks made about the objectpointers in PyO3, all errors in PyWeakref_GetObject are already caught. Therefor there is no need to check for these errors in the non-ffi layer.

Add towncrier changes

f18ba78

Merge branch 'PyO3:main' into weakref

b5fb139

SuperJappie08 added 3 commits February 14, 2024 12:28

Update weakref type doctests to use py.run_bound

ee95efb

Fix to adhere to MSRV

d28da4e

Make weakref tests independent from macros feature

dff0914

SuperJappie08 force-pushed the weakref branch from d24fc7c to dff0914 Compare February 15, 2024 17:22

Change Weakref tests

e06e383

Remove forgotten Debug marcos

f140841

davidhewitt reviewed Feb 23, 2024

View reviewed changes

SuperJappie08 added 7 commits March 8, 2024 15:15

Processed .gitignore and PyResultExt feedback

cd056af

Change new methods of weakref types to remove dangling pointers

130c800

Merge branch 'main' into weakref

0b097fc

Change to reflect deprecation of PyErr::value for PyErr::value_bound

dd3ad8e

Change Tests so different class name in older python versions is acco…

efdd0ff

…unted for

Change formatting

972c168

Make tests ABI3 compatible

30c4742

davidhewitt mentioned this pull request Mar 10, 2024

Add error messages when compiling for ABI3 with unsupported macro features #3945

Open

SuperJappie08 added 8 commits March 11, 2024 12:42

Prevent the use of PyClass in test for weakref under abi3 Python 3.7 …

3cc55c9

…and 3.8

Disable weakref types when targeting PyPy

a570f00

Remove needless borrow from CallableProxy test

dc40185

Add Borrowed variants of upgrade and upgrade exact to trait

513ab14

Added tests for weakref borrow_upgrade methods

1ad66e8

Change PyWeakRefMethods method names to be more consistent

ede188a

Change weakref constructors to take PyAny for main target

4a0a0a9

Add track_caller to all panicing weakref methods

f2404e0

davidhewitt reviewed Mar 23, 2024

View reviewed changes

SuperJappie08 added 9 commits March 25, 2024 18:29

Add PyWeakRefMethods::upgrade*_as_unchecked

3778161

Fix PyWeakProxy and PyWeakCallableProxy Documentation

b4007bd

Replace deprecated wrap_pyfunction with bound equivalent

6f2aaf1

Merge branch 'PyO3:main' into weakref

142ab29

Add (Generic) PyWeakref Type

a45c043

Reworked Proxy types into one (PyWeakrefProxy)

da776ed

Rename PyWeakRef to PyWeakrefReference

d1bf16c

Merge branch 'PyO3:main' into weakref

97d73e9

Change PyWeakrefReference to only use type pointer when it exists

c7ab375

SuperJappie08 requested a review from davidhewitt April 18, 2024 16:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add weakref Python types #3835

Add weakref Python types #3835

SuperJappie08 commented Feb 14, 2024

SuperJappie08 commented Feb 14, 2024

codspeed-hq bot commented Feb 14, 2024 •

edited

davidhewitt commented Feb 22, 2024

SuperJappie08 commented Feb 22, 2024

davidhewitt left a comment

davidhewitt commented Feb 23, 2024

SuperJappie08 commented Mar 8, 2024 •

edited

SuperJappie08 commented Mar 9, 2024

davidhewitt commented Mar 9, 2024

davidhewitt left a comment

davidhewitt Mar 23, 2024

SuperJappie08 Mar 25, 2024 •

edited

davidhewitt Mar 26, 2024

davidhewitt Mar 26, 2024

SuperJappie08 Mar 27, 2024

davidhewitt Apr 3, 2024

SuperJappie08 Apr 3, 2024 •

edited

SuperJappie08 Apr 5, 2024

SuperJappie08 Apr 18, 2024

SuperJappie08 commented Apr 18, 2024

SuperJappie08 commented May 7, 2024

Add weakref Python types #3835

Are you sure you want to change the base?

Add weakref Python types #3835

Conversation

SuperJappie08 commented Feb 14, 2024

SuperJappie08 commented Feb 14, 2024

codspeed-hq bot commented Feb 14, 2024 • edited

Merging #3835 will not alter performance

Summary

davidhewitt commented Feb 22, 2024

SuperJappie08 commented Feb 22, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt commented Feb 23, 2024

SuperJappie08 commented Mar 8, 2024 • edited

SuperJappie08 commented Mar 9, 2024

davidhewitt commented Mar 9, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SuperJappie08 Mar 25, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SuperJappie08 Apr 3, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SuperJappie08 commented Apr 18, 2024

SuperJappie08 commented May 7, 2024

codspeed-hq bot commented Feb 14, 2024 •

edited

SuperJappie08 commented Mar 8, 2024 •

edited

SuperJappie08 Mar 25, 2024 •

edited

SuperJappie08 Apr 3, 2024 •

edited