Object protocols without specialization #961

kngwyu · 2020-06-06T08:51:25Z

For #210.
Now all pyclass has a method registory for protocols:

impl pyo3::class::proto_methods::HasProtoRegistry for MyClass {
    fn registory() -> &'static pyo3::class::proto_methods::PyProtoRegistry {
        static REGISTRY: PyProtoRegistry = PyProtoRegistry::new();
        &REGISTRY
    }
}

, where PyProtoRegistory is a set of method tables:

#[doc(hidden)]
pub struct PyProtoRegistry {
    async_methods: AtomicPtr<PyAsyncMethods>,
    basic_methods: AtomicPtr<PyObjectMethods>,
    ...
}

.

Using the ctor crate, we insert some initialization codes for this registry.
For example, when we write this code:

#[pyproto]
impl PyObjectProtocol for MyClass {
    fn __hash__(&self) -> PyResult<isize> {
        Ok(self.val as isize)
    }
}

, (roughly) the following code is generated:

#[ctor]
fn __init_Object_2817873() {
    let mut table = pyo3::class::PyObjectMethods::default();
    table.set_hash::<MyClass>();
    <MyClass as pyo3::class::proto_methods::HasProtoRegistry>::registory().set_basic_methods(table);
}

One disadvantage of this solution is it needs redundant Box::new for some protocols(basic, iter, and so on). However, to keep PyProtoRegistry simple and small, I chose this design.

Unfortunately, some protocols are not yet tested.

TODO

Add tests for async protocols
- Maybe we need to change the trait definition to make it work?
Add tests for descr protocols
Maybe need some more doc comments?

kngwyu · 2020-06-06T09:05:39Z

I forgot to mention that this PR has a breaking change. nb_bool is now moved under PyNumberProtocol from PyObjectProtocol.

davidhewitt

Thanks so much for this - it's excellent to see that this code is shorter than what was there before and should unblock us for compiling pyo3 on stable! 🎉

There's a couple things I'd like to see changed:

Typo registory -> registry in several places
Let's fix the Send + Sync bounds in compile_error test for PyClass: Send #948 rather than adding in this PR.

I forgot to mention that this PR has a breaking change. nb_bool is now moved under PyNumberProtocol from PyObjectProtocol.

I understand why you moved this, but I am not sure if it is the right change from a user design.

The related C API is listed in the Object Protocol: https://docs.python.org/3/c-api/object.html?highlight=pyobject_istrue

Does it make the implementation much more complicated if we can keep this in the PyObjectProtocol?

pyo3-derive-backend/src/pyclass.rs

src/class/proto_methods.rs

davidhewitt · 2020-06-07T10:06:20Z

src/class/proto_methods.rs

+        self.async_methods
+            .store(Box::into_raw(Box::new(methods)), Ordering::SeqCst)


Technically all of these could leak memory if called for a second time.

Yes, but the current code does the same. See https://github.com/PyO3/pyo3/blob/v0.10.1/src/pyclass.rs#L151.
The difference is now we cause memory leak for all protocols(e.g., iter and basic).
We can avoid this by changing the pub struct PyProtoRegistry like the below:

pub struct PyProtoRegistry { /// Protocols that needs specific table types async_methods: AtomicPtr<ffi::PyAsyncMethods>, buffer_methods: AtomicPtr<ffi::PyBufferProcs>, mapping_methods: AtomicPtr<ffi::PyMappingMethods>, number_methods: AtomicPtr<ffi::PyNumberMethods>, sequence_methods: AtomicPtr<ffi::PySequenceMethods>, /// Other methods. /// AtomicUsize is used instead of `Atomic<Option<ffi::func>>` or so because /// we can't use function pointer in const function. tp_str: AtomicUsize, tp_repr: AtomicUsize. .... }

.

I chose this design first but abandoned it due to its ugliness.

Cool, I think your design choice is fine. Can we maybe just add a docstring for now to warn calling these twice will leak?

src/class/pyasync.rs

src/instance.rs

src/pycell.rs

kngwyu · 2020-06-07T13:19:41Z

Thank you for the review!

Let's fix the Send + Sync bounds in #948 rather than adding in this PR.

I accidentally pushed these changes when I was playing with them. I'll revert them.

Does it make the implementation much more complicated if we can keep this in the PyObjectProtocol?

I don't have any good solution now, unfortunately.

davidhewitt · 2020-06-07T20:26:27Z

I don't have any good solution now, unfortunately.

👍 ok. I have a half-baked idea but it's hard to write down as I haven't properly thought about it yet. I'll try to find time tomorrow evening to experiment with it whether it can help and explain it better then.

kngwyu · 2020-06-08T00:12:25Z

I have a half-baked idea but it's hard to write down as I haven't properly thought about it yet.

Thank you but wait for a bit...
I don't think most users know that __bool__ is classified as a part of Object protocols.
Thus I don't think it's worth doing if we need more codes to do so.

EDITED:
I just tried to move nb_bool under ObjectProtocol again but the resulting code is not so smart 🤔

    // nb_bool is a part of PyObjectProtocol, but should be placed under tp_as_number
    let mut nb_bool = None;
    // basic methods
    if let Some(basic) = T::basic_methods() {
        unsafe { basic.as_ref() }.update_typeobj(type_object);
        nb_bool = unsafe { basic.as_ref() }.nb_bool;
    }

    // number methods
    type_object.tp_as_number = T::number_methods()
        .map(|mut p| {
            unsafe { p.as_mut() }.nb_bool = nb_bool;
            p.as_ptr()
        })
        .unwrap_or_else(|| {
            if nb_bool.is_some() {
                let mut number_methods = ffi::PyNumberMethods::default();
                number_methods.nb_bool = nb_bool;
                Box::into_raw(Box::new(number_methods))
            } else {
                ptr::null_mut()
            }
        });

Still not sure we should do this...

kngwyu · 2020-06-08T03:06:05Z

At last, I decided to move nb_bool under PyObjectProtocol again.
I don't think the code is very ugly but please check it.

davidhewitt · 2020-06-08T18:16:42Z

pyo3-derive-backend/src/pyproto.rs

+    // E.g., if we have `__add__`, we need to skip `set_radd`.
+    let mut skipped_setters = HashSet::new();


Do you think that we still need skipped_setters once we implement a solution for #844?

pyo3-derive-backend/src/pyproto.rs

src/class/pyasync.rs

davidhewitt

At last, I decided to move nb_bool under PyObjectProtocol again.
I don't think the code is very ugly but please check it.

I think what you've done is nice. It's good to avoid the unnecessary breaking change 😄

This branch is looking good, I'm very excited by it! As well as your TODO for docs and tests I would be interested to hear what you think about how we could solve #844 with this new implementation.

src/class/macros.rs

src/pyclass.rs

davidhewitt · 2020-06-08T18:32:23Z

It would also be good to add a test for #940 in this PR - if I read your implementation correctly I think that should already be solved by what you've done!

kngwyu · 2020-06-13T10:44:19Z

Added a test for __await__.
I was really confused by Python's async/await and noticed some things:

Returning Future from Rust is hard.
To return some value, we have to make StopIterException have the value.

But for now, I think this is sufficient for testing proc-macros.

kngwyu · 2020-06-13T11:22:27Z

OMG...

(Windows + Python3.8)

tests/test_dunder.rs

madhavajay · 2020-06-15T07:14:01Z

🍿🤓

kngwyu · 2020-06-15T14:42:24Z

I would be interested to hear what you think about how we could solve #844 with this new implementation.

I think we can reuse the same trick as __set/delitem__ for that, but maybe it should be done in another PR 🤔

davidhewitt · 2020-06-15T19:18:25Z

I think we can reuse the same trick as set/delitem for that, but maybe it should be done in another PR 🤔

I'm happy with that - this PR is already quite large 😄

kngwyu · 2020-06-17T03:46:55Z

Now I think this PR is ready.
Though I added more comments than usual, please point out what you think is unclear.

davidhewitt

This is looking fantastic to me! +1,373 / -2,245 - nice job 🚀

davidhewitt · 2020-06-17T08:04:26Z

pyo3-derive-backend/src/pyproto.rs

@@ -76,7 +82,7 @@ fn impl_proto_impl(
                } else {
                    quote!(0)
                };
-                // TODO(kngwyu): doc
+                // TODO(kngwyu): Set ml_doc


Let's create an issue for this so that we remember to do it?

konstin · 2020-06-17T08:39:22Z

The test_await was failing for me locally (python3.6 from apt in virtualenv), this is the workaround I used to make them pass.

Traceback (most recent call last):
  File "<string>", line 12, in <module>
  File "/usr/lib/python3.6/asyncio/events.py", line 694, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "/usr/lib/python3.6/asyncio/events.py", line 602, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Dummy-1'.
test test_await ... FAILED

diff --git a/tests/test_dunder.rs b/tests/test_dunder.rs
index 67a66fb49..47f22fc91 100644
--- a/tests/test_dunder.rs
+++ b/tests/test_dunder.rs
@@ -612,7 +612,11 @@ async def main():
 # but we see still errors on Github actions...
 if sys.platform == "win32" and sys.version_info >= (3, 8, 0):
     asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
-loop = asyncio.get_event_loop()
+try:
+    loop = asyncio.get_event_loop()
+except RuntimeError: # RuntimeError: There is no current event loop in thread 'Dummy-1'.
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
 assert loop.run_until_complete(main()) is None
 loop.close()
 "#

kngwyu · 2020-06-17T13:17:58Z

@konstin
Thanks, I pushed the change based on your diff.

davidhewitt · 2020-06-18T08:20:35Z

🎉

davidhewitt requested changes Jun 7, 2020

View reviewed changes

davidhewitt mentioned this pull request Jun 7, 2020

Add nightly-optimizations feature #927

Closed

1 task

davidhewitt reviewed Jun 8, 2020

View reviewed changes

davidhewitt approved these changes Jun 8, 2020

View reviewed changes

src/class/macros.rs Show resolved Hide resolved

src/pyclass.rs Show resolved Hide resolved

konstin mentioned this pull request Jun 9, 2020

Stable Rust #969

Merged

kngwyu added 6 commits June 13, 2020 15:24

Remove specialization from basic/buffer/descr/iter protocols

7967874

Remove specialization from gc/mapping/number protocols

0d08296

Remove specialization from async/sequence methods

d678093

Avoid function definition in setters for readability

05effe2

Fix typo and revert unintended changes

ac2c51c

Move nb_bool under PyObjectProtocol again

f322771

kngwyu force-pushed the slot-provider branch 2 times, most recently from 1d6295c to ed31c70 Compare June 13, 2020 10:42

kngwyu force-pushed the slot-provider branch from ed31c70 to 280ce0b Compare June 13, 2020 11:05

kngwyu force-pushed the slot-provider branch from 280ce0b to 78577c6 Compare June 13, 2020 12:19

davidhewitt reviewed Jun 14, 2020

View reviewed changes

tests/test_dunder.rs Show resolved Hide resolved

tests/test_dunder.rs Outdated Show resolved Hide resolved

tests/test_dunder.rs Outdated Show resolved Hide resolved

kngwyu force-pushed the slot-provider branch from 78577c6 to c8fae71 Compare June 14, 2020 13:40

Async methods now takes Receiver

8c4162d

kngwyu force-pushed the slot-provider branch from c8fae71 to 8c4162d Compare June 14, 2020 13:43

Descr get/set now take receiver

2e1ed2c

kngwyu added 2 commits June 17, 2020 02:17

Use Relaxed ordering since setters are called by a single thread

f7fa008

Mark protocols setters as doc(hidden)

2a69367

Add more comments for #[pyproto] related parts

fc3fa70

kngwyu force-pushed the slot-provider branch from 245dc53 to fc3fa70 Compare June 17, 2020 04:21

davidhewitt approved these changes Jun 17, 2020

View reviewed changes

Make test_await more robust

1207c2d

kngwyu force-pushed the slot-provider branch from da381e5 to 1207c2d Compare June 17, 2020 16:58

kngwyu added 2 commits June 18, 2020 15:49

Merge branch 'master' into slot-provider

a044f3c

Add a CHANGELOG entry for 961

8a0c9bf

kngwyu merged commit 390ff5f into PyO3:master Jun 18, 2020

kngwyu deleted the slot-provider branch June 18, 2020 08:19

kngwyu mentioned this pull request Jun 23, 2020

Cannot return option from PySequenceProtocol __getitem__ #798

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Object protocols without specialization #961

Object protocols without specialization #961

kngwyu commented Jun 6, 2020 •

edited

kngwyu commented Jun 6, 2020

davidhewitt left a comment

davidhewitt Jun 7, 2020

kngwyu Jun 7, 2020 •

edited

davidhewitt Jun 7, 2020

kngwyu commented Jun 7, 2020

davidhewitt commented Jun 7, 2020

kngwyu commented Jun 8, 2020 •

edited

kngwyu commented Jun 8, 2020

davidhewitt Jun 8, 2020

davidhewitt left a comment

davidhewitt commented Jun 8, 2020

kngwyu commented Jun 13, 2020 •

edited

kngwyu commented Jun 13, 2020

madhavajay commented Jun 15, 2020

kngwyu commented Jun 15, 2020

davidhewitt commented Jun 15, 2020

kngwyu commented Jun 17, 2020

davidhewitt left a comment

davidhewitt Jun 17, 2020

konstin commented Jun 17, 2020

kngwyu commented Jun 17, 2020

davidhewitt commented Jun 18, 2020

		self.async_methods
		.store(Box::into_raw(Box::new(methods)), Ordering::SeqCst)

		// E.g., if we have `__add__`, we need to skip `set_radd`.
		let mut skipped_setters = HashSet::new();

Object protocols without specialization #961

Object protocols without specialization #961

Conversation

kngwyu commented Jun 6, 2020 • edited

TODO

kngwyu commented Jun 6, 2020

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt Jun 7, 2020

Choose a reason for hiding this comment

kngwyu Jun 7, 2020 • edited

Choose a reason for hiding this comment

davidhewitt Jun 7, 2020

Choose a reason for hiding this comment

kngwyu commented Jun 7, 2020

davidhewitt commented Jun 7, 2020

kngwyu commented Jun 8, 2020 • edited

kngwyu commented Jun 8, 2020

davidhewitt Jun 8, 2020

Choose a reason for hiding this comment

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt commented Jun 8, 2020

kngwyu commented Jun 13, 2020 • edited

kngwyu commented Jun 13, 2020

madhavajay commented Jun 15, 2020

kngwyu commented Jun 15, 2020

davidhewitt commented Jun 15, 2020

kngwyu commented Jun 17, 2020

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt Jun 17, 2020

Choose a reason for hiding this comment

konstin commented Jun 17, 2020

kngwyu commented Jun 17, 2020

davidhewitt commented Jun 18, 2020

kngwyu commented Jun 6, 2020 •

edited

kngwyu Jun 7, 2020 •

edited

kngwyu commented Jun 8, 2020 •

edited

kngwyu commented Jun 13, 2020 •

edited