Allow more methods to take interned arguments #2312

mejrs · 2022-04-17T23:39:14Z

I have yet to add some docs to these methods
Changelog

mejrs · 2022-04-17T23:42:13Z

src/types/string.rs

+impl<'a> IntoPy<Py<PyString>> for &'a str {
+    #[inline]
+    fn into_py(self, py: Python<'_>) -> Py<PyString> {
+        PyString::new(py, self).into()
+    }
+}
+


This is a (mild) breaking change because of inference errors, but not too bad.

Just to make sure I understand things correctly: The comment is informational as the other changes are breaking anyway and this is therefore targeting 0.17.0?

Is the previous instance into PyObject even useful anymore?

Is the previous instance into PyObject even useful anymore?

I think so as Py<PyString> does not automatically "decay" into Py<PyAny> and the existing impl might be used in generic context to produce PyObject for stashing away.

Easy to get PyObject from a Py though.

Easy to get PyObject from a Py though.

I think that in generic context, one needs to be able to express this in a single trait bound, e.g. calling into_py twice or into_py followed by into to go from T to Py<S> to PyObject is not necessarily the same as going from T to PyObject.

Just to make sure I understand things correctly: The comment is informational as the other changes are breaking anyway and this is therefore targeting 0.17.0?

Mainly it means that things like let thing = "thing".into_py(py); stop compiling

To avoid making lots of annoying inference errors we could potentially combine this PR with #2316, which would enable us to select on types which have IntoPyObject<Target = PyString> instead of IntoPy<Py<PyString>>.

birkenfeld · 2022-04-18T05:55:21Z

Very nice!

adamreichold

Looks good to me.

One remaining concern that I see is that we have made several methods "trivially generic" which were not generic before. Should we aim to reduce code bloat and factor out the non-generic parts into inner help functions? On the other hand, the functions in question do not appear to be huge, so I am unsure whether this would be worth the extra effort.

src/types/any.rs

mejrs · 2022-04-21T23:00:42Z

Should we aim to reduce code bloat and factor out the non-generic parts into inner help functions?

They are pretty small, I think it's fine.

Would it make sense to unify them?

Done 👍

davidhewitt · 2022-04-23T12:43:18Z

Looks good to me too. The only question I have outstanding is whether we want to merge this now or wait until we can also merge with #2316 so that we don't cause users to go through extra churn of type inference failures in the interim.

(FWIW I have a bad feeling that #2316 gets a little upset with some of the more gnarly #[pyproto] code, so that might be blocked until 0.18 when we can kill the feature off entirely.)

mejrs · 2022-04-23T20:13:00Z

I'm feeling that what we're doing with traits is a bit ad-hoc.

Maybe we need to sit down and think about what we don't like, what we would like, and what a good path to getting there is.

davidhewitt · 2022-04-26T07:06:15Z

I'm feeling that what we're doing with traits is a bit ad-hoc.

Maybe we need to sit down and think about what we don't like, what we would like, and what a good path to getting there is.

I agree very much with this sentiment. I'll write my current backlog of ideas up in a separate issue as a starting point.

As for this PR... looks good to me! The concern I have with merging before we make a decision on #2316 is that this may cause users to have to add : PyObject annotations to keep existing .into_py() uses compiling. If we then merge #2316 these annotations may then become type errors. (I guess #2316 will probably break a bit of existing code too, so maybe that's not actually a reason to hold back this PR?) What's your opinion on this?

mejrs · 2022-04-26T21:58:46Z

src/instance.rs

        unsafe {
-            let args = args.into_py(py).into_ptr();
-            let kwargs = kwargs.into_ptr();
-            let ptr = ffi::PyObject_GetAttr(self.as_ptr(), name.to_object(py).as_ptr());


Sorry this is a bit messy since resolving the merge conflict.

How do you feel about:

pyo3::ffi::Something(name.to_object(py).as_ptr());

versus

let ptr = name.to_object(py).into_ptr(); pyo3::ffi::Something(ptr); ffi::Py_DECREF(ptr);

(or something else?)

Personally I prefer the latter. IMO the former is error prone and gives a bad example.

Why do you feel the former is error prone? I think the second is more error prone because missing the Py_DECREF call is leaking memory just like missing free() in C.

Perhaps a middle ground is to make it obvious we're using RAII through Py?

let name: PyObject = name.to_object(py); pyo3::ffi::Something(name.as_ptr());

Why do you feel the former is error prone? I think the second is more error prone because missing the Py_DECREF call is leaking memory just like missing free() in C.

I'm mostly worried about someone reading it and and writing it slightly differently:

let ptr = name.to_object(py).as_ptr(); pyo3::ffi::Something(ptr);

which produces a dangling pointer, as the python object is dropped at the end of the statement (I should add documentation about this).

Perhaps a middle ground is to make it obvious we're using RAII through Py?

let name: PyObject = name.to_object(py); pyo3::ffi::Something(name.as_ptr());

Sure 👍

'm mostly worried about someone reading it and and writing it slightly differently:

This is an excellent point which I'd totally missed. IMO the middle ground above is probably the only sensible way to arrange this then. If you like, I'll submit a PR to fix the whole codebase to use this pattern once this PR merged? That way you can just worry about the modified functions here without another merge conflict.

mejrs · 2022-04-26T22:01:37Z

What's your opinion on this?

I think that's fine, it's breaking either way.

davidhewitt

Ok, let's head towards merging this in 0.17 and maybe thinking about IntoPyObject for 0.18. I'll try to write up my thoughts on traits tomorrow.

Got to confess I'm really not a fan of calling Py_DECREF or Py_XDECREF whenever we can just let Py do it in Drop implementation.

davidhewitt · 2022-04-27T18:59:04Z

src/instance.rs

        unsafe {
-            let args = args.into_py(py).into_ptr();
-            let kwargs = kwargs.into_ptr();
-            let ptr = ffi::PyObject_GetAttr(self.as_ptr(), name.to_object(py).as_ptr());


Why do you feel the former is error prone? I think the second is more error prone because missing the Py_DECREF call is leaking memory just like missing free() in C.

Perhaps a middle ground is to make it obvious we're using RAII through Py?

let name: PyObject = name.to_object(py); pyo3::ffi::Something(name.as_ptr());

src/instance.rs

src/types/any.rs

davidhewitt

👍 Looks great, thanks very much for all the tidy ups!

A final thought: as we're committing to merging this and producing extra inference errors in PyO3 0.17, can we please add a section to migration.md to explain this to users?

mejrs · 2022-04-28T21:48:13Z

A final thought: as we're committing to merging this and producing extra inference errors in PyO3 0.17, can we please add a section to migration.md to explain this to users?

Sure.

I'm also concerned about some missing codecov here, so I want to add a couple of tests as well.

mejrs · 2022-04-30T21:39:02Z

Should be done now :)

davidhewitt · 2022-05-02T17:57:04Z

Thanks for merging, looks great, sorry I had a busy last few days and didn't get around to looking at this sooner. 👍

mejrs commented Apr 17, 2022

View reviewed changes

mejrs force-pushed the intern-more branch 8 times, most recently from eb814f4 to 5154778 Compare April 20, 2022 23:15

mejrs added 2 commits April 21, 2022 02:51

Allow more methods to take interned arguments

8e2e3af

Changelog

7f96584

mejrs force-pushed the intern-more branch from 5154778 to 7f96584 Compare April 21, 2022 01:00

adamreichold approved these changes Apr 21, 2022

View reviewed changes

src/types/any.rs Outdated Show resolved Hide resolved

src/types/any.rs Show resolved Hide resolved

Unify name bounds

fe49a81

mejrs and others added 2 commits April 25, 2022 00:00

Merge branch 'main' into intern-more

ceaf835

Resolve merge conflict

c7319e1

mejrs requested a review from davidhewitt April 25, 2022 21:36

Merge branch 'main' into intern-more

90db8a2

mejrs commented Apr 26, 2022

View reviewed changes

davidhewitt reviewed Apr 27, 2022

View reviewed changes

reduce use of py_decref

b75c3e9

davidhewitt reviewed Apr 28, 2022

View reviewed changes

Add some attr tests

f1a926f

Update migration

cf07605

mejrs force-pushed the intern-more branch from 7e9385b to cf07605 Compare April 30, 2022 21:33

mejrs merged commit dce4377 into PyO3:main May 2, 2022

mejrs mentioned this pull request May 2, 2022

Expand on AsPyPointer docs and un-ignore doc examples #2350

Merged

davidhewitt mentioned this pull request Jun 25, 2023

add PyDict::get_item_with_error for PyPy #3270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow more methods to take interned arguments #2312

Allow more methods to take interned arguments #2312

mejrs commented Apr 17, 2022 •

edited

mejrs Apr 17, 2022

adamreichold Apr 18, 2022

birkenfeld Apr 18, 2022

adamreichold Apr 18, 2022

birkenfeld Apr 18, 2022

adamreichold Apr 18, 2022

mejrs Apr 18, 2022

davidhewitt Apr 19, 2022

birkenfeld commented Apr 18, 2022

adamreichold left a comment •

edited

mejrs commented Apr 21, 2022

davidhewitt commented Apr 23, 2022

mejrs commented Apr 23, 2022

davidhewitt commented Apr 26, 2022

mejrs Apr 26, 2022 •

edited

davidhewitt Apr 27, 2022

mejrs Apr 27, 2022

davidhewitt Apr 27, 2022

mejrs commented Apr 26, 2022

davidhewitt left a comment

davidhewitt Apr 27, 2022

davidhewitt left a comment

mejrs commented Apr 28, 2022

mejrs commented Apr 30, 2022

davidhewitt commented May 2, 2022

Allow more methods to take interned arguments #2312

Allow more methods to take interned arguments #2312

Conversation

mejrs commented Apr 17, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

birkenfeld commented Apr 18, 2022

adamreichold left a comment • edited

Choose a reason for hiding this comment

mejrs commented Apr 21, 2022

davidhewitt commented Apr 23, 2022

mejrs commented Apr 23, 2022

davidhewitt commented Apr 26, 2022

mejrs Apr 26, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mejrs commented Apr 26, 2022

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidhewitt left a comment

Choose a reason for hiding this comment

mejrs commented Apr 28, 2022

mejrs commented Apr 30, 2022

davidhewitt commented May 2, 2022

mejrs commented Apr 17, 2022 •

edited

adamreichold left a comment •

edited

mejrs Apr 26, 2022 •

edited