Add from_py_with attribute #1411

daniil-konovalenko · 2021-02-01T14:47:29Z

This PR addresses #1239.

It adds a #[pyo3(from_py_with = "function")] attribute that allows to customize conversion between Python-native and Rust-native types.
At the moment only argument attributes are supported. I plan on adding support for derive in the near future.

I've made a couple of design decisions along the way, that can be changed if needed:

I chose from_py_with instead of from_python_with from the original proposal because it was shorter to type. Also it goes well with the trait name FromPyObject.
The path to the conversion function is quoted instead of being a raw identifier, mainly because serde does it. If there are any reasons it shouldn't be quoted, please let me know.

There's still a bunch of stuff left to do:

support from_py_with in derive(FromPyObject)
refactor parse_arg_attributes into smaller functions
add tests with usage in #[pymethods]
support from_py_with for enums/tuple structs Support from_py_with on struct tuples and enums #1441
add tests for errors
add a CHANGELOG entry
add docs (tracking issue Guide: #[pyo3(name = "")] and #[pyo3(from_py_with = "")] #1432)

nw0 · 2021-02-02T09:55:53Z

Just bikeshedding: would you consider 'from_py'? It's even more succinct and I think "with" is pretty clear once you add the equals sign.

…

On 1 Feb 2021, at 10:47 PM, Daniil Konovalenko ***@***.***> wrote: This PR addresses #1239. It adds a #[pyo3(from_py_with = "function")] attribute that allows to customize conversion between Python-native and Rust-native types. At the moment only argument attributes are supported. I plan on adding support for derive in the near future. I've made a couple of design decisions along the way, that can be changed if needed: I chose from_py_with instead of from_python_with from the original proposal because it was shorter to type. Also it goes well with the trait name FromPyObject. The path to the conversion function is quoted instead of being a raw identifier, mainly because serde does this. If there are any reasons it shouldn't be quoted, please let me know. TODO: support from_python_with in derive(FromPyObject) refactor parse_arg_attributes into smaller functions add tests with usage in #[pymethods] add tests for errors add a CHANGELOG entry add docs You can view, comment on, or merge this pull request online at: #1411 Commit Summary implement draft of from_py_with for function arguments add a test File Changes M pyo3-macros-backend/src/method.rs (43) M pyo3-macros-backend/src/module.rs (11) M pyo3-macros-backend/src/pyfunction.rs (46) M pyo3-macros-backend/src/pymethod.rs (10) M pyo3-macros-backend/src/pyproto.rs (6) M tests/test_pyfunction.rs (34) Patch Links: https://github.com/PyO3/pyo3/pull/1411.patch https://github.com/PyO3/pyo3/pull/1411.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

davidhewitt · 2021-02-02T21:43:24Z

Thanks very much for this PR. I'm very excited to see it!

I'm very busy at the moment though hope to get a moment to read it by the weekend and share my full thoughts and feedback then.

kngwyu

Thanks!
I think the API is fine, but there's some room for improvements in the implementation (can we reduce mut)?

pyo3-macros-backend/src/pyfunction.rs

daniil-konovalenko · 2021-02-08T10:48:40Z

Just bikeshedding: would you consider 'from_py'?

I think it's worth considering, but I personally prefer from_py_with more because it feels more familiar to me.

In any case, I feel that the final naming decision should be made by the maintainers, as they have much better understanding of pyo3's API as a whole than I do

davidhewitt

Thanks again for this, and sorry for the delay from me getting round to reading it!

The implementation looks good and has some decent test coverage; I added a few comments on places in the code where I have some questions and ideas to improve it a bit.

Regarding name - I think you're correct that from_py_with is better than from_python_with.

I also think that from_py_with is better than from_py, as we're keeping in line with the precedent made by serde.

Would be nice to hear if @kngwyu has any strong preference on name also.

tests/test_class_basics.rs

pyo3-macros-backend/src/method.rs

pyo3-macros-backend/src/from_pyobject.rs

pyo3-macros-backend/src/method.rs

pyo3-macros-backend/src/module.rs

pyo3-macros-backend/src/pyfunction.rs

davidhewitt

Thanks for the continued work on this, this is looking increasingly solid.

I took another read just now and had a few more ideas...

pyo3-macros-backend/src/attrs.rs

pyo3-macros-backend/src/pyfunction.rs

pyo3-macros-backend/src/pymethod.rs

pyo3-macros-backend/src/pyproto.rs

tests/test_pyfunction.rs

daniil-konovalenko · 2021-02-11T17:27:25Z

CI just got updated to Rust 1.50 and it broke formatting, plus there's a couple of new Clippy rules...

davidhewitt · 2021-02-11T18:11:46Z

👍 I'll fix in a bit and you can rebase this PR on it after

daniil-konovalenko · 2021-02-11T18:28:45Z

Awesome, thank you!

davidhewitt · 2021-02-11T22:32:51Z

See #1422 - just waiting for a second pair of eyes on it...

davidhewitt · 2021-02-14T08:27:42Z

#1422 is merged, so if you rebase things should work now!

daniil-konovalenko · 2021-02-14T14:52:17Z

Rebased, I hope it wouldn't mess up the comments too much

davidhewitt

Thanks, the implementation looks great to me!

I'm happy with the design here, I think this is good to merge. @kngwyu are you also happy with this?

I see that the TODO list still includes CHANGELOG entry and docs.

I think docs should go in the guide. TBH the #[name = ...] attribute is also undocumented; I was going to move it to #[pyo3(name = ...)] and add a guide section to document attributes better at the same time. If you like you can just open an issue to document #[pyo3(from_py_with)] and I'll do it at the same time.

CHANGELOG entry - I suggest something like this:

- Add #[pyo3(from_py_with = "...")]` attribute for function arguments and struct fields to override the default from-Python conversion. [#1411](https://github.com/PyO3/pyo3/pull/1411)

tests/test_class_basics.rs

davidhewitt · 2021-02-14T17:56:19Z

tests/test_frompyobject.rs

+    some_object: String,
+}
+
+#[test]


Final thought; could we please add tests for tuple structs and enum variants? Just to make sure that all cases are covered and remain covered.

Sure, thank you for the reminder!

I've finally got to this and I have a question. At the moment we provide from_py_with for struct fields only.
Do you have in mind that from_py_with should be supported for tuple structs and enums as well?

Hmm, that's a good point, I didn't really specify those on #1239 when I proposed the idea.

I think for tuple structs with #[derive(FromPyObject)] it's quite natural to want to be able to annotate fields in the same way. Maybe let's skip on deciding what to do with enum variants for now. That'll be easy to add later anyway.

If tuple structs are going to be a lot of extra work, perhaps let's merge this PR for now and add an issue to remember to support them later?

Thanks for clearing this out! I'll try supporting tuple structs, maybe it won't be that hard.
In any case, I think it's worth adding a compiler error (warning?) when from_py_with is used on types that are not yet supported

daniil-konovalenko · 2021-02-15T11:53:10Z

@davidhewitt
Thank you for the thorough review!
I like the changelog line that you proposed, I think it summarizes the changes really well.
I'll try to fix the remaining issues on the following week, probably closer to the weekend.
If you'll be able to update the guide it would be awesome! I've created an issue for this: #1432.

I'm still a little bit bummed about this:

pyo3/pyo3-macros-backend/src/method.rs

Lines 136 to 138 in d43c19b

    
           let inputs_empty = sig.inputs.is_empty(); 
        
           let sig_span = sig.span(); 
        
           let inputs_span = sig.inputs.span();

I have tried a few things but I can't seem to fix it completely.
My approach was to split up FnSpec::parse into different functions, it helped to find out which parts of the parsing code use which references.
Turns out the main problem is this:

an immutable reference to sig must be valid during the whole FnSpec::parse
we need sig.inputs.iter() to parse fn_spec etc.
we need sig.inputs.iter_mut() to parse attributes.
sig.inputs.iter_mut() can't live at the same time as &sig.inputs.iter()

If we just use the mutable iterator everywhere, then we would need the additional variables listed above. Also it introduces unnecessary mutability to parse_method_receiver.

The other approach I tried is this:

Use an immutable iterator to parse fn_type. Count how many elements (N) from sig.inputs we used.
Obtain a mutable iterator and skip the first N elements, then parse the rest of the arguments.
This approach is somewhat awkward. I've got it to compile, but it does not pass all the tests yet.

There's also the third approach:

Parse everything immutably
remove attributes from the arguments.
see Add from_py_with attribute #1411 (comment) for the details.
I'm not a fan of this one because it would be easy to forget to remove attributes. Also it adds an implicit dependency between the parsing code and the removal code.

At this point I'm stuck. I would love to hear any ideas that you have on this.

kngwyu · 2021-02-15T13:04:11Z

Would be nice to hear if @kngwyu has any strong preference on name also.

No, I'm fine with the current name.

kngwyu · 2021-02-15T13:26:27Z

I am now still playing in the code but will leave some suggestions in a day.

daniil-konovalenko · 2021-02-15T15:12:56Z

I've pushed the second approach implementation. It's awkward around skipping arguments, but otherwise I like it more than what I had before.

kngwyu

Thanks, your last commit nicely simplifies the code. 👍🏼
I left some comments to make the intention clearer.

pyo3-macros-backend/src/method.rs

daniil-konovalenko · 2021-02-16T19:06:30Z

@kngwyu Thanks for the suggestions! It looks a lot better now

kngwyu

Thanks, I think now it's ready 👍🏼

davidhewitt

This all looks great to me, thanks for your hard work on this. The implementation of FnSpec::parse is a lot better now, really nice job.

Few final nits and then let's merge this. It could also be worth squashing this down to one or two commits from 22.

CHANGELOG.md

pyo3-macros-backend/src/method.rs

* allow from_py_with inside #[derive(FromPyObject)] * split up FnSpec::parse

kngwyu · 2021-02-20T05:04:04Z

add tests for enums/tuple structs

Is this the only blocker?

daniil-konovalenko · 2021-02-20T10:16:09Z

Is this the only blocker?

I forgot to update the TODO: I still need to implement from_py_with for tuple structs and enums, but it can be deferred to a future PR.
See our discussion with David:
#1411 (comment)

kngwyu · 2021-02-20T15:15:13Z

I see, thank you!

kngwyu reviewed Feb 4, 2021

View reviewed changes

pyo3-macros-backend/src/pyfunction.rs Outdated Show resolved Hide resolved

pyo3-macros-backend/src/pyfunction.rs Outdated Show resolved Hide resolved

kngwyu reviewed Feb 6, 2021

View reviewed changes

pyo3-macros-backend/src/pyfunction.rs Outdated Show resolved Hide resolved

davidhewitt reviewed Feb 8, 2021

View reviewed changes

daniil-konovalenko force-pushed the from-py-with branch from e2ed1fc to b06d211 Compare February 9, 2021 15:58

davidhewitt reviewed Feb 10, 2021

View reviewed changes

daniil-konovalenko force-pushed the from-py-with branch from d79f1f3 to d43c19b Compare February 14, 2021 14:51

davidhewitt linked an issue Feb 14, 2021 that may be closed by this pull request

Add #[pyo3(from_python_with = ...)] attribute #1239

Closed

davidhewitt approved these changes Feb 14, 2021

View reviewed changes

daniil-konovalenko force-pushed the from-py-with branch from de028bf to 13efe6e Compare February 15, 2021 15:39

kngwyu reviewed Feb 16, 2021

View reviewed changes

pyo3-macros-backend/src/method.rs Outdated Show resolved Hide resolved

pyo3-macros-backend/src/method.rs Outdated Show resolved Hide resolved

pyo3-macros-backend/src/method.rs Outdated Show resolved Hide resolved

pyo3-macros-backend/src/method.rs Outdated Show resolved Hide resolved

kngwyu reviewed Feb 16, 2021

View reviewed changes

pyo3-macros-backend/src/method.rs Outdated Show resolved Hide resolved

kngwyu approved these changes Feb 18, 2021

View reviewed changes

davidhewitt approved these changes Feb 18, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

pyo3-macros-backend/src/method.rs Outdated Show resolved Hide resolved

add #[pyo3(from_py_with="...")] attribute

cae3bbb

* allow from_py_with inside #[derive(FromPyObject)] * split up FnSpec::parse

daniil-konovalenko force-pushed the from-py-with branch from e1f2c1e to cae3bbb Compare February 18, 2021 14:08

kngwyu merged commit 554cffd into PyO3:master Feb 20, 2021

daniil-konovalenko mentioned this pull request Feb 20, 2021

Support from_py_with on struct tuples and enums #1441

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add from_py_with attribute #1411

Add from_py_with attribute #1411

daniil-konovalenko commented Feb 1, 2021 •

edited

nw0 commented Feb 2, 2021 via email

davidhewitt commented Feb 2, 2021

kngwyu left a comment

daniil-konovalenko commented Feb 8, 2021 •

edited

davidhewitt left a comment

davidhewitt left a comment

daniil-konovalenko commented Feb 11, 2021 •

edited

davidhewitt commented Feb 11, 2021

daniil-konovalenko commented Feb 11, 2021

davidhewitt commented Feb 11, 2021

davidhewitt commented Feb 14, 2021

daniil-konovalenko commented Feb 14, 2021

davidhewitt left a comment

davidhewitt Feb 14, 2021

daniil-konovalenko Feb 14, 2021

daniil-konovalenko Feb 18, 2021

davidhewitt Feb 18, 2021

daniil-konovalenko Feb 19, 2021 •

edited

daniil-konovalenko commented Feb 15, 2021 •

edited

kngwyu commented Feb 15, 2021

kngwyu commented Feb 15, 2021

daniil-konovalenko commented Feb 15, 2021 •

edited

kngwyu left a comment

daniil-konovalenko commented Feb 16, 2021

kngwyu left a comment

davidhewitt left a comment

kngwyu commented Feb 20, 2021

daniil-konovalenko commented Feb 20, 2021

kngwyu commented Feb 20, 2021

Add from_py_with attribute #1411

Add from_py_with attribute #1411

Conversation

daniil-konovalenko commented Feb 1, 2021 • edited

nw0 commented Feb 2, 2021 via email

davidhewitt commented Feb 2, 2021

kngwyu left a comment

Choose a reason for hiding this comment

daniil-konovalenko commented Feb 8, 2021 • edited

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt left a comment

Choose a reason for hiding this comment

daniil-konovalenko commented Feb 11, 2021 • edited

davidhewitt commented Feb 11, 2021

daniil-konovalenko commented Feb 11, 2021

davidhewitt commented Feb 11, 2021

davidhewitt commented Feb 14, 2021

daniil-konovalenko commented Feb 14, 2021

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt Feb 14, 2021

Choose a reason for hiding this comment

daniil-konovalenko Feb 14, 2021

Choose a reason for hiding this comment

daniil-konovalenko Feb 18, 2021

Choose a reason for hiding this comment

davidhewitt Feb 18, 2021

Choose a reason for hiding this comment

daniil-konovalenko Feb 19, 2021 • edited

Choose a reason for hiding this comment

daniil-konovalenko commented Feb 15, 2021 • edited

kngwyu commented Feb 15, 2021

kngwyu commented Feb 15, 2021

daniil-konovalenko commented Feb 15, 2021 • edited

kngwyu left a comment

Choose a reason for hiding this comment

daniil-konovalenko commented Feb 16, 2021

kngwyu left a comment

Choose a reason for hiding this comment

davidhewitt left a comment

Choose a reason for hiding this comment

kngwyu commented Feb 20, 2021

daniil-konovalenko commented Feb 20, 2021

kngwyu commented Feb 20, 2021

daniil-konovalenko commented Feb 1, 2021 •

edited

daniil-konovalenko commented Feb 8, 2021 •

edited

daniil-konovalenko commented Feb 11, 2021 •

edited

daniil-konovalenko Feb 19, 2021 •

edited

daniil-konovalenko commented Feb 15, 2021 •

edited

daniil-konovalenko commented Feb 15, 2021 •

edited