Add `anyhow-integration` feature which implements From<anyhow::Error> for PyErr #1822

chris-laplante · 2021-08-23T02:05:42Z

This makes it possible to use anyhow with pyo3. Includes documentation for feature in guide.

davidhewitt

Hmmm.

I agree anyhow is a very useful library, and I understand why this is desirable.

I don't much like how this makes anyhow::Error always map to RuntimeError, which misses out on the richness of Python's range of exception types.

I'd be interested in hearing other maintainer's opinions on whether they think this is a useful feature to add to the library. I could go either way, however my stance at the moment is that I'd like to be persuaded this is necessary.

Cargo.toml

davidhewitt · 2021-08-24T06:15:25Z

guide/src/features.md

+### `anyhow-integration`
+
+This feature makes it possible to return [`anyhow::Result<T>`](https://docs.rs/anyhow/1.0.43/anyhow/type.Result.html) from functions and methods exposed to Python. It does so by adding `impl From<anyhow::Error> for PyErr`. Currently, the conversion simply stringifies the `anyhow::Error` and shoves it into a `PyRuntimeError`. As a consequence, there is no way to convert a `PyErr` back to the original `anyhow::Error`. It is mostly intended as a developer convenience.


We should probably make the section with num_bigint etc into an Optional Dependencies section and add anyhow there.

Are you saying I should rename the 'Advanced features' section in features.md to 'Optional features'?

I was thinking more like #1831 - so I went ahead and made that PR 😄

If you put this anyhow feature into that new "Optional Dependencies" section (once merged), that seems most appropriate to me.

messense · 2021-08-24T06:23:42Z

IMO it's fine to use RuntimeError since using anyhow usually means the developer don't care about the exact error type.

birkenfeld · 2021-08-24T06:45:04Z

I tend to agree with @messense, this is a nice convenience feature for application code where the error type is mostly irrelevant.

For library code, many Python libraries make their own exception class or hierarchy and convert upstream exceptions to those. This is obviously not possible to offer from PyO3 and needs some converter functions in the respective library.

I'm not sure about the formatting, though: is {:?} the correct one? (anyhow::Error also can have context information; this should not be discarded if possible.)

birkenfeld · 2021-08-24T06:46:04Z

Of course, it's also possible to provide, under the same feature, a method on PyErr for converting anyhow::Error into arbitrary exception classes if wanted.

mejrs · 2021-08-24T09:25:08Z

Thanks!

Would you be willing to add an entry and doc example here?

mejrs · 2021-08-24T16:04:15Z

my stance at the moment is that I'd like to be persuaded this is necessary.

I think this is also how I feel.

I also think it is inconsistent that eyre has this conversion there and that we'd have anyhows here. But that isn't necessarily a problem to me.

chris-laplante · 2021-08-24T16:25:59Z

I'm not sure about the formatting, though: is {:?} the correct one? (anyhow::Error also can have context information; this should not be discarded if possible.)

From my testing, the context is correctly preserved :)

chris-laplante · 2021-08-24T16:27:01Z

Thanks!

Would you be willing to add an entry and doc example here?

I assume this was directed at me :) in which case will do!

chris-laplante · 2021-08-24T16:29:02Z

my stance at the moment is that I'd like to be persuaded this is necessary.

I think this is also how I feel.

I also think it is inconsistent that eyre has this conversion there and that we'd have anyhows here. But that isn't necessarily a problem to me.

I did think about submitting this change to anyhow instead. But, there are only a handful of popular error handling crates. Contrast that to the bazillion possible crates you may want to use in an application. I feel like it would be asking too much to add support for all those crates into the error handler ones.

davidhewitt

👍 it sounds like we're generally in favour of adding this, so let's proceed with doing so!

davidhewitt · 2021-08-24T21:08:32Z

src/err/mod.rs

+#[cfg(feature = "anyhow")]
+impl From<anyhow::Error> for PyErr {
+    fn from(err: anyhow::Error) -> Self {
+        crate::exceptions::PyRuntimeError::new_err(format!("{:?}", err))
+    }
+}


From a code organisation perspective I'd quite like this to go into a new file src/conversions/anyhow.rs, and it would be nice to also add tests for it!

…Err impl This makes it possible to use anyhow::Result<T> as the return type for methods and functions exposed to Python. The current implementation just stringifies the anyhow::Error before shoving it into a PyRuntimeError. Conversion back to the anyhow::Error is not possible, but it is better than nothing. Signed-off-by: Chris Laplante <chris.laplante@agilent.com>

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>

chris-laplante · 2021-08-27T14:53:49Z

I'm getting a really cryptic macro error when I try to run the WIP test locally. Could someone please take a look? Here is the output. I'm using the nightly toolchain.

   Compiling pyo3 v0.14.3 (/home/laplante/repos/pyo3)
error[E0277]: the trait bound `pyo3::derive_utils::PyFunctionArguments<'_>: From<python::Python<'_>>` is not satisfied
   --> src/lib.rs:361:36
    |
361 |         &|py| pyo3::paste::expr! { [<__pyo3_get_function_ $function_name>] }(py)
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `From<python::Python<'_>>` is not implemented for `pyo3::derive_utils::PyFunctionArguments<'_>`
    |
   ::: src/conversions/anyhow.rs:30:24
    |
30  |             let func = wrap_py
```function!(produce_ok_result)(py).unwrap();
    |                        ----------------------------------- in this macro invocation
    |
    = help: the following implementations were found:
              <pyo3::derive_utils::PyFunctionArguments<'a> as From<&'a pyo3::prelude::PyModule>>
              <pyo3::derive_utils::PyFunctionArguments<'a> as From<pyo3::Python<'a>>>
    = note: required because of the requirements on the impl of `Into<pyo3::derive_utils::PyFunctionArguments<'_>>` for `python::Python<'_>`
note: required by a bound in `__pyo3_get_function_produce_ok_result`
   --> src/conversions/anyhow.rs:17:5
    |
17  |     #[pyfunction]
    |     ^^^^^^^^^^^^^ required by this bound in `__pyo3_get_function_produce_ok_result`
18  |     fn produce_ok_result() -> anyhow::Result<String> {
    |        ----------------- required by a bound in this
    = note: this error originates in the macro `proc_macro_call` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0599]: the method `to_object` exists for reference `&pyo3::types::PyCFunction`, but its trait bounds were not satisfied
   --> src/lib.rs:478:44
    |
478 |           let d = [$((stringify!($val), $val.to_object($py)),)+].into_py_dict($py);
    |                                              ^^^^^^^^^ method cannot be called on `&pyo3::types::PyCFunction` due to unsatisfied trait bounds
    |
   ::: /home/laplante/repos/pyo3/src/types/function.rs:11:1
    |
11  |   pub struct PyCFunction(PyAny);
    |   ------------------------------ doesn't satisfy `pyo3::types::PyCFunction: conversion::ToPyObject`
    |
   ::: src/conversions/anyhow.rs:32:13
    |
32  | /             py_run!(
33  | |                 py,
34  | |                 func,
35  | |                 r#"
36  | |             func()
37  | |             "#
38  | |             );
    | |______________- in this macro invocation
    |
    = note: the following trait bounds were not satisfied:
            `pyo3::types::PyCFunction: conversion::ToPyObject`
            which is required by `&pyo3::types::PyCFunction: conversion::ToPyObject`
    = help: items from traits can only be used if the trait is in scope
    = note: the following trait is implemented but not in scope; perhaps add a `use` for it:
            `use pyo3::ToPyObject;`
    = note: this error originates in the macro `$crate::py_run_impl` (in Nightly builds, run with -Z macro-backtrace for more info)

davidhewitt · 2021-08-27T19:16:27Z

Looks like wrap_pyfunction! probably assumes the prelude is imported. IMO that's a bug.

EDIT: opened #1837 to fix that

davidhewitt · 2021-08-28T09:34:15Z

src/conversions/anyhow.rs

+    #[pyfunction]
+    fn produce_ok_result() -> anyhow::Result<String> {
+        Ok(String::from("OK buddy"))
+    }
+
+    #[pyfunction]
+    fn produce_err_result() -> anyhow::Result<String> {
+        anyhow::bail!("error time")
+    }


It's actually not ideal to use #[pyfunction] within unit tests of pyo3's lib, because it relies on pyo3 being available as in the global namespace. We happen to have a self-dev-dependency on pyo3 but it causes unexpected breakages - see #1817 for more detail.

The point here is that it's probably better to just test that let x: PyErr = produce_err_result().unwrap_err().into() works ok, and could check the string contents of the error are as expected.

(If you really want to check the #[pyfunction] case, that's good too, but it'll need to go in a separate file in tests/.)

mejrs · 2021-10-08T18:52:45Z

Hey @chris-laplante - this looks done or almost done 😃 Are you able to finish it up?

If not - I'd like to finish it up. I've been working on a similar eyre feature and I want to push out it and this at a similar time.

chris-laplante · 2021-10-15T14:58:56Z

Hey @chris-laplante - this looks done or almost done 😃 Are you able to finish it up?

If not - I'd like to finish it up. I've been working on a similar eyre feature and I want to push out it and this at a similar time.

Hey @mejrs sorry for the lack of reply - I have been busy with other things unfortunately. If you would like to take over I'd appreciate it :)

mejrs · 2021-10-16T00:58:48Z

I think that is everything.

davidhewitt

LGTM, thanks for finishing this off! Just a couple of final comments around documentation.

davidhewitt · 2021-10-16T08:53:36Z

CHANGELOG.md

@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add `PyAny::py` as a convenience for `PyNativeType::py`. [#1751](https://github.com/PyO3/pyo3/pull/1751)
 - Add implementation of `std::ops::Index<usize>` for `PyList`, `PyTuple` and `PySequence`. [#1825](https://github.com/PyO3/pyo3/pull/1825)
 - Add range indexing implementations of `std::ops::Index` for `PyList`, `PyTuple` and `PySequence`. [#1829](https://github.com/PyO3/pyo3/pull/1829)
+- Add `anyhow` feature which provides `impl From<anyhow::Error> for PyErr`. [#1822](https://github.com/PyO3/pyo3/pull/1822)


Let's put this in the Packaging section too and make sure this and the eyre entry are written the same way?

davidhewitt · 2021-10-16T08:57:37Z

guide/src/features.md

+### `anyhow`
+
+This feature makes it possible to return [`anyhow::Result<T>`](https://docs.rs/anyhow/1.0.43/anyhow/type.Result.html) from functions and methods exposed to Python. It does so by adding `impl From<anyhow::Error> for PyErr`. Currently, the conversion simply stringifies the `anyhow::Error` and shoves it into a `PyRuntimeError`. As a consequence, there is no way to convert a `PyErr` back to the original `anyhow::Error`. It is mostly intended as a developer convenience.


Similarly I think it'd be nice for this paragraph and the eyre one to read pretty much identically, given that the features are almost equivalent. I don't mind whether it's this one or the eyre one that changes; depends how much detail you think is appropriate.

davidhewitt · 2021-10-16T09:01:02Z

tests/test_anyhow.rs

@@ -0,0 +1,154 @@
+#![cfg(feature = "anyhow")]
+
+//! A conversion from [anyhow]’s [`Error`] type to [`PyErr`].


Was it intentional to have this documentation in the test?

I was slightly tipsy and I guess I wrote it twice 😊

Haha whoopsie! 😂

chris-laplante · 2021-10-18T13:57:46Z

Thank you @mejrs and @davidhewitt so much for finishing this up :). I appreciate it!

davidhewitt reviewed Aug 24, 2021

View reviewed changes

davidhewitt mentioned this pull request Aug 24, 2021

PyO3 integration - offer to rehome eyre-rs/eyre#57

Closed

chris-laplante added 4 commits August 26, 2021 11:11

Document anyhow feature in the guide

58c3e89

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>

update changelog to document anyhow feature

d5313c5

WIP adding tests

defc2dd

chris-laplante force-pushed the anyhow branch from c35b48d to defc2dd Compare August 26, 2021 15:45

davidhewitt reviewed Aug 28, 2021

View reviewed changes

mejrs mentioned this pull request Oct 13, 2021

Implement eyre feature #1893

Merged

mejrs added this to the 0.15 milestone Oct 14, 2021

davidhewitt mentioned this pull request Oct 14, 2021

0.15 Release #1921

Closed

2 tasks

Merge branch 'main' into anyhow

f12db7f

mejrs added 4 commits October 16, 2021 00:46

Finish up anyhow feature

3650516

Fix formatting

5297969

Fix tests

1aa3d3d

Fix tests

a46f11e

mejrs requested a review from davidhewitt October 16, 2021 00:58

davidhewitt approved these changes Oct 16, 2021

View reviewed changes

Apply review suggestions

99caed7

davidhewitt merged commit 3b94f4b into PyO3:main Oct 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `anyhow-integration` feature which implements From<anyhow::Error> for PyErr #1822

Add `anyhow-integration` feature which implements From<anyhow::Error> for PyErr #1822

chris-laplante commented Aug 23, 2021

davidhewitt left a comment

davidhewitt Aug 24, 2021

chris-laplante Aug 25, 2021

davidhewitt Aug 25, 2021

messense commented Aug 24, 2021

birkenfeld commented Aug 24, 2021 •

edited

birkenfeld commented Aug 24, 2021

mejrs commented Aug 24, 2021

mejrs commented Aug 24, 2021

chris-laplante commented Aug 24, 2021

chris-laplante commented Aug 24, 2021

chris-laplante commented Aug 24, 2021

davidhewitt left a comment

davidhewitt Aug 24, 2021

chris-laplante commented Aug 27, 2021

davidhewitt commented Aug 27, 2021 •

edited

davidhewitt Aug 28, 2021

mejrs commented Oct 8, 2021

chris-laplante commented Oct 15, 2021

mejrs commented Oct 16, 2021

davidhewitt left a comment

davidhewitt Oct 16, 2021

davidhewitt Oct 16, 2021

davidhewitt Oct 16, 2021

mejrs Oct 16, 2021

davidhewitt Oct 17, 2021

chris-laplante commented Oct 18, 2021

		### `anyhow-integration`

		This feature makes it possible to return [`anyhow::Result<T>`](https://docs.rs/anyhow/1.0.43/anyhow/type.Result.html) from functions and methods exposed to Python. It does so by adding `impl From<anyhow::Error> for PyErr`. Currently, the conversion simply stringifies the `anyhow::Error` and shoves it into a `PyRuntimeError`. As a consequence, there is no way to convert a `PyErr` back to the original `anyhow::Error`. It is mostly intended as a developer convenience.

		### `anyhow`

		This feature makes it possible to return [`anyhow::Result<T>`](https://docs.rs/anyhow/1.0.43/anyhow/type.Result.html) from functions and methods exposed to Python. It does so by adding `impl From<anyhow::Error> for PyErr`. Currently, the conversion simply stringifies the `anyhow::Error` and shoves it into a `PyRuntimeError`. As a consequence, there is no way to convert a `PyErr` back to the original `anyhow::Error`. It is mostly intended as a developer convenience.

		@@ -0,0 +1,154 @@
		#![cfg(feature = "anyhow")]

		//! A conversion from [anyhow]’s [`Error`] type to [`PyErr`].

Add anyhow-integration feature which implements From<anyhow::Error> for PyErr #1822

Add anyhow-integration feature which implements From<anyhow::Error> for PyErr #1822

Conversation

chris-laplante commented Aug 23, 2021

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

messense commented Aug 24, 2021

birkenfeld commented Aug 24, 2021 • edited

birkenfeld commented Aug 24, 2021

mejrs commented Aug 24, 2021

mejrs commented Aug 24, 2021

chris-laplante commented Aug 24, 2021

chris-laplante commented Aug 24, 2021

chris-laplante commented Aug 24, 2021

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chris-laplante commented Aug 27, 2021

davidhewitt commented Aug 27, 2021 • edited

Choose a reason for hiding this comment

mejrs commented Oct 8, 2021

chris-laplante commented Oct 15, 2021

mejrs commented Oct 16, 2021

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chris-laplante commented Oct 18, 2021

Add `anyhow-integration` feature which implements From<anyhow::Error> for PyErr #1822

Add `anyhow-integration` feature which implements From<anyhow::Error> for PyErr #1822

birkenfeld commented Aug 24, 2021 •

edited

davidhewitt commented Aug 27, 2021 •

edited