Always call PyNumber_Index when casting from Python to a C++ integral type, also pre-3.8 #2801

YannickJadoul · 2021-01-16T23:07:04Z

Description

See the remaining decision to be made in #2698: #2698 (comment)

In one sentence, this makes the casting from Python objects to C++ integer types consistent across Python versions, "backporting" Python 3.8 behavior on handling __index__ to pre-3.8 Pythons.

As @bstaletic pointed out to me, we have done this before, most often backporting Python 3 behavior to Python 2 (unicode, for example), but also in #2616, "backporting" 3.8 behavior.

Given the original issue fixed by #2698, and especially the discussion following that ("everything with __index__ is an integer type"), I think this is the logical to minimize surprises when developing pybind11 libraries. See also the changed test demonstrating how things are more consistent across versions.

I'm cheekily adding this to 2.6.2, because in my eyes, it's still part of #2698, and I'd like to see a decision made ("no, because ..." is also fine, btw; then we close this PR! But it doesn't make sense to me to delay this simple decision on policy.). This PR mainly makes it easier to judge the changes (after soon merging #2698, at least).

Suggested changelog entry:

When casting to a C++ integer, `__index__` is always called and not considered as conversion, consistent with Python 3.8+.

YannickJadoul · 2021-01-17T00:42:39Z

Aaaaand ICC segfaults. Why!?
It's been slightly over a day since we merged that PR...

~~Valgrind seems perfectly happy, btw.~~

Resolved in #2801; thanks, @henryiii!

YannickJadoul · 2021-01-17T01:17:33Z

include/pybind11/cast.h

+                if (!tmp) {
+                    PyErr_Clear();
+                    return false;
+                }
+                do_decref = true;
+                obj = tmp;
+            }
+#endif
+            if (std::is_unsigned<py_type>::value) {
+                py_value = as_unsigned<py_type>(obj.ptr());


This might seems like a reasonably big change, but after this PR, I want to fix #2786, which involves a minor refactoring of casting to C++ integer types (to ensure future consistency with py::int_::operator int()), so keep that in mind when reviewing, please ;-)

If we think that consistency between Python < 3.8 and >= 3.8 versions is a nice thing to have, then I personally don't really think this is a too high implementation price to pay.

(Also, #2786's fix shouldn't be complex either, so if you're able to wait 1 or 2 more days, it can also still be a fix to go into 2.6.2. But we need to draw a line somewhere, ofc.)

… type, also pre-3.8

henryiii · 2021-01-17T03:50:20Z

There was an issue in setuptools 51.3.0 fixed in 51.3.1, I've restarted the build. pypa/setuptools#2535

YannickJadoul · 2021-01-17T12:19:21Z

There was an issue in setuptools 51.3.0 fixed in 51.3.1, I've restarted the build. pypa/setuptools#2535

Thanks :-) I'd noticed setuptools had release suspiciously recently, but I didn't manage to figure out what was going in anymore, at 2 am.

henryiii · 2021-01-19T23:24:20Z

I'm mildly in favor. We don't want to rush a release out; after I get the PRs that were blocked by the GitHub API issue & CMake, I'd like @rwgk to run a global test before we pull the trigger on a release. So there's a bit more time.

YannickJadoul · 2021-01-19T23:28:23Z

In principle, this is ready. Surely things can still be cleaned up, but let's first make a decision on what we want?

#2802 (fixing #2786) is a bit more of a mess, though. I had hoped I could still get it in, together with this, but ... well, I don't know, right now.

rwgk · 2021-01-19T23:39:11Z

OOPS, this is #2801, I mistakingly thought it's #2698. I just deleted my previous comment. Sorry!

rwgk · 2021-01-19T23:43:27Z

I'd like @rwgk to run a global test before we pull the trigger on a release

Will do. (I'm pretty neutral on this PR yes/no for v2.6.2.)

include/pybind11/cast.h

rwgk · 2021-01-20T04:31:22Z

include/pybind11/cast.h

+                    PyErr_Clear();
+                    return false;
+                }
+                do_decref = true;


index_owner = reinterpret_steal<object>(tmp);

That way you don't need the second #if PY_VERSION_HEX < 0x03080000 below and this code become exception safe.

I'd also use idx (or similar) instead of tmp, to be more descriptive.

I thought of/tried that, but didn't want to incur an overhead refcounting on Python >= 3.8, and this is also what CPython does.

But wait, maybe you mean something else, that doesn't need this! I'll give this a shot :-)

Yesss, that does work out beautifully! Thanks!

Still seeing if this could easily be refactored out.

OK, it's quite hard to refactor into a separate private function without incurring an additional inc_ref/dec_ref, it seems. It's already cleaner than before, though, so is it fine to leave like this for now?

…rnings in >=3.8

rwgk · 2021-01-20T21:54:45Z

Thanks @YannickJadoul, that looks great! I'll run this through our big testing system asap (probably tonight).

YannickJadoul · 2021-01-20T22:10:30Z

Thanks @YannickJadoul, that looks great! I'll run this through our big testing system asap (probably tonight).

Good, thanks! I'm pretty confident this PR does as it says (given our own tests and how they caught some corner cases), but it's good to have an idea how this "backported behavior" interacts in a larger context.

bstaletic

This looks fine to me.

rwgk · 2021-01-22T02:04:11Z

I'm seeing 2 test failures that look related to this PR.
We're still on Python 3.6, so the #if PY_VERSION_HEX < 0x03080000 branch kicks in.
Debugging.

rwgk · 2021-01-22T04:12:24Z

In both failing tests a NumPy array with one float was passed for an int arg. I mailed fixes, boiling down to int(arr[0]), with comment:

A NumPy array was passed instead of an integer. The implicit conversion is disabled by pybind11 PR #2801, which backports a behavior change introduced by Python 3.8, for consistency.

@YannickJadoul, is that description accurate?

henryiii · 2021-01-22T17:49:10Z

I think the current (in this PR) behavior is the correct one; a NumPy float should not be automatically converted to an int. If the they want to support floats, there should be a float function/method/constructor that does the conversion. NumPy no longer allows floats in indexing, as well.

YannickJadoul · 2021-01-22T17:53:35Z

Huh, this is weird though. I thought this PR would only make things more permissive! Let me try out a few things.

rwgk · 2021-01-22T17:54:48Z

I agree with @henryiii and I'm OK if you want to merge for 2.6.2, although strictly speaking it's bending the rules to introduce this behavior change in a minor release, now that we know there are things that will break.

henryiii · 2021-01-22T17:55:31Z

Are you sure that current master without this PR doesn't also trigger that? Assuming it's the other PR that caused this?

henryiii · 2021-01-22T17:56:20Z

It's breaking something that shouldn't have worked, though, so it's a bit of a grey area.

rwgk · 2021-01-22T17:57:38Z

Yes, sure. Before attempting a fix, I commented out the `return false;` after `PyNumberIndex`, and the `src_or_index = index;` assignment, which made the test work.

…

On Fri, Jan 22, 2021 at 9:55 AM Henry Schreiner ***@***.***> wrote: Are you sure that current master without this PR doesn't also trigger that? Assuming it's the other PR that caused this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2801 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFUZACDRCKRNLOWGP65LNLS3G32FANCNFSM4WFT3GAA> .

YannickJadoul · 2021-01-22T17:57:45Z

$ python3.6
Python 3.6.9 (default, Oct  8 2020, 12:12:24) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> x = np.array(42)
>>> int(x)
42
>>> list(range(100))[x]
42
>>> y = np.array(3.14159)
>>> int(y)
3
>>> list(range(100))[y]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: only integer scalar arrays can be converted to a scalar index

Yeah, so int(some_float_array) works (also in 3.8), so I would think that we still expect to convert this on noconvert(false), right?

rwgk · 2021-01-22T20:12:24Z

Thanks @YannickJadoul, I'll run this PR through our big old testing mill again, later tonight.

FYI: In the meantime one team already merged my fix for them. (I'm still waiting for feedback on the second fix.)

YannickJadoul · 2021-01-22T20:18:19Z

Thanks @YannickJadoul, I'll run this PR through our big old testing mill again, later tonight.

FYI: In the meantime one team already merged my fix for them. (I'm still waiting for feedback on the second fix.)

Oh, OK. I don't think they're even to blame, actually, since things did just work on 3.8. Thanks for finding this!

I still need to figure out how to get 2.7 to at least pass tests. (Maybe not worth putting too much effort into 2.7.)

rwgk · 2021-01-22T20:24:21Z

The failures were actually useful. I don't think the code was working as intended by the original authors, only scraping by.
But I agree with your approach to revising this PR, to fully match Python 3.8 behavior.
Python 2.7 is on it's last leg as far as I see, not worth a significant effort.

YannickJadoul · 2021-01-22T20:28:58Z

Ah, good to know it was at least worth the effort, then!
For the record, I believe that some of these kinds of conversions start producing deprecation warnings from 3.8+ onwards (I had to silence them). Python just has a really weird approach to deprecation warnings and tends to hide them (though it's already better in more recent versions, I believe).

YannickJadoul · 2021-01-22T21:13:50Z

OK, got it. Python 2's long vs. int. Sigh

YannickJadoul · 2021-01-23T01:29:08Z

I'll get back to this/fix this tomorrow, btw.
I'm still confused, since it seems that both PyLong_AsLong as well as PyNumber_Long call __int__? The docs are slightly confusing, so I'll check out CPython's source, but I need some sleep and a fresh perspective on this.

YannickJadoul

OK, this ~~fixes~~ patches things, adding some more duct tape here and there.

Seems like refactoring/restructuring the float/int type caster is overdue, as well, but I propose to not cram that into this PR anymore.

Also, yes, this could use another test round, @rwgk. Who knows what I missed, this time.
Next to that test round, it's also worth checking: is this series of tests the behavior we want?

YannickJadoul · 2021-01-23T17:47:50Z

tests/test_builtin_casters.py

@@ -286,14 +300,20 @@ def cant_convert(v):
    assert noconvert(7) == 7
    cant_convert(3.14159)
    assert convert(DeepThought()) == 42
-    require_implicit(DeepThought())
+    requires_conversion(DeepThought())
+    assert convert(DoubleThought()) == 0  # Fishy; `int(DoubleThought)` == 42


This is not great, but kind of a consequence of us saying "everything with __index__ is already an int, so don't try converting".

YannickJadoul · 2021-01-23T17:53:25Z

include/pybind11/cast.h

+                py_value = as_unsigned<py_type>(src_or_index.ptr());
+            } else { // signed integer:
+                py_value = sizeof(T) <= sizeof(long)
+                    ? (py_type) PyLong_AsLong(src_or_index.ptr())


Oh, another note: this results in warnings, and I think that's correct. Because it's not just getting out the long from the PyLong object, but it's also trying to do the conversion.

Yes, the C API is quite messy here. And it's only made worse by the structure of this caster.

rwgk · 2021-01-24T17:20:10Z

FYI: I tried running this through our global testing last night but something went wrong, the tests only ran with a previous version of this PR. I'll try again.

rwgk · 2021-01-24T17:46:45Z

To keep track of an observation, below is the error I saw in the first global testing run. With the current version of this PR the same test passes.

For completeness: the dreamplace code used here is ~1 year behind github.
The non-matching args were: tensor(3.), tensor(3.)

  File "dreamplace/ops/electric_potential/electric_overflow.py", line 240, in forward
    self.num_threads
TypeError: fixed_density_map(): incompatible function arguments. The following argument types are supported:
    1. (arg0: at::Tensor, arg1: at::Tensor, arg2: at::Tensor, arg3: at::Tensor, arg4: at::Tensor, arg5: float, arg6: float, arg7: float, arg8: float, arg9: float, arg10: float, arg11: int, arg12: int, arg13: int, arg14: int, arg15: int, arg16: int, arg17: int) -> at::Tensor

Invoked with: tensor([ 124.0602,    0.0000,  499.0000,    0.0000,  315.0000,   85.0000,
         124.0602,  100.0000,  499.0000,  101.0000,   65.0000,  105.0000]), tensor([   1.8796,    0.0000,    0.0000,    0.0000,  120.0000,   80.0000]), tensor([   1.8796,    0.0000,    0.0000,    0.0000,  120.0000,   40.0000]), tensor([  31.2500,   93.7500,  156.2500,  218.7500,  281.2500,  343.7500,
         406.2500,  468.7500]), tensor([  31.2500,   93.7500,  156.2500,  218.7500,  281.2500,  343.7500,
         406.2500,  468.7500]), 0.0, 0.0, 500.0, 500.0, tensor(62.5000), tensor(62.5000), 1, 5, 8, 8, tensor(3.), tensor(3.), 8

YannickJadoul · 2021-01-25T00:16:20Z

To keep track of an observation, below is the error I saw in the first global testing run. With the current version of this PR the same test passes.

This should now also be covered by our own tests. I added the TypeErrorThought (yes, this naming joke got out of hand quite quickly) in caa5382, which demonstrated the observed failure. And of course, the current version of the passes this new test.

rwgk

This PR passed the Google global testing now.
Thanks @YannickJadoul!

henryiii · 2021-01-25T04:41:39Z

tests/test_builtin_casters.py

@@ -256,6 +256,13 @@ class DeepThought(object):
        def __int__(self):
            return 42

+    class DoubleThought(object):


Double meaning it has index and int?

Yes, that's what I meant :-) As admitted yesterday:

yes, this naming joke got out of hand quite quickly

Should I still pick better names?

Yes. IntIndexThought, at least. :)

Done so. This should make things more clear? More boring as well, but ...

7fd2db5 has only changes to our own tests, so if these pass, look good/better to you, and you're happy with the way DoubleThought/IntAndIndex is handled, then we can merge this, @henryiii

Yes. IntIndexThought, at least. :)

I liked the mix of HHGG and 1984, but yes ;-)

henryiii · 2021-01-25T19:55:52Z

LGTM!

YannickJadoul · 2021-01-25T20:05:33Z

Thanks, all! Another tiny bit of progress :-)

YannickJadoul mentioned this pull request Jan 16, 2021

Only allow integer type_caster to call __int__ method when conversion is allowed; always call __index__ #2698

Merged

YannickJadoul added this to the v2.6.2 milestone Jan 17, 2021

YannickJadoul force-pushed the noconvert-int-index-pre-3.8 branch from 6c44020 to 657d7f6 Compare January 17, 2021 00:56

YannickJadoul commented Jan 17, 2021

View reviewed changes

YannickJadoul added 2 commits January 17, 2021 03:02

Always call PyNumber_Index when casting from Python to a C++ integral…

2231d9c

… type, also pre-3.8

Fixed on PyPy

2eb35ac

YannickJadoul force-pushed the noconvert-int-index-pre-3.8 branch from 657d7f6 to 2eb35ac Compare January 17, 2021 02:03

YannickJadoul mentioned this pull request Jan 17, 2021

Factoring out make_constructor from type_caster_base (for reuse under PR #2672). #2798

Closed

rwgk reviewed Jan 20, 2021

View reviewed changes

Simplify use of PyNumber_Index, following @rwgk's idea, and ignore wa…

064d671

…rnings in >=3.8

YannickJadoul requested a review from bstaletic January 20, 2021 21:53

bstaletic approved these changes Jan 21, 2021

View reviewed changes

Fix tests on 3.6 <= Python < 3.8

cffc586

YannickJadoul force-pushed the noconvert-int-index-pre-3.8 branch from 4770ac1 to cffc586 Compare January 22, 2021 19:04

No, I don't have an uninitialized variable

8356073

YannickJadoul force-pushed the noconvert-int-index-pre-3.8 branch from 309e6a8 to 8356073 Compare January 22, 2021 19:35

Fix use of __index__ on Python 2

2092295

YannickJadoul force-pushed the noconvert-int-index-pre-3.8 branch from fac0aa0 to 2092295 Compare January 23, 2021 17:46

YannickJadoul commented Jan 23, 2021

View reviewed changes

rwgk approved these changes Jan 25, 2021

View reviewed changes

henryiii approved these changes Jan 25, 2021

View reviewed changes

Make types in test_int_convert more ~boring~ descriptive

7fd2db5

YannickJadoul merged commit 0bb8ca2 into pybind:master Jan 25, 2021

YannickJadoul deleted the noconvert-int-index-pre-3.8 branch January 25, 2021 20:05

github-actions bot added the needs changelog Possibly needs a changelog entry label Jan 25, 2021

henryiii removed the needs changelog Possibly needs a changelog entry label Jan 25, 2021

henryiii mentioned this pull request Jan 26, 2021

chore: prepare for the 2.6.2 release #2821

Merged

YannickJadoul mentioned this pull request Jan 31, 2021

chore: get PyPy 3.7 wheels using NumPy 1.20 #2837

Merged

henryiii mentioned this pull request Feb 8, 2022

fix: __index__ on Enum should always be present. #3700

Merged

rwgk mentioned this pull request Feb 10, 2023

FWD pybind11 google/pybind11k#2801

Closed

Always call PyNumber_Index when casting from Python to a C++ integral type, also pre-3.8 #2801

Always call PyNumber_Index when casting from Python to a C++ integral type, also pre-3.8 #2801

Conversation

YannickJadoul commented Jan 16, 2021

Description

Suggested changelog entry:

YannickJadoul commented Jan 17, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii commented Jan 17, 2021

YannickJadoul commented Jan 17, 2021

henryiii commented Jan 19, 2021

YannickJadoul commented Jan 19, 2021

rwgk commented Jan 19, 2021

rwgk commented Jan 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwgk commented Jan 20, 2021

YannickJadoul commented Jan 20, 2021

bstaletic left a comment

Choose a reason for hiding this comment

rwgk commented Jan 22, 2021

rwgk commented Jan 22, 2021

henryiii commented Jan 22, 2021

YannickJadoul commented Jan 22, 2021

rwgk commented Jan 22, 2021

henryiii commented Jan 22, 2021

henryiii commented Jan 22, 2021

rwgk commented Jan 22, 2021 via email

YannickJadoul commented Jan 22, 2021

rwgk commented Jan 22, 2021

YannickJadoul commented Jan 22, 2021

rwgk commented Jan 22, 2021

YannickJadoul commented Jan 22, 2021

YannickJadoul commented Jan 22, 2021 • edited

YannickJadoul commented Jan 23, 2021

YannickJadoul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwgk commented Jan 24, 2021

rwgk commented Jan 24, 2021

YannickJadoul commented Jan 25, 2021

rwgk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii commented Jan 25, 2021

YannickJadoul commented Jan 25, 2021

YannickJadoul commented Jan 17, 2021 •

edited

YannickJadoul commented Jan 22, 2021 •

edited