Add regression tests for builtins.pow and object.__reduce__

[AlexWaygood]

As discussed in #1339 (comment) (among other places).

At the moment, this PR has only pyright running on the test_cases directory. We can potentially add mypy once there's a mypy release that supports assert_type.

[JelleZijlstra]

Thanks for setting this up! It's going to be very helpful. I'll send PRs later for my zip_longest cases from the other day.

[Akuli]

While this is correct, it might need a tl;dr in bold. As discussed, it's difficult to convince some people that they don't need to write tests.

[AlexWaygood]

How's it look after b2e59b5?

[Akuli]

• Do we really want this to be in both pyrightconfigs?
• Does mypy support assert_type() yet? If it does, we could instead point mypy_primer at typeshed's test_cases. That way we would know if a PR to mypy breaks the tests.

[AlexWaygood]

• I don't really understand how the two pyrightconfigs work with each other ¯\_(ツ)_/¯. Happy to have it in only one if that's better.
• No, mypy doesn't support assert_type yet. The PR was recently merged, and might make it into 0.950, but also might not. See Release 0.950 planning mypy#12579 (comment)

[JelleZijlstra]

Mypy supports assert_type() on master (I added it last week). I am hoping to convince Jukka to put it in the upcoming 0.950 release. We should make mypy run on this directory too as soon as 0.950 is released, or use master if we're feeling brave.

[AlexWaygood]

Update: assert_type has been cherry-picked onto the 0.950 branch, so we should be able to point mypy_primer at this directory as soon as 0.950 is released.

[JelleZijlstra]

Hopefully we can just run mypy regularly, no need to use mypy-primer.

[hauntsaninja]

Can we keep the comments in builtins? This is a super common point of clarification and there is value in having the stubs be self-documenting.

Another suggestion could be to move this under tests. Also note we have the @tests directory for each third party stubs, which might be better since the equivalent tests for third party stubs could involve dependencies.

[AlexWaygood]

Can we keep the comments in builtins? This is a super common point of clarification and there is value in having the stubs be self-documenting.

Done (78b9dd0).

Another suggestion could be to move this under tests.

To me, it doesn't really feel like this stuff "fits in" with the other content in tests particularly well -- most of the content there is scripts for running tests, whereas this is test data. But I don't feel that strongly about it; happy to make the change if others agree. 🙂

Also note we have the @tests directory for each third party stubs, which might be better since the equivalent tests for third party stubs could involve dependencies.

Yeah, the third-party stubs would complicate things. Perhaps we could cross that bridge when we come to it?

[JelleZijlstra]

Hopefully we can just run mypy regularly, no need to use mypy-primer.

[srittau]

Thanks for doing this!

[srittau]

In the long term we should probably also allow test that fail a type checker to be able to test the negative case.

[AlexWaygood]

Yup. Once we're running mypy on this directory, we can do that by # type: ignoring (with specific error codes) lines that we want to fail and running mypy with --warn-unused-ignores. We can't use that flag globally when running mypy on typeshed, but we should be able to use it for this directory.