Representation of ParamSpecs at runtime compared to Callable

[Gobot1234]

          I think a list representation is more consistent with the text of PEP 612 and the runtime representation of Callable. We should change the runtime repr of ParamSpec generics to be more like Callable, because right now they're inconsistent:

>>> Callable[[int, str], bool]
typing.Callable[[int, str], bool]
>>> P = ParamSpec("P")
>>> T = TypeVar("T")
>>> class MyCallable(Generic[P, T]): pass
... 
>>> MyCallable[[int, str], bool]
__main__.MyCallable[(<class 'int'>, <class 'str'>), bool]
>>> get_args(Callable[[int, str], bool])
([<class 'int'>, <class 'str'>], <class 'bool'>)
>>> get_args(MyCallable[[int, str], bool])
((<class 'int'>, <class 'str'>), <class 'bool'>)

Originally posted by @JelleZijlstra in python/typing#1274 (comment)

Linked PRs

• gh-102615: Use list instead of tuple in repr of paramspec #102637
• gh-102615: Fix type vars substitution of collections.abc.Callable and custom generics with ParamSpec #102681

[Fidget-Spinner]

Just clarifying for anyone who plans to contribute: underneath both __args__ are the same (a flat non-nested tuple of t ypes) get_args just shows the string representation of Callable to wrap [] around the args.

[sobolevn]

underneath both args are the same (a flat non-nested tuple of t ypes)

No, they are different:

>>> MyCallable[[int, str], bool].__args__
((<class 'int'>, <class 'str'>), <class 'bool'>)
>>> Callable[[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)

I've sent a PR with the repr change.

[Fidget-Spinner]

underneath both args are the same (a flat non-nested tuple of t ypes)

No, they are different:

>>> MyCallable[[int, str], bool].__args__
((<class 'int'>, <class 'str'>), <class 'bool'>)
>>> Callable[[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)

I've sent a PR with the repr change.

Oh woops, I'm wrong then :). That's a bug. The tuples should be flattened and they should be the same. I remember trying to flatten all the args representations when I implemented PEP 612.

>>> Callable[P, T][[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)
>>> collections.abc.Callable[P, T][[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)

There's a very important reason why they need to be flattened -- generic expects a list of types in __args__, it does not handle tuples. So double substitution will break if there are tuples.

E.g this works.

>>> Callable[P, T][[P, str], bool][int]
typing.Callable[[int, str], bool]

But ugh these two are broken

# Broken in collections.abc :(, needs fixing
>>> collections.abc.Callable[P, T][[P, str], bool][int]
TypeError: Callable must be used as Callable[[arg, ...], result].
# Broken in typing
>>> MyCallable[P, T][[P, str], bool][int]
TypeError: __main__.MyCallable[(~P, <class 'str'>), bool] is not a generic clas

[sobolevn]

That's a bug

Yeah, I was not sure if this is intentional or a bug. Got it now!
Will change my PR to address it soon :)

[sobolevn]

After reading quite a lot of tests, I think that

>>> MyCallable[[int, str], bool].__args__
((<class 'int'>, <class 'str'>), <class 'bool'>)

is correct. For example, take a look at this test:

def test_multiple_paramspecs_in_user_generics(self):
        P = ParamSpec("P")
        P2 = ParamSpec("P2")

        class X(Generic[P, P2]):
            f: Callable[P, int]
            g: Callable[P2, str]

        G1 = X[[int, str], [bytes]]
        G2 = X[[int], [str, bytes]]
        self.assertNotEqual(G1, G2)
        self.assertEqual(G1.__args__, ((int, str), (bytes,)))
        self.assertEqual(G2.__args__, ((int,), (str, bytes)))

If we are going to flatten G1 = X[[int, str], [bytes]] to have (int, str, bytes) - it won't be clear what we mean here.

So, I think it is intentional: args are not flattened for ParamSpec substitution.

@Fidget-Spinner, however, your examples are broken indeed.

[Fidget-Spinner]

Thanks for looking into this. I vaguely recall that for user defined generics with ParamSpec, we don't flatten because of the following:

Generic[P1, int, P2]

The subsitution of the above, if flattened, would be impossible to reconstruct with get_args.

I need to dig through the history. I think I left a comment somewhere about this.

[Fidget-Spinner]

I think the only thing that needs fixing here is the repr and collections.abc.Callable then. Thanks again Nikita!

[AlexWaygood]

But ugh these two are broken

>>> # Broken in collections.abc :(, needs fixing
>>> collections.abc.Callable[P, T][[P, str], bool][int]
TypeError: Callable must be used as Callable[[arg, ...], result].
>>> # Broken in typing
>>> MyCallable[P, T][[P, str], bool][int]
TypeError: __main__.MyCallable[(~P, <class 'str'>), bool] is not a generic clas

Are these two examples valid uses of ParamSpec? I thought you had to use Concatenate if you wanted to prepend parameters to a ParamSpec variable in a parameters list. And I thought appending parameters to a ParamSpec variable in a parameters list was disallowed altogether by PEP-612.

So surely the runtime is correct in disallowing both of these: they both involve a ParamSpec being substituted with an invalid parameters list?

[Fidget-Spinner]

@AlexWaygood thanks. Sorry I recalled wrongly the semantics of ParamSpec. I now see that PEP 612 forbids these uses. So does this mean the actual fix is to

a. disallow it in typing (because currently only collections.abc and typing.Generic correctly fails) OR
b. do nothing?

which do you prefer?

@sobolevn I'm sorry for leading you on a wild goose chase. Thank you for your efforts in all of this. Sorry for wasting your time. In the end, the right fix might truly be just to fix the repr. I think I'll wait for a week or so to see the consensus on this thread.

TLDR: I'm wrong and I think Alex is right. Nikita has spent a great deal of effort on understanding this issue. So I'll defer judgement to both Nikita and Alex, I think I'm too rusty with typing right now to make the right call.

[AlexWaygood]

@Fidget-Spinner I don't think you've got anything to apologise for — the spec is horrendously complicated for PEP-612! I only spotted the error on, like, the 5th read :)

My top priority here would be improving the error message so that it's clearer for users (including typing maintainers, including me!) exactly why the runtime is raising a TypeError here. It's too easy to forget the finer points of PEP-612.

In terms of harmonising the behaviour between collections.abc.Callable and typing.Callable — if we can disallow the invalid uses in typing as well, then I'm open to it. But if it would add significant complexity, I'd probably be inclined to just leave it.

[sobolevn]

@Fidget-Spinner no worries, I've spent my time on a very interesting topic :)

I now agree that there are several related, but distinct problems:

1. repr of ParamSpec in user generics

2. typing.Callable behaves differently and allows incorrect substitution that collections.abc.Callable does not allow:

    >>> import typing
    >>> P = typing.ParamSpec('P')
    >>> T = typing.TypeVar('T')
    >>> typing.Callable[P, T][[P, str], bool][int]
    typing.Callable[[int, str], bool]

   While:

    >>> import collections.abc
    >>> collections.abc.Callable[P, T][[P, str], bool][int]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/sobolev/Desktop/cpython/Lib/_collections_abc.py", line 501, in __getitem__
        return _CallableGenericAlias(Callable, tuple(new_args))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/sobolev/Desktop/cpython/Lib/_collections_abc.py", line 456, in __new__
        raise TypeError(
    TypeError: Callable must be used as Callable[[arg, ...], result].

3. There's a suspicious if case that is probably not tested / works incorrectly: gh-102615: Fix type vars substitution of collections.abc.Callable and custom generics with ParamSpec #102681 (comment)

4. There are not enough tests for all the corner cases we have discussed here

So, my plan is:

1. Fix the repr
2. We can fix the typing.Callable to behave the same way (why?), or just ignore it, because it is quite complex to get right
3. Run tests with coverage to find out what is going on and fix it / add tests / remove these lines
4. Add new tests

Please, bare with me and my multiple PRs 😆

Thanks everyone for the productive discussion!

[sobolevn]

Oh, and better errors (if we can) as 5.

[AlexWaygood]

The original issue here has been fixed, and we won't be backporting the fix (see #102637 (review) for the rationale). As such, I'm going to close this issue and open followup issues for the other problems we identified in this thread. @sobolevn has already opened the first followup issue:

• Improve coverage of _collections_abc._CallableGenericAlias #102721

[AlexWaygood]

Two more followup issues:

• ParamSpec substitution: difference in behaviour between collections.abc.Callable and typing.Callable #102723
• Typing: improve error messages for invalid ParamSpec substitutions #102725