bpo-44863: In TypedDict allow inherit from Generic and preserve bases

[sransara]

Copied from: https://bugs.python.org/issue44863

TypedDict PEP-589 says:
A TypedDict cannot inherit from both a TypedDict type and a non-TypedDict base class.

So the current implementation has:
if type(base) is not _TypedDictMeta: raise TypeError(...)

This restricts the user from defining generic TypedDicts in the natural class based syntax:
class Pager(TypedDict, Generic[T]): ...

Although PEP 589 doesn't explicitly state generic support, I believe it is complete in covering the specification even if generics were involved (at least for the class based syntax).

I have tried putting together a PEP from guidance of typing-sig https://github.com/sransara/py-generic-typeddict/blob/master/pep-9999.rst. There is not much new contributions by that draft, except for specifying the alternative syntax and being more explicit about Generics.

So I'm wondering if it would be possible to relax the constraint: TypedDict inheritance to include Generic. In my point of view Generic is more of a mixin, so it doesn't go against the PEP 589. Or is this change big enough to warrant a PEP?

https://bugs.python.org/issue44863

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@sransara

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

[uriyyo]

Could you please add News file using blurb? You can find more information here.

[sransara]

I have one issue that I'm not sure if needs to be resolved.

_GenericAlias doesn't proxy the dunder attributes. So we are not able to say:

class A(TypedDict, Generic[T]):
    foo: T

assert(A[str].__required_keys__  == frozenset(['foo']) # Raises Attribute error
assert(A[str].__origin__.__required_keys__  == frozenset(['foo']) # Works

I'm just mulling over if the __class_getitem__ should also be overridden just to override the __getattr__.

Any opinions or ideas?

Edit 1: In a following commit, I chose to override __getitem__ and make it return from types.GenericAlias, because that seems to do the __origin__ proxying as I wanted (I'm not entirely sure how types.GenericAlias does the job though).

[uriyyo]

I think there are no need to define __getitem__ method as far as Generic class has __class_getitem__ method.

[sransara]

I had two issues by not overriding the default:

1. Dunder attributes are not proxied

class A(TypedDict, Generic[T]):
    foo: T

assert(A[str].__required_keys__  == frozenset(['foo']) # Raises Attribute error
assert(A[str].__origin__.__required_keys__  == frozenset(['foo']) # Works

2. May be a bug in the original code. But even without the changes in this PR, following is valid at runtime (digging in what I think happens is that it inherits a __getitem__ from dict meta):

class G(TypedDict):
    g: str

print(G[int]) # G is subscriptable although not a Generic

[uriyyo]

@Fidget-Spinner What is your opinion about issue mentioned by Samodya?

[Fidget-Spinner]

For 1st issue:
That happens because _BaseGenericAlias.__getattr__ intentionally doesn't proxy dunders to self.__origin__ https://github.com/python/cpython/blob/main/Lib/typing.py#L966. I don't know what's the correct behavior until I read the final version of the proposed PEP update/ PEP. Sorry.

For the 2nd behavior:
Since python 3.9, builtin generics support subscripting thanks to PEP 585. E.g. dict[str, int]. Since subclassing TypedDict subclasses dict, it also pulls dict.__class_getitem__ (and hence the type supports arbitrary subscripting). A quick way to verify this:

>>> type(G[int])
<class 'types.GenericAlias'>
>>> G.__mro__
(<class '__main__.G'>, <class 'dict'>, <class 'object'>)

[sransara]

Re 1: Dunder proxying behavior isn't specified in the PEP draft either :) I switched gears to add this feature through a bpo and implementation as I am not sure whether this change needed a PEP/PEP update (my reasoning in the ticket description). By any chance do you know a PEP that specifies the behavior of GenericAlias for another type?

Re 2: I see. Thank you. But that does seem like something that shouldn't be allowed for a non generic Typeddict (Not a big deal though since it doesn't really do anything).

[sransara]

Considering following is the behavior for normal Generic class, I will revert the overriding of __getitem__ and not proxy the dunder attributes.

class A(Generic[T]):
    a: T

print(A.__annotations__) # {'a': ~T}
print(A[str].__annotations__) # raises AttributeError
print(A[str].__origin__.__annotations__) # {'a': ~T}

[sobolevn]

Awesome feature! I've been waiting for it for a long time 🎉

[Fidget-Spinner]

I left some small notes below. I'll try to do a full and more thorough review eventually when the PEP solidifies. Thanks for moving this change forward!

[Fidget-Spinner]

Is there a reason why you're using the builtin types.GenericAlias vs the typing typing._GenericAlias version?

IMO, I'd prefer you use the typing version. It's much easier to subclass and customize. It also supports older Python versions if you're porting this to typing_extensions. types.GenericAlias only works for Python 3.9 onwards. (Also, I think there isn't a precedent for things in typing returning a types.GenericAlias.)

[sransara]

types.GenericAlias was chosen just to circumvent the fact that typing._GenericAlias doesn't proxy the dunder attributes (specifically __required_keys__, __optional_keys__ etc.). Given your reasoning we can make the change to override __getattr__ of typing._GenericAlias. That is assuming we want to proxy those attributes.

[Fidget-Spinner]

For 1st issue:
That happens because _BaseGenericAlias.__getattr__ intentionally doesn't proxy dunders to self.__origin__ https://github.com/python/cpython/blob/main/Lib/typing.py#L966. I don't know what's the correct behavior until I read the final version of the proposed PEP update/ PEP. Sorry.

For the 2nd behavior:
Since python 3.9, builtin generics support subscripting thanks to PEP 585. E.g. dict[str, int]. Since subclassing TypedDict subclasses dict, it also pulls dict.__class_getitem__ (and hence the type supports arbitrary subscripting). A quick way to verify this:

>>> type(G[int])
<class 'types.GenericAlias'>
>>> G.__mro__
(<class '__main__.G'>, <class 'dict'>, <class 'object'>)

[sransara]

@serhiy-storchaka Agreed that this PR shouldn't try to meddle with previously allowed behavior in non generic TypedDicts. Using the suggestion from your comment, the commit 80e9104 reverts forbidding of parametrization of non-generic TypedDict types.

[JelleZijlstra]

We're missing one big thing: documentation. We should add a versionchanged: 3.11 note in https://docs.python.org/3.11/library/typing.html#typing.TypedDict saying that TypedDicts can now be generic.

[JelleZijlstra]

I see the existing code did this already, but checking the class name only seems wrong. Let me see if I can find a user-visible bug based on this.

[serhiy-storchaka]

It could be written as type(cls) != _ProtocolMeta, but let left this to other issue.

[serhiy-storchaka]

It could be written as type(cls) != _ProtocolMeta, but let left this to other issue.

[JelleZijlstra]

I'll update the branch to apply Serhyi's suggestions and add docs.

@serhiy-storchaka are you OK with merging this and the generic namedtuple PR once your comments are resolved?

[serhiy-storchaka]

Yes, I am OK with this. I just want to write tests that subscription works for arbitrary TypedDicts and named tuples (since the idea of forbidding it was rejected).

[davidfstr]

I just want to write tests that subscription works for arbitrary TypedDicts and named tuples (since the idea of forbidding it was rejected).

At least on my end I was rejecting subscription arbitrary TypedDicts in previous Python versions that happened to be forward-compatible with the use of generic TypeVars proposed in this thread.

I think it is questionable to allow subscripting TypedDicts in a way incompatible with this thread since then the semantics of that subscripting isn't clear.

[JelleZijlstra]

At least on my end I was rejecting subscription arbitrary TypedDicts in previous Python versions that happened to be forward-compatible with the use of generic TypeVars proposed in this thread.

Subscripting TypedDict classes worked in 3.9 and 3.10, but not 3.8. I assume that was accidental. But now that the cat is out of the bag, I'd rather not change the user-visible behavior.

[theahura]

@JelleZijlstra just to clarify, there are currently no plans to support generic typeddicts in 3.8? That would be a bummer

[AlexWaygood]

@JelleZijlstra just to clarify, there are currently no plans to support generic typeddicts in 3.8? That would be a bummer

Python 3.8 is in security-only mode (and has been for a while). This is a new feature, so being able to create generic typing.TypedDicts won't be backported; you'll only get it on Python 3.11+.

typing_extensions.TypedDict (the third-party backport) now supports generic TypedDicts, however. If you're using pyright, you should be able to use generic typing_extensions.TypedDicts on Python 3.8 right now. (Mypy doesn't support them yet; see python/mypy#3863. I don't know about other type-checkers.)