Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663

Merged
merged 40 commits into from May 3, 2022
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
6cfdc5f
Allow TypedDict to inherit from Generic
sransara Aug 7, 2021
c022e44
TypedDict preserve MRO
sransara Aug 7, 2021
2fa00b8
Fix base classes to be just generic and dict
sransara Sep 3, 2021
b829d8a
Simplify addiotion of Generic base class
sransara Sep 3, 2021
c5762df
Fully proxy origin by using types.GenericAlias
sransara Sep 3, 2021
4c5db0a
Add NEWS blurb
sransara Sep 3, 2021
34189dd
Add better NEWS blurb
sransara Sep 3, 2021
0cfd637
Add more testcases
sransara Sep 3, 2021
5e808af
Add implicit any test case
sransara Sep 4, 2021
a2482ff
Fix issue with plain generic inheritance
sransara Sep 4, 2021
071c1b6
Include bases assertion
sransara Sep 4, 2021
3ce517b
Update NEWS blurb with better formatting
sransara Sep 5, 2021
2f003e0
Revert overriding of getitem and not proxy dunders
sransara Sep 5, 2021
964f7d3
Use alternative way to find if Generic base is needed
sransara Sep 6, 2021
c3c0e51
Make it clear when Generic base is included
sransara Sep 6, 2021
d34c99e
Add getitem:so TD is subscriptable only if Generic
sransara Sep 6, 2021
cea66c4
Fix test consistency for empty params
sransara Sep 6, 2021
0ed4326
Test for adding new generic arg in child class
sransara Sep 6, 2021
b1fcd16
Update Lib/typing.py
sransara Apr 30, 2022
2905f29
Merge branch 'main' into py-generic-typeddict-simple
serhiy-storchaka Apr 30, 2022
afa5e51
Fix merge error.
serhiy-storchaka Apr 30, 2022
94138f6
Fix trailing spaces.
serhiy-storchaka Apr 30, 2022
2572930
Merge branch 'main' into py-generic-typeddict-simple
serhiy-storchaka May 1, 2022
c2e1d8d
Fix indentation
sransara May 2, 2022
d3d9456
Remove trailing commas in tuples
sransara May 2, 2022
311852c
Add test with flipped bases
sransara May 2, 2022
dc98753
Move implicit any test to on its own case
sransara May 2, 2022
9bed1d4
Add checks for orig_bases and mro
sransara May 2, 2022
6c152e7
Remove specialization from generic get_type_hints
sransara May 2, 2022
f88201b
Check type hints of inherited generic typeddict
sransara May 2, 2022
6ea95df
Remove unused statement
sransara May 2, 2022
80e9104
Add class method through the metaclass
sransara May 2, 2022
dbbb707
Fix generic base when inherited from implicit any
sransara May 2, 2022
56b69e0
Fix whitespacing with reindent.py
sransara May 2, 2022
4a40825
fix tuple
JelleZijlstra May 2, 2022
4b50ae2
remove unnecessary __class_getitem__ override
JelleZijlstra May 2, 2022
ecc7726
Merge remote-tracking branch 'upstream/main' into py-generic-typeddic…
JelleZijlstra May 2, 2022
5b5a983
docs
JelleZijlstra May 2, 2022
99a1430
Merge branch 'main' into py-generic-typeddict-simple
JelleZijlstra May 2, 2022
79c2bb4
Merge branch 'main' into py-generic-typeddict-simple
JelleZijlstra May 2, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
7 changes: 6 additions & 1 deletion Lib/test/_typed_dict_helper.py
Expand Up @@ -10,9 +10,14 @@ class Bar(_typed_dict_helper.Foo, total=False):

from __future__ import annotations

from typing import Optional, TypedDict
from typing import Generic, Optional, TypedDict, TypeVar

OptionalIntType = Optional[int]

class Foo(TypedDict):
a: OptionalIntType

OptionableT = TypeVar("OptionableT")

class FooGeneric(TypedDict, Generic[OptionableT]):
a: Optional[OptionableT]
29 changes: 26 additions & 3 deletions Lib/test/test_typing.py
Expand Up @@ -2943,13 +2943,22 @@ def __add__(self, other):

Label = TypedDict('Label', [('label', str)])

TDG = TypeVar("TDG")

class Point2D(TypedDict):
x: int
y: int

class Point2DGeneric(Generic[TDG], TypedDict):
a: TDG
b: TDG

class Bar(_typed_dict_helper.Foo, total=False):
b: int

class BarGeneric(_typed_dict_helper.FooGeneric[TDG], total=False):
b: int

class LabelPoint2D(Point2D, Label): ...

class Options(TypedDict, total=False):
Expand Down Expand Up @@ -4070,7 +4079,7 @@ def test_basics_functional_syntax(self):
self.assertEqual(jim['id'], 1)
self.assertEqual(Emp.__name__, 'Emp')
self.assertEqual(Emp.__module__, __name__)
self.assertEqual(Emp.__bases__, (dict,))
self.assertIn(dict, Emp.__bases__)
self.assertEqual(Emp.__annotations__, {'name': str, 'id': int})
self.assertEqual(Emp.__total__, True)

Expand All @@ -4085,7 +4094,7 @@ def test_basics_keywords_syntax(self):
self.assertEqual(jim['id'], 1)
self.assertEqual(Emp.__name__, 'Emp')
self.assertEqual(Emp.__module__, __name__)
self.assertEqual(Emp.__bases__, (dict,))
self.assertIn(dict, Emp.__bases__)
self.assertEqual(Emp.__annotations__, {'name': str, 'id': int})
self.assertEqual(Emp.__total__, True)

Expand Down Expand Up @@ -4135,7 +4144,7 @@ def test_py36_class_syntax_usage(self):
self.assertEqual(LabelPoint2D.__name__, 'LabelPoint2D')
self.assertEqual(LabelPoint2D.__module__, __name__)
self.assertEqual(LabelPoint2D.__annotations__, {'x': int, 'y': int, 'label': str})
self.assertEqual(LabelPoint2D.__bases__, (dict,))
self.assertIn(dict, LabelPoint2D.__bases__)
self.assertEqual(LabelPoint2D.__total__, True)
self.assertNotIsSubclass(LabelPoint2D, typing.Sequence)
not_origin = Point2D(x=0, y=1)
Expand All @@ -4157,6 +4166,17 @@ def test_pickle(self):
EmpDnew = pickle.loads(ZZ)
self.assertEqual(EmpDnew({'name': 'jane', 'id': 37}), jane)

def test_pickle_generic(self):
point = Point2DGeneric(a=5.0, b=3.0)
for proto in range(pickle.HIGHEST_PROTOCOL + 1):
z = pickle.dumps(point, proto)
point2 = pickle.loads(z)
self.assertEqual(point2, point)
self.assertEqual(point2, {'a': 5.0, 'b': 3.0})
ZZ = pickle.dumps(Point2DGeneric, proto)
Point2DGenericNew = pickle.loads(ZZ)
self.assertEqual(Point2DGenericNew({'a': 5.0, 'b': 3.0}), point)

def test_optional(self):
EmpD = TypedDict('EmpD', name=str, id=int)

Expand Down Expand Up @@ -4228,6 +4248,9 @@ def test_get_type_hints(self):
{'a': typing.Optional[int], 'b': int}
)

def test_get_type_hints_generic(self):
assert set(get_type_hints(BarGeneric)) == {'a', 'b'}
sransara marked this conversation as resolved.
Show resolved Hide resolved


class IOTests(BaseTestCase):

Expand Down
15 changes: 13 additions & 2 deletions Lib/typing.py
Expand Up @@ -2252,10 +2252,12 @@ def __new__(cls, name, bases, ns, total=True):
Subclasses and instances of TypedDict return actual dictionaries.
"""
for base in bases:
if type(base) is not _TypedDictMeta:
if not (type(base) is _TypedDictMeta or base is Generic):
sransara marked this conversation as resolved.
Show resolved Hide resolved
raise TypeError('cannot inherit from both a TypedDict type '
'and a non-TypedDict base class')
tp_dict = type.__new__(_TypedDictMeta, name, (dict,), ns)

generic_base = (Generic,) if any(issubclass(b, Generic) for b in bases) else ()
tp_dict = type.__new__(_TypedDictMeta, name, (*generic_base, dict,), ns)
JelleZijlstra marked this conversation as resolved.
Show resolved Hide resolved

annotations = {}
own_annotations = ns.get('__annotations__', {})
Expand Down Expand Up @@ -2294,6 +2296,15 @@ def __subclasscheck__(cls, other):

__instancecheck__ = __subclasscheck__

def __getitem__(cls, params):
if not isinstance(params, tuple):
params = (params,)
msg = "Parameters to generic types must be types."
params = tuple(_type_check(p, msg) for p in params)
nparams = len(cls.__dict__.get("__parameters__", ()))
_check_generic(cls, params, nparams)
return types.GenericAlias(cls, params)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are no need to define __getitem__ method as far as Generic class has __class_getitem__ method.

Copy link
Contributor Author

@sransara sransara Sep 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had two issues by not overriding the default:

  1. Dunder attributes are not proxied
class A(TypedDict, Generic[T]):
    foo: T

assert(A[str].__required_keys__  == frozenset(['foo']) # Raises Attribute error
assert(A[str].__origin__.__required_keys__  == frozenset(['foo']) # Works
  1. May be a bug in the original code. But even without the changes in this PR, following is valid at runtime (digging in what I think happens is that it inherits a __getitem__ from dict meta):
class G(TypedDict):
    g: str

print(G[int]) # G is subscriptable although not a Generic

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Fidget-Spinner What is your opinion about issue mentioned by Samodya?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For 1st issue:
That happens because _BaseGenericAlias.__getattr__ intentionally doesn't proxy dunders to self.__origin__ https://github.com/python/cpython/blob/main/Lib/typing.py#L966. I don't know what's the correct behavior until I read the final version of the proposed PEP update/ PEP. Sorry.

For the 2nd behavior:
Since python 3.9, builtin generics support subscripting thanks to PEP 585. E.g. dict[str, int]. Since subclassing TypedDict subclasses dict, it also pulls dict.__class_getitem__ (and hence the type supports arbitrary subscripting). A quick way to verify this:

>>> type(G[int])
<class 'types.GenericAlias'>
>>> G.__mro__
(<class '__main__.G'>, <class 'dict'>, <class 'object'>)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re 1: Dunder proxying behavior isn't specified in the PEP draft either :) I switched gears to add this feature through a bpo and implementation as I am not sure whether this change needed a PEP/PEP update (my reasoning in the ticket description). By any chance do you know a PEP that specifies the behavior of GenericAlias for another type?

Re 2: I see. Thank you. But that does seem like something that shouldn't be allowed for a non generic Typeddict (Not a big deal though since it doesn't really do anything).

Copy link
Contributor Author

@sransara sransara Sep 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering following is the behavior for normal Generic class, I will revert the overriding of __getitem__ and not proxy the dunder attributes.

class A(Generic[T]):
    a: T

print(A.__annotations__) # {'a': ~T}
print(A[str].__annotations__) # raises AttributeError
print(A[str].__origin__.__annotations__) # {'a': ~T}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why you're using the builtin types.GenericAlias vs the typing typing._GenericAlias version?

IMO, I'd prefer you use the typing version. It's much easier to subclass and customize. It also supports older Python versions if you're porting this to typing_extensions. types.GenericAlias only works for Python 3.9 onwards. (Also, I think there isn't a precedent for things in typing returning a types.GenericAlias.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

types.GenericAlias was chosen just to circumvent the fact that typing._GenericAlias doesn't proxy the dunder attributes (specifically __required_keys__, __optional_keys__ etc.). Given your reasoning we can make the change to override __getattr__ of typing._GenericAlias. That is assuming we want to proxy those attributes.



def TypedDict(typename, fields=None, /, *, total=True, **kwargs):
"""A simple typed namespace. At runtime it is equivalent to a plain dict.
Expand Down
@@ -0,0 +1,4 @@
Allow ``TypedDict`` subclasses to also include ``Generic`` as a base class
in class based syntax. Thereby allowing the user to define a ``Generic
TypedDict``, just like a user defined ``Generic class`` but with
``TypedDict`` semantics.
sransara marked this conversation as resolved.
Show resolved Hide resolved