Can't create generic NamedTuple as of py3.9

[FHTMitchell]

BPO	43923
Nosy	@gvanrossum, @rhettinger, @ericvsmith, @serhiy-storchaka, @graingert, @ilevkivskyi, @dlukes, @JelleZijlstra, @FHTMitchell, @sobolevn, @Fidget-Spinner, @AlexWaygood, @juliusgeo
PRs	bpo-43923: Revert "bpo-40185: Refactor typing.NamedTuple (GH-19371)" #31679 bpo-43923: Allow NamedTuple multiple inheritance #31779 gh-116241: Add support of multiple inheritance with typing.NamedTuple #31781

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-23.16:22:56.566>
labels = ['type-bug', '3.9', '3.10', '3.11']
title = "Can't create generic NamedTuple as of py3.9"
updated_at = <Date 2022-03-10.21:38:13.215>
user = 'https://github.com/FHTMitchell'

bugs.python.org fields:

activity = <Date 2022-03-10.21:38:13.215>
actor = 'behackett'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-04-23.16:22:56.566>
creator = 'FHTMitchell'
dependencies = []
files = []
hgrepos = []
issue_num = 43923
keywords = ['patch']
message_count = 25.0
messages = ['391705', '406083', '406124', '414541', '414557', '414558', '414559', '414563', '414567', '414568', '414570', '414584', '414585', '414589', '414592', '414593', '414705', '414709', '414754', '414791', '414792', '414815', '414816', '414817', '414830']
nosy_count = 16.0
nosy_names = ['gvanrossum', 'rhettinger', 'eric.smith', 'python-dev', 'serhiy.storchaka', 'graingert', 'levkivskyi', 'dlukes', 'JelleZijlstra', 'behackett', 'FHTMitchell', 'Steven Silvester', 'sobolevn', 'kj', 'AlexWaygood', 'juliusgeo']
pr_nums = ['31679', '31779', '31781']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43923'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[FHTMitchell]

As of python 3.9, you now can't have multiple inheritance with typing.NamedTuple subclasses. This seems sensible, until you realise that typing.Generic works via inheritance. This fails whether or not from __future__ import annotations is enabled.

example:

class Group(NamedTuple, Generic[T]):
     key: T
     group: List[T]
 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-fc238c4826d7> in <module>
----> 1 class Group(NamedTuple, Generic[T]):
      2     key: T
      3     group: List[T]
      4 

~/.conda/envs/py39/lib/python3.9/typing.py in _namedtuple_mro_entries(bases)
   1818 def _namedtuple_mro_entries(bases):
   1819     if len(bases) > 1:
-> 1820         raise TypeError("Multiple inheritance with NamedTuple is not supported")
   1821     assert bases[0] is NamedTuple
   1822     return (_NamedTuple,)

TypeError: Multiple inheritance with NamedTuple is not supported

This worked fine in python 3.7 and 3.8 and as I understand it was one of the motivating cases for PEP-560.

The change was made as part of bpo-40185: Refactor typing.NamedTuple. Whilst the obvious alternative is "use dataclasses", they don't have the same runtime properties or implications as namedtuples.

[dlukes]

This is unfortunate, especially since it used to work... Going forward, is the intention not to support this use case? Or is it possible that support for generic NamedTuples will be re-added in the future?

[rhettinger]

+1 for reverting this change and restoring the previous behavior.

[gvanrossum]

Couldn't there be a subtler solution than rolling back #63570?

[JelleZijlstra]

Was this ever documented to work? We have now disallowed this behavior in 3.9 and 3.10 with few complaints, so it doesn't seem that important to restore it. Also, static type checkers generally disallow generic NamedTuples.

[gvanrossum]

Mypy seems to allow this:

from typing import NamedTuple, TypeVar, Generic, List, Tuple

T = TypeVar("T")

class New(NamedTuple, Generic[T]):
    x: List[T]
    y: Tuple[T, T]

It's true that pyright doesn't, but maybe that's because it doesn't work in 3.9-3.10?

[JelleZijlstra]

It doesn't really. If you do x = New([1], (2, 3)) you get:

main.py:11: error: List item 0 has incompatible type "int"; expected "T"
main.py:11: error: Argument 2 to "New" has incompatible type "Tuple[int, int]"; expected "Tuple[T, T]"

https://mypy-play.net/?mypy=latest&python=3.10&gist=a13c7a33c55a3aeee95324d46cd03ffd

[gvanrossum]

So if it doesn't work in mypy why bother making it work at runtime? Is there an actual use case that broke? (There might be, but probably esoteric if nobody's run into it until now.)

[AlexWaygood]

I actually have quite a few use cases for this feature. It's true that type checkers don't (yet) support it, but that doesn't mean that it should be disallowed at runtime. In fact, allowing it at runtime will surely give type checkers room to experiment with implementing this feature if it is requested by enough users. As it is, they are blocked from doing so.

[AlexWaygood]

Is there an actual use case that broke?

No, because this was never usable in the first place. But there are those who wish it were usable :)

[serhiy-storchaka]

Please don't revert all changes. It will not make it working right in any case. See bpo-44863 for more correct approach.

[gvanrossum]

Can you be more specific about your use cases?

[AlexWaygood]

Consider the typeshed stub for concurrent.futures.DoneAndNotDoneFutures. At runtime this is a collections.namedtuple, but in the stub, we need it to be generic to allow precise type inference. But we can't have a generic NamedTuple, so the stub is currently this:

class DoneAndNotDoneFutures(Sequence[set[Future[_T]]]):
    @property
    def done(self) -> set[Future[_T]]: ...
    @property
    def not_done(self) -> set[Future[_T]]: ...
    def __new__(_cls, done: set[Future[_T]], not_done: set[Future[_T]]) -> DoneAndNotDoneFutures[_T]: ...
    def __len__(self) -> int: ...
    @overload
    def __getitem__(self, __i: SupportsIndex) -> set[Future[_T]]: ...
    @overload
    def __getitem__(self, __s: slice) -> DoneAndNotDoneFutures[_T]: ...

Until two days ago, this stub actually had a bug: done and not_done were both given as writeable attributes, whereas they are read-only properties at runtime.

With generic NamedTuples, we could write the stub for the class far more simply (and more accurately) like this:

class DoneAndNotDoneFutures(NamedTuple, Generic[_T]):
    done: set[Future[_T]]
    not_done: set[Future[_T]]

And in code that actually needs to run at runtime, I frequently find it frustrating that I have to use dataclasses instead of NamedTuples if I want a simple class that just happens to be generic. dataclasses are great, but for small, lightweight classes, I prefer to use NamedTuples where possible. I often find that I don't need to use the full range of features dataclasses provide; and NamedTuples are often more performant than dataclasses, especially in cases where there's a lot of tuple unpacking.

[gvanrossum]

Okay, that's a sensible use case.

I do doubt your intuition of preferring named tuples over dataclasses a bit. This seems to encourage premature optimization. I'd say for simple cases use plain tuples (most performant), for complex cases use dataclasses (named fields and many other features that you may eventually want).

Compare concurent.futures.wait()'s return type (a named tuple) to asyncio.tasks.wait()'s return type (a plain tuple). I don't think that naming the fields of the return tuple (awkwardly :-) makes the c.f.wait() API easier to understand than the asyncio.wait() API.

Maybe named tuples, like typed dicts, are "in-between" solutions on the spectrum of data types (tuple - named tuple - dataclass; dict - typed dict - dataclass), and we should encourage people to use the neighboring solutions instead.

I'd rather spend efforts making dataclasses faster than adding features to named tuples.

[AlexWaygood]

I sense we'll have to agree to disagree on the usefulness of NamedTuples in the age of dataclasses :)

For me, I find the simplicity of the underlying idea behind namedtuples — "tuples with some properties bolted on" — very attractive. Yes, standard tuples are more performant, but it's great to have a tool in the arsenal that's essentially the same as a tuple (and is backwards-compatible with a tuple, for APIs that require a tuple), but can also, like dataclasses, be self-documenting. (You're right that DoneAndNotDoneFutures isn't a great example of this.)

But I agree that this shouldn't be a priority if it's hard to accomplish; and there'll certainly be no complaints from me if energy is invested into making dataclasses faster.

[graingert]

The main advantage for my usecase is support for heterogeneous unpacking

On Sat, Mar 5, 2022, 6:04 PM Alex Waygood <report@bugs.python.org> wrote:

Alex Waygood <Alex.Waygood@Gmail.com> added the comment:

I sense we'll have to agree to disagree on the usefulness of NamedTuples
in the age of dataclasses :)

For me, I find the simplicity of the underlying idea behind namedtuples —
"tuples with some properties bolted on" — very attractive. Yes, standard
tuples are more performant, but it's great to have a tool in the arsenal
that's essentially the same as a tuple (and is backwards-compatible with a
tuple, for APIs that require a tuple), but can also, like dataclasses, be
self-documenting. (You're right that DoneAndNotDoneFutures isn't a great
example of this.)

But I agree that this shouldn't be a priority if it's hard to accomplish;
and there'll certainly be no complaints from me if energy is invested into
making dataclasses faster.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43923\>

---

[StevenSilvester]

The use case that prompted #31679 is that we are adding typings to PyMongo. We are late to using typings, because we only recently dropped Python 2.7 support.

We have an existing options class that subclasses NamedTuple. We would like to make that class Generic, but are currently blocked.

Our current workaround is to create a separate stub file that uses class CodecOptions(Tuple, Generic[T]) and explicitly re-declares the NamedTuple API.

Switching to dataclass would be disruptive, since we still support Python 3.6 and only rely on the standard library. We would also require a major version update since it would be an API change.

[gvanrossum]

Playing tricks where compile-time and run-time see slightly different types is probably more productive than trying to revert a PR that was in Python 3.9 and 3.10. :-)

I'm not opposed to supporting generic NamedTuple, but I expect the fix will never hit 3.9 and 3.10, and it needs to be a "fix forward" PR.

Would you mind closing the "revert" PR unmerged?

[StevenSilvester]

I agree we're stuck with the typing stub workaround for our use case. We can re-submit a "fix forward" PR.

[serhiy-storchaka]

PR 31781 is a simple PR which enables multiple inheritance with NamedTuple. As a side effect, it adds support of generic NamedTuple.

I am not sure that all details work as expected. It is easy to limit multiple inheritance only for Generic if needed.

[AlexWaygood]

+1 for the more minimal changeset proposed in PR 31781. I've never felt a need for NamedTuple multiple inheritance other than with Generic, so wouldn't be opposed to restricting it only to Generic.

[juliusgeo]

What about Protocol? It is possible to create a dataclass that is a protocol, so it would be nicer from a symmetry perspective to allow it on both dataclasses and NamedTuples.

[dendron2000]

Hi, just saying that I have ran into that issue as well. My use case is packing some arguments for a function call which would look somewhat like this (a much simplified example):

T = TypeVar("T")

class MessageCallArgs(NamedTuple, Generic[T]):
    caller: Actor
    actorAddress: TypedActorAddress[T]

so that later a function can be called:

class A:
    @staticmethod
    def foo(caller: Actor, actorAddress: TypedActorAddress["A"], name: str):
        # Remote call here

callArgs = MessageCallArgs(caller, actorAddress)
A.foo(*callArgs , "blah-blah")

@dataclass is simply not usable in this case as it does not implicitly convert to a tuple and therefore cannot be straightforwardly used for arguments unpacking. A.foo(*dataclass.astuple(callArgs)... looks super-ugly and superfluous.

Using just NamedTuple looses type information which is useful for type-checking.

So far I have reverted to using just tuple[T] but this looses all benefits of NamedTuple, like named fields and a user-defined type.

It seems to be me that the reason to forbid the creation of generic named tuples is purely technical, which is unfortunate. I do not think that this case is rare, contrary to what has been said in this thread.

[serhiy-storchaka]

It did not actually worked in 3.8:

>>> class Group(NamedTuple, Generic[T]):
...      key: T
...      group: List[T]
... 
>>> Group[int]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'type' object is not subscriptable
>>> Group.__bases__
(<class 'tuple'>,)

A NamedTuple is not generic, it is just that other base classes were silently ignored.

[serhiy-storchaka]

#92027 is a limited version of #31781 that only allows combining NamedTuple with Generic.

[serhiy-storchaka]

There is a problem related to the fact that tuple has the __class_getitem__ method. And if Generic[T] follows NamedTuple, it is tuple.__class_getitem__ called when subscript the type.

It is not specific to NamedTuple, this problem exists when you inherit from any generic class and Generic:

>>> from typing import *
>>> T = TypeVar('T')
>>> class A(tuple, Generic[T]): ...
... 
>>> class B(Generic[T], tuple): ...
... 
>>> A.__bases__
(<class 'tuple'>, <class 'typing.Generic'>)
>>> B.__bases__
(<class 'typing.Generic'>, <class 'tuple'>)
>>> A[int]
__main__.A[int]
>>> B[int]
__main__.B[int]
>>> type(A[int])
<class 'types.GenericAlias'>
>>> type(B[int])
<class 'typing._GenericAlias'>
>>> A[int, str]
__main__.A[int, str]
>>> B[int, str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 332, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 1767, in __class_getitem__
    _check_generic(cls, params, len(cls.__parameters__))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 253, in _check_generic
    raise TypeError(f"Too {'many' if alen > elen else 'few'} arguments for {cls};"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Too many arguments for <class '__main__.B'>; actual 2, expected 1

And named tuple types are unexpectedly subscribtable without Generic (since they are tuple subclasses):

>>> class NT(NamedTuple):
...     x: int
... 
>>> NT[str, float]
__main__.NT[str, float]

The problem is how to create non-generic (or generic with the specified parameters) subclasses of generic types.

[JelleZijlstra]

@serhiy-storchaka good catch! I'm not too concerned about non-generic subclasses of tuple being subscriptable. In general there are a lot of things you can do around annotations that are meaningless to type checkers, and we can just leave it to the static checkers to detect that.

But this does also affect NamedTuple with your PR:

>>> from typing import *
>>> T = TypeVar("T")
>>> U = TypeVar("U")
>>> 
>>> class N1(NamedTuple, Generic[T, U]):
...     a: T
...     b: U
... 
>>> class N2(Generic[T, U], NamedTuple):
...     a: T
...     b: U
... 
>>> N1[int, str]
__main__.N1[int, str]
>>> N2[int, str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jelle/py/cpython/Lib/typing.py", line 329, in inner
    return cached(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jelle/py/cpython/Lib/typing.py", line 1761, in __class_getitem__
    if any(isinstance(t, ParamSpec) for t in cls.__parameters__):
                                             ^^^^^^^^^^^^^^^^^^
AttributeError: type object 'N2' has no attribute '__parameters__'

That's pretty confusing behavior. (I guess this is why you added the "do not merge" label to #92027.)

What if we just do this?

diff --git a/Lib/typing.py b/Lib/typing.py
index 5b34508edc..e8fc840f1e 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -2768,7 +2768,10 @@ def __new__(cls, typename, bases, ns):
             if base is not _NamedTuple and base is not Generic:
                 raise TypeError(
                     'can only inherit from a NamedTuple type and Generic')
-        bases = tuple(tuple if base is _NamedTuple else base for base in bases)
+        if Generic in bases:
+            bases = (tuple, Generic)
+        else:
+            bases = (tuple,)
         types = ns.get('__annotations__', {})
         default_names = []
         for field_name in types:

Then things work as expected with either order for the base classes.

[serhiy-storchaka]

Consider the following example:

>>> import collections
>>> from typing import *
>>> T = TypeVar('T')
>>> Group = collections.namedtuple('Group', 'key group')[T, list[T]]

It looks somewhere like a generic named tuple. But it conflicts with what is proposed here.

[gvanrossum]

So that just inherits __class_getitem__ (CORRECTED) from tuple. Isn't it the job of the static checker to reject this? Runtime allows all kinds of nonsense, like list[int,int,int]. Is this any different?

[gvanrossum]

In any case if you want to make a class that isn't indexable but inherits from a class that is (e.g. namedtuple inheriting from tuple) you can just set __class_getitem__ = None in the class.

[AlexWaygood]

I agree with Guido.

[serhiy-storchaka]

In any case if you want to make a class that isn't indexable but inherits from a class that is (e.g. namedtuple inheriting from tuple) you can just set __class_getitem__ = None in the class.

#92106, #92107.

I have also opened a topic on Python-Dev about blessing the __dunder__ = None idiom: https://mail.python.org/archives/list/python-dev@python.org/thread/YGAK34DRWJFSIV2VZ4NC2J24XO37GCMM/