Feedback: experimental syntax for inlined TypedDict

[tuchandra]

In the typing-sig thread Inline syntax for TypedDict, someone proposed an experimental syntax that looks roughly like dict[{"key1": str, "key2": int}]. Shortly after, Eric Traut added experimental support to pyright. I've created this post to share my feedback, in the same spirit as my message in the PEP 695 forum thread.

Usability improvements

tl;dr I've been playing with this for a few days now and find the syntax to be intuitive and pleasant to use. It provides meaningful benefits (making typing easier, enabling types that are specific to one location, much nicer support of nested structure) without the overhead of having to define throwaway classes.

Typing one-off dicts is more "worth it"

I often run into dicts that have a known structure, but that I don't need more than once. One example is a function that reads from a file or hits some API, does something with the dict, and then discards it when returning.

• Defining a class "feels" like too much overhead; the type/class and function are defined at the same level, but the type only exists to support the function.
• There is a usability downside in having to name a new symbol.
• It's possible that someone could import the type and start using it elsewhere, divorced from the original context

Using the new syntax, I don't need to come up with a throwaway name. The type definition is inherently linked to the function it supports. No one can become an inadvertent consumer.

def f(x: dict[{"ENV_VAR": str, "OTHER_VALUE": str}]): ...

Typing nested dicts is much easier

Creating nested TypedDicts today is cumbersome. Every level of nesting has to be defined and named individually, since call expressions are not allowed in annotations. The intermediate levels of nesting are often not themselves meaningful or only meaningful in the context of the parent.

# what we have to do
IntermediateTD = TypedDict("IntermediateTD", {"y": int})  # I don't want to name this
TD = TypedDict(TD, {"x": int, "nested_key": IntermediateTD})

# this isn't allowed
class TD(TypedDict):
    x: int
    nested_key: TypedDict("_NeedsAName", { ... })  # call expression not allowed

# nor is this
TD = TypedDict("TD", {"x": int, "nested_key": TypedDict("_", { ... } )})

# this (proposed) doesn' require any intermediate defs
def f(x: dict[{"x": int, "nested_key": dict[{"y": int}]}]): ...

For the record, this is still hard to read (perhaps harder—that's what, 5 closing chars in a row?). But this kind of anonymous nesting was simply not possible before; this improves the usability of typing. There are compromises, too, where the overall type could be defined as a standalone entity with only nested_key defined inline.

IDE support

I know that language server autocompletion is not a standard, but being able to define one-off dicts creates a smoother developer experience. It's more likely that someone will add a type definition inline simply because it's easier than defining a new class.

I noticed this in test code, too. I often see/write test functions that are parameterized by a list of dicts, where each dict describes one test case. I don't think I'd ever define a TypedDict here—it feels like overkill—but defining it inline gives me LSP completion and type-checking warnings with very little extra work.

Eric noted as much in his original email and I wholeheartedly agree:

I've spent some time playing with this support, and I like it even more than I had anticipated. The language server support (completion suggestions, etc.) for TypedDict types in pyright make this especially compelling.

Finally, it's easy to adopt this gradually. We can already make an annotation x: dict incrementally more precise by adding e.g., x: dict[str, int]. The proposed syntax of x: dict[{"key": int, "key2": int}] is a natural extension of this, being a small diff that improves usability.

Closing

From Eric's original email, again:

I think this proposal is worth moving forward as a PEP. It should be a relatively short one. There's a good chance the SC will approve it because it doesn't require any changes to the runtime, it doesn't introduce any new syntax, and it composes well with existing type features.

I agree with this. The biggest downside is that it can be hard to read, but I think it's worth the tradeoff; this makes TypedDicts much more usable than they are today.

[sobolevn]

Yes, I am working on a PEP for this :)

There are several things to get right here, I will publish it soon enough (but I think, I won't make it into 3.12).

[hauntsaninja]

Why I like TypedDict[{...}]:

It's much more obvious that class TypedDict: ... and TypedDict[{...}] are the same concept. It's easier to search for and learn.

This makes dict more like a special form, whereas previously it was equally as powerful as user defined generics — and a non-obvious special form since you don't import it from typing. I don't like complicating the mental model of what symbols are special and what ones aren't.

If we end up extending what inline TypedDicts can do (e.g. @final, total=..., inheritance etc), we don't have to worry about impact of sharing the symbol with dict collections.

The goal of PEP 585 was making sure that users of untyped Python could do the intuitive thing of putting the types they already used in their annotations, instead of having a parallel set of types — not necessarily to reduce imports.

While I agree that at the margin avoiding an extra import is a nice convenience, I don't feel it's important enough to outweigh the considerations above, especially since as proposed the inline syntax is not a full substitute for the definition syntax. Obviously others may feel differently.

Another reason to feel differently is if you feel the difference between dict collections and typed dicts isn't very big.

[sobolevn]

+1 on using TypedDict[] for it

Quoting myself on this:

Rejected Ideas
==============

Reusing dict and typing.Dict types to accept a single type argument
-------------------------------------------------------------------

Right now ``builtins.dict`` and ``typing.Dict`` are regular types.
That are define like::

  class dict(MutableMapping[_KT, _VT], Generic[_KT, _VT]): ...

It follows all regular Python's convetions about generics and typevars.
It would be impossible to define overloads for one / two typing arguments
in ``typeshed`` with existing typing techniques.

So, this would make ``dict`` a special case that must be specially handled in typecheckers. This would make ``dict`` basically a special form.

Considering the imortant role that ``dict`` plays in Python, this might negatevily affect multiple use-cases and users.

Moreover, reusing ``dict`` for this purpose might not be enough: right now ``dict`` has immutable and widely used immutable typing alternative - ``Mapping``.

There's an open (at the time of writting) proposal (:pep:`705`) to add a ``TypedMapping`` type which might be used the same way for the same purpose, except being immutable.

[hauntsaninja]

Ah, good point about TypedMapping

[tmke8]

Has anyone suggested TypedDict({...}) with round brackets? It could be seen as an "anonymous TypedDict" and is easy to understand as a syntax, I think. On the other hand, I can't think of any other inline type annotations using round brackets, so maybe it's too weird.

EDIT: I think this also looks nice though that's probably subjective:

type MyDict = TypedDict({"foo": int, "bar": str})

[erictraut]

I think it's a bad idea to use call expressions within type expressions (annotations). Call expressions are relatively expensive to process, and the rules for evaluating them are not standardized in type checkers (due to differences in overload matching, constraint solving, etc.). For those reasons, I think it's a slippery slope for call expressions to appear anywhere within a type expression. Right now, all major type checkers emit errors if you attempt to use a call expression within a type annotation, and I'd like to keep it that way.

[Akuli]

After the new PEP, are there any uses for TypedDict that wouldn't be covered by the new syntax?

For example, consider:

class Foo(TypedDict):
    bar: int
    baz: str
    lol: str
    wat: float

If I understand correctly, this can now be written as:

Foo: TypeAlias = dict[{
    "bar": int,
    "baz": str,
    "lol": str,
    "wat": float,
}]

[sobolevn]

I agree. I left type aliases out of scope for now, they have quite a lot of problems.

[superbobry]

@sobolevn does that mean it is a static error to use the new syntax on the RHS of a type alias?

I think @Akuli has a point. The proposal adds a new slightly different way to define a TypedDict, which would co-exist with the old one. Is the added complexity really justified? If yes, can we find the syntax which support all TypedDict features, so that the choice between the two is purely syntactic?

[erictraut]

We could assume @final for inlined TypedDict definitions because there's no way to create a subclass from them.

I'm thinking it's not important to support total in an inlined mechanism. The total parameter is largely obsolete now that Required and NotRequired are part of the type system. If you really feel the need to specify total, you can fall back on the current class-based TypedDict declaration mechanism.

Inheritance and docstrings should be out of scope for inlined definitions, IMO. If you need these additional bells and whistles, you can use the current mechanism.

@sobolevn, what problems do you see with the use of inlined TypedDicts and type aliases? I didn't encounter any static type checking issues when I implemented this support in pyright. Perhaps there are issues with runtime type checkers?

[sobolevn]

Things that I think are quite controversial:

1. Inheritance, it is not possible. You cannot extend existing inline typeddict aliases and you cannot extend existing typeddicts in inline typeddict alias definition. But, on the other hand we can always infer these typeddicts as @final
2. Recursive inline typeddict aliases can be a problem (this really depends on a type-checker's implementation): T = TypedDict[{'a': 'T'}]
3. pickle support is broken. Right now you need to explicitly set names to all things in a module that you want to be pickleable (I even think about proposing module-level __set_name__ hook to fix this). Inline typeddict aliases do not have a name in this form. It can be T = TypedDict('T')[{'a': int}] or similar, but this will further complicate the TypedDict's runtime implementation

But, I am not invested in this opinion at all, if all things can be solved - I will be very happy.

[PIG208]

Recursive inline typeddict aliases can be a problem

IIRC, this is not supported by the class syntax either, which led to a discussion about the inability to properly type a json object. Maybe a forward reference can help here? It might be out of scope for this proposal though.

I remember from the typing meetup, it was proposed by someone to make the base type a type parameter alongside the inline TypedDict (something like dict[{...}, BaseTypedDict]). I'm interested in learning if we have put forward a design like that for discussion anywhere else.

[Zomatree]

I was messing around with the experimental support of this in Pyright, would using Typevars in the dict be in the scope of this PEP? Currently Pyright errors on this but I don't know whether that's because its still experimental.

something akin to this example:

K = TypeVar("K", bound=LiteralString)
V = TypeVar("V")

def create(key: K, value: V) -> dict[{K: V}]:
    return {key: value}
    
my_dict = create("a", 1)
reveal_type(my_dict["a"])  # Literal[1]

[erictraut]

No, this syntax would require concrete keys and values.

There is an active discussion for a type system extension that would allow manipulation of keys and values.

[Kroppeb]

Should this also allow for something like this?

def extra[D:TypedDict](data: D) -> TypedDict[{**D, 'score':float}]:
    return {
        **data,
        'score': 0.0,
    }
        

def peel[D:TypedDict](data:TypedDict[{**D, 'score': float}]) -> D:
    copy = data.copy()
    del copy['score']
    return copy


def parse_score[D:TypedDict](data: TypedDict[{**D, 'score': str}]) -> TypedDict[{**D, 'score':float}]
    copy = data.copy()
    copy['score'] = float(copy['score'])
    return copy