From 8142da0385c1a806aa5b1f74eb5779e2a995efd6 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 27 Nov 2020 15:56:03 -0800 Subject: [PATCH 1/9] Document PEP 585, 563, 604 and more. Fixes #8629, fixes #8523 This creates a new page to document issues arising from discrepancies between the runtime and annotations. I felt this was better, rather than force-fitting things into existing pages and "common issues", for instance, it prevents us from having to explain PEP 563 in several different places. I do still list the runtime errors you'd get in the "common issues" page to preserve SEO :-) "String literal types", "Class name forward references", and "Import cycles" are basically the same as where they were copied over from. This also factors out the documentation of PEP 604 that I promised when merging that PR (it seemed pretty verbose, particularly for the "kinds of types" page). It's also a good place to document PEP 613, when we get around to supporting that. --- docs/source/builtin_types.rst | 5 + docs/source/common_issues.rst | 121 ++---------- docs/source/index.rst | 1 + docs/source/kinds_of_types.rst | 161 +--------------- docs/source/runtime_troubles.rst | 310 +++++++++++++++++++++++++++++++ 5 files changed, 339 insertions(+), 259 deletions(-) create mode 100644 docs/source/runtime_troubles.rst diff --git a/docs/source/builtin_types.rst b/docs/source/builtin_types.rst index 7ad5fbcfa2cc..5aa84922307d 100644 --- a/docs/source/builtin_types.rst +++ b/docs/source/builtin_types.rst @@ -38,3 +38,8 @@ though they are similar to abstract base classes defined in :py:mod:`collections.abc` (formerly ``collections``), they are not identical. In particular, prior to Python 3.9, the built-in collection type objects do not support indexing. + +In Python 3.9 and later, built-in collection type objects support indexing. This +means that you can use built-in classes or those from :py:mod:`collections.abc` +instead of importing from :py:mod:`typing`. See :ref:`generic-builtins` for more +details. diff --git a/docs/source/common_issues.rst b/docs/source/common_issues.rst index 3867e168bd6a..79c94ac2baff 100644 --- a/docs/source/common_issues.rst +++ b/docs/source/common_issues.rst @@ -210,6 +210,21 @@ checking would require a large number of ``assert foo is not None`` checks to be inserted, and you want to minimize the number of code changes required to get a clean mypy run. +Issues with code at runtime +--------------------------- + +Idiomatic use of type annotations can sometimes run up against what a given +version of Python considers legal code. These can result in some of the +following errors when trying to run your code: + +* ``ImportError`` from circular imports +* ``NameError: name 'X' is not defined`` from forward references +* ``TypeError: 'type' object is not subscriptable`` from types that are not generic at runtime +* ``ImportError`` or ``ModuleNotFoundError`` from use of stub definitions not available at runtime +* ``TypeError: unsupported operand type(s) for |: 'type' and 'type'`` from use of new syntax + +For dealing with these, see :ref:`runtime_troubles`. + Mypy runs are slow ------------------ @@ -499,112 +514,6 @@ to see the types of all local variables at once. Example: run your code. Both are always available and you don't need to import them. - -.. _import-cycles: - -Import cycles -------------- - -An import cycle occurs where module A imports module B and module B -imports module A (perhaps indirectly, e.g. ``A -> B -> C -> A``). -Sometimes in order to add type annotations you have to add extra -imports to a module and those imports cause cycles that didn't exist -before. If those cycles become a problem when running your program, -there's a trick: if the import is only needed for type annotations in -forward references (string literals) or comments, you can write the -imports inside ``if TYPE_CHECKING:`` so that they are not executed at runtime. -Example: - -File ``foo.py``: - -.. code-block:: python - - from typing import List, TYPE_CHECKING - - if TYPE_CHECKING: - import bar - - def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]': - return [arg] - -File ``bar.py``: - -.. code-block:: python - - from typing import List - from foo import listify - - class BarClass: - def listifyme(self) -> 'List[BarClass]': - return listify(self) - -.. note:: - - The :py:data:`~typing.TYPE_CHECKING` constant defined by the :py:mod:`typing` module - is ``False`` at runtime but ``True`` while type checking. - -Python 3.5.1 doesn't have :py:data:`~typing.TYPE_CHECKING`. An alternative is -to define a constant named ``MYPY`` that has the value ``False`` -at runtime. Mypy considers it to be ``True`` when type checking. -Here's the above example modified to use ``MYPY``: - -.. code-block:: python - - from typing import List - - MYPY = False - if MYPY: - import bar - - def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]': - return [arg] - -.. _not-generic-runtime: - -Using classes that are generic in stubs but not at runtime ----------------------------------------------------------- - -Some classes are declared as generic in stubs, but not at runtime. Examples -in the standard library include :py:class:`os.PathLike` and :py:class:`queue.Queue`. -Subscripting such a class will result in a runtime error: - -.. code-block:: python - - from queue import Queue - - class Tasks(Queue[str]): # TypeError: 'type' object is not subscriptable - ... - - results: Queue[int] = Queue() # TypeError: 'type' object is not subscriptable - -To avoid these errors while still having precise types you can either use -string literal types or :py:data:`~typing.TYPE_CHECKING`: - -.. code-block:: python - - from queue import Queue - from typing import TYPE_CHECKING - - if TYPE_CHECKING: - BaseQueue = Queue[str] # this is only processed by mypy - else: - BaseQueue = Queue # this is not seen by mypy but will be executed at runtime. - - class Tasks(BaseQueue): # OK - ... - - results: 'Queue[int]' = Queue() # OK - -If you are running Python 3.7+ you can use ``from __future__ import annotations`` -as a (nicer) alternative to string quotes, read more in :pep:`563`. For example: - -.. code-block:: python - - from __future__ import annotations - from queue import Queue - - results: Queue[int] = Queue() # This works at runtime - .. _silencing-linters: Silencing linters diff --git a/docs/source/index.rst b/docs/source/index.rst index 4a8cb59cf1a9..3295f9ca38cc 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -35,6 +35,7 @@ Mypy is a static type checker for Python 3 and Python 2.7. type_inference_and_annotations kinds_of_types class_basics + runtime_troubles protocols python2 dynamic_typing diff --git a/docs/source/kinds_of_types.rst b/docs/source/kinds_of_types.rst index facc5da5a64c..ccb843325775 100644 --- a/docs/source/kinds_of_types.rst +++ b/docs/source/kinds_of_types.rst @@ -243,93 +243,24 @@ more specific type: .. _alternative_union_syntax: -Alternative union syntax ------------------------- +X | Y syntax for Unions +----------------------- -`PEP 604 `_ introduced an alternative way -for writing union types. Starting with **Python 3.10** it is possible to write -``Union[int, str]`` as ``int | str``. Any of the following options is possible +:pep:`604` introduced an alternative way for spelling union types. In Python +3.10 and later, it is possible to write ``Union[int, str]`` as ``int | str``. It +is possible to use this syntax in versions of Python where it isn't supported by +the runtime with some limitations, see :ref:`runtime_troubles`. .. code-block:: python from typing import List - # Use as Union t1: int | str # equivalent to Union[int, str] - # Use as Optional t2: int | None # equivalent to Optional[int] - # Use in generics - t3: List[int | str] # equivalent to List[Union[int, str]] - - # Use in type aliases - T4 = int | None - x: T4 - - # Quoted variable annotations - t5: "int | str" - - # Quoted function annotations - def f(t6: "int | str") -> None: ... - - # Type comments - t6 = 42 # type: int | str - -It is possible to use most of these even for earlier versions. However there are some -limitations to be aware of. - -.. _alternative_union_syntax_stub_files: - -Stub files -"""""""""" - -All options are supported, regardless of the Python version the project uses. - -.. _alternative_union_syntax_37: - -Python 3.7 - 3.9 -"""""""""""""""" - -It is necessary to add ``from __future__ import annotations`` to delay the evaluation -of type annotations. Not using it would result in a ``TypeError``. -This does not apply for **type comments**, **quoted function** and **quoted variable** annotations, -as those also work for earlier versions, see :ref:`below `. - -.. warning:: - - Type aliases are **NOT** supported! Those result in a ``TypeError`` regardless - if the evaluation of type annotations is delayed. - - Dynamic evaluation of annotations is **NOT** possible (e.g. ``typing.get_type_hints`` and ``eval``). - See `note PEP 604 `_. - Use ``typing.Union`` or **Python 3.10** instead if you need those! - -.. code-block:: python - - from __future__ import annotations - - t1: int | None - - # Type aliases - T2 = int | None # TypeError! - -.. _alternative_union_syntax_older_version: - -Older versions -"""""""""""""" - -+------------------------------------------+-----------+-----------+-----------+ -| Python Version | 3.6 | 3.0 - 3.5 | 2.7 | -+==========================================+===========+===========+===========+ -| Type comments | yes | yes | yes | -+------------------------------------------+-----------+-----------+-----------+ -| Quoted function annotations | yes | yes | | -+------------------------------------------+-----------+-----------+-----------+ -| Quoted variable annotations | yes | | | -+------------------------------------------+-----------+-----------+-----------+ -| Everything else | | | | -+------------------------------------------+-----------+-----------+-----------+ + # Usable in type comments + t3 = 42 # type: int | str .. _strict_optional: @@ -565,82 +496,6 @@ valid for any type, but it's much more useful for a programmer who is reading the code. This also makes it easier to migrate to strict ``None`` checking in the future. -Class name forward references -***************************** - -Python does not allow references to a class object before the class is -defined. Thus this code does not work as expected: - -.. code-block:: python - - def f(x: A) -> None: # Error: Name A not defined - ... - - class A: - ... - -In cases like these you can enter the type as a string literal — this -is a *forward reference*: - -.. code-block:: python - - def f(x: 'A') -> None: # OK - ... - - class A: - ... - -Starting from Python 3.7 (:pep:`563`), you can add the special import ``from __future__ import annotations``, -which makes the use of string literals in annotations unnecessary: - -.. code-block:: python - - from __future__ import annotations - - def f(x: A) -> None: # OK - ... - - class A: - ... - -.. note:: - - Even with the ``__future__`` import, there are some scenarios that could still - require string literals, typically involving use of forward references or generics in: - - * :ref:`type aliases `; - * :ref:`casts `; - * type definitions (see :py:class:`~typing.TypeVar`, :py:func:`~typing.NewType`, :py:class:`~typing.NamedTuple`); - * base classes. - - .. code-block:: python - - # base class example - class A(Tuple['B', 'C']): ... # OK - class B: ... - class C: ... - -Of course, instead of using a string literal type or special import, you could move the -function definition after the class definition. This is not always -desirable or even possible, though. - -Any type can be entered as a string literal, and you can combine -string-literal types with non-string-literal types freely: - -.. code-block:: python - - def f(a: List['A']) -> None: ... # OK - def g(n: 'int') -> None: ... # OK, though not useful - - class A: pass - -String literal types are never needed in ``# type:`` comments and :ref:`stub files `. - -String literal types must be defined (or imported) later *in the same -module*. They cannot be used to leave cross-module references -unresolved. (For dealing with import cycles, see -:ref:`import-cycles`.) - .. _type-aliases: Type aliases diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst new file mode 100644 index 000000000000..6df28c9924a0 --- /dev/null +++ b/docs/source/runtime_troubles.rst @@ -0,0 +1,310 @@ +.. _runtime_troubles: + +Annotation issues at runtime +============================ + +Idiomatic use of type annotations can sometimes run up against what a given +version of Python considers legal code. This section describes these scenarios +and explains how to get your code running again. Generally speaking, we have +three tools at our disposal: + +* For Python 3.7 through 3.9, use of ``from __future__ import annotations`` + (:pep:`563`), made the default in Python 3.10 and later +* Use of string literal types or type comments +* Use of ``typing.TYPE_CHECKING`` + +We provide a description of these before moving onto discussion of specific +problems you may encounter. + +.. _string-literal-types: + +String literal types +-------------------- + +Type comments can't cause runtime errors because comments are not evaluated by +Python. Using string literal types sidesteps the problem of annotations that +would cause runtime errors in a similar way. + +Any type can be entered as a string literal, and you can combine +string-literal types with non-string-literal types freely: + +.. code-block:: python + + def f(a: List['A']) -> None: ... # OK + def g(n: 'int') -> None: ... # OK, though not useful + + class A: pass + +String literal types are never needed in ``# type:`` comments and :ref:`stub files `. + +String literal types must be defined (or imported) later *in the same module*. +They cannot be used to leave cross-module references unresolved. (For dealing +with import cycles, see :ref:`import-cycles`.) + +.. _future-annotations: + +Annotations import from future (PEP 563) +---------------------------------------- + +Many of the issues described here are caused by Python trying to evaluate +annotations. From Python 3.10 on, Python will no longer attempt to evaluate +function and variable annotations. This behaviour is made available in Python +3.7 and later through the use of ``from __future__ import annotations``. + +This can be thought of as automatic string literal-ification of all function and +variable annotations. Note that function and variable annotations are still +required to be valid Python syntax. For more details, see :pep:`563`. + +.. note:: + + Even with the ``__future__`` import, there are some scenarios that could + still require string literals or result in errors, typically involving use + of forward references or generics in: + + * :ref:`type aliases `; + * :ref:`casts `; + * type definitions (see :py:class:`~typing.TypeVar`, :py:func:`~typing.NewType`, :py:class:`~typing.NamedTuple`); + * base classes. + + .. code-block:: python + + # base class example + class A(Tuple['B', 'C']): ... # OK + class B: ... + class C: ... + +.. note:: + + Some libraries may have use cases for dynamic evaluation of annotations, for + instance, through use of ``typing.get_type_hints`` or ``eval``. If your + annotation would raise an error when evaluated (say by using :pep:`604` + syntax with Python 3.9), you may need to be careful when using such + libraries. + +.. _typing-type-checking: + +typing.TYPE_CHECKING +-------------------- + +The :py:mod:`typing` module defines a :py:data:`~typing.TYPE_CHECKING` constant +that is ``False`` at runtime but treated as ``True`` while type checking. + +Since code inside ``if TYPE_CHECKING:`` is not executed at runtime, it provides +a convenient way to tell mypy something without the code being evaluated at +runtime. This is most useful for resolving :ref:`import cycles `. + +.. note:: + + Python 3.5.1 and below don't have :py:data:`~typing.TYPE_CHECKING`. An + alternative is to define a constant named ``MYPY`` that has the value + ``False`` at runtime. Mypy considers it to be ``True`` when type checking. + +Class name forward references +----------------------------- + +Python does not allow references to a class object before the class is +defined (aka forward reference). Thus this code does not work as expected: + +.. code-block:: python + + def f(x: A) -> None: ... # NameError: name 'A' is not defined + class A: ... + +Starting from Python 3.7, you can add the special import ``from __future__ import annotations``, +which makes the use of string literals in annotations unnecessary: + +.. code-block:: python + + from __future__ import annotations + + def f(x: A) -> None: ... # OK + class A: ... + +For Python 3.6 and below, you can enter the type as a string literal or type comment: + +.. code-block:: python + + def f(x: 'A') -> None: ... # OK + + # Also OK + def g(x): # type: (A) -> None + ... + + class A: ... + +Of course, instead of using a string literal type or special import, you could move the +function definition after the class definition. This is not always +desirable or even possible, though. + +.. _import-cycles: + +Import cycles +------------- + +An import cycle occurs where module A imports module B and module B +imports module A (perhaps indirectly, e.g. ``A -> B -> C -> A``). +Sometimes in order to add type annotations you have to add extra +imports to a module and those imports cause cycles that didn't exist +before. This can lead to errors at runtime like: + +.. code-block:: + + ImportError: cannot import name 'b' from partially initialized module 'A' (most likely due to a circular import) + +If those cycles do become a problem when running your program, +there's a trick: if the import is only needed for type annotations in +forward references (string literals) or comments, you can write the +imports inside ``if TYPE_CHECKING:`` so that they are not executed at runtime. +Example: + +File ``foo.py``: + +.. code-block:: python + + from typing import List, TYPE_CHECKING + + if TYPE_CHECKING: + import bar + + def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]': + return [arg] + +File ``bar.py``: + +.. code-block:: python + + from typing import List + from foo import listify + + class BarClass: + def listifyme(self) -> 'List[BarClass]': + return listify(self) + +.. _not-generic-runtime: + +Using classes that are generic in stubs but not at runtime +---------------------------------------------------------- + +Some classes are declared as generic in stubs, but not at runtime. + +In Python 3.8 and lower, there are several examples within the standard library, +for instance, :py:class:`os.PathLike` and :py:class:`queue.Queue`. Subscripting +such a class will result in a runtime error: + +.. code-block:: python + + from queue import Queue + + class Tasks(Queue[str]): # TypeError: 'type' object is not subscriptable + ... + + results: Queue[int] = Queue() # TypeError: 'type' object is not subscriptable + +To avoid errors from use of these generics in annotations, just use the +:ref:`future annotations import` (or string literals or type +comments for Python 3.6 and below). + +To avoid errors when inheriting from these classes, things are a little more +complicated and you need to use :ref:`typing.TYPE_CHECKING +`: + +.. code-block:: python + + from typing import TYPE_CHECKING + from queue import Queue + + if TYPE_CHECKING: + BaseQueue = Queue[str] # this is only processed by mypy + else: + BaseQueue = Queue # this is not seen by mypy but will be executed at runtime + + class Tasks(BaseQueue): # OK + ... + + task_queue: Tasks + reveal_type(task_queue.get()) # Reveals str + +If your subclass is also generic, you can using something like the following: + +.. code-block:: python + + from typing import TYPE_CHECKING, TypeVar, Generic + from queue import Queue + + _T = TypeVar("_T") + if TYPE_CHECKING: + class _MyQueueBase(Queue[_T]): pass + else: + class _MyQueueBase(Generic[_T], Queue): pass + + class MyQueue(_MyQueueBase[_T]): pass + + task_queue: MyQueue[str] + reveal_type(task_queue.get()) # Reveals str + + +Using types defined in stubs but not at runtime +----------------------------------------------- + +Sometimes stubs that you're using may define types you wish to re-use that do +not exist at runtime. Importing these types naively will cause your code to fail +at runtime with ``ImportError`` or ``ModuleNotFoundError``. Similar to previous +sections, these can be dealt with by using :ref:`typing.TYPE_CHECKING +`: + +.. code-block:: python + + from typing import TYPE_CHECKING + if TYPE_CHECKING: + from _typeshed import SupportsLessThan + +.. _generic-builtins: + +Using generic builtins +---------------------- + +Starting with Python 3.9 (:pep:`585`), the type objects of many collections in +the standard library support subscription at runtime. This means that you no +longer have to import the equivalents from :py:mod:`typing`; you can simply use +the built-in collections or those from :py:mod:`collections.abc` + +.. code-block:: python + + from collections.abc import Sequence + x: list[str] + y: dict[int, str] + z: Sequence[str] = x + +If you use ``from __future__ import annotations``, mypy will understand this +syntax in Python 3.7 and later. However, since this will not be supported by the +Python interpreter at runtime, make sure you're aware of the caveats mentioned +in the notes at :ref:`future annotations import`. + +Using new syntax +---------------- + +Starting with Python 3.10 (:pep:`604`), you can spell union types as ``x: int | +str``, instead of ``x: typing.Union[int, str]``. + +If you use ``from __future__ import annotations``, mypy will understand this +syntax in Python 3.7 and later. However, since this will not be supported by the +Python interpreter at runtime (if evaluated, ``int | str`` will raise +``TypeError: unsupported operand type(s) for |: 'type' and 'type'``), make sure +you're aware of the caveats mentioned in the notes at :ref:`future annotations +import`. You can also use the new syntax in string literal +types or type comments. + +Using new additions to the typing module +---------------------------------------- + +You may find yourself wanting to use features added to the :py:mod:`typing` +module in earlier versions of Python than the addition, for example, using any +of ``Literal``, ``Protocol``, ``TypedDict`` with Python 3.6. + +The easiest way to do this is to install and use the ``typing_extensions`` +package for the relevant imports, for example: + +.. code-block:: python + + from typing_extensions import Literal + x: Literal["open", "close"] From 5bc7167830ae4e4ac1af8f3f8e77e9ffa721cb28 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 27 Nov 2020 16:26:15 -0800 Subject: [PATCH 2/9] more edits --- docs/source/runtime_troubles.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index 6df28c9924a0..e3536ac49de3 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -22,8 +22,8 @@ String literal types -------------------- Type comments can't cause runtime errors because comments are not evaluated by -Python. Using string literal types sidesteps the problem of annotations that -would cause runtime errors in a similar way. +Python. In a similar way, using string literal types sidesteps the problem of +annotations that would cause runtime errors. Any type can be entered as a string literal, and you can combine string-literal types with non-string-literal types freely: @@ -151,11 +151,11 @@ before. This can lead to errors at runtime like: ImportError: cannot import name 'b' from partially initialized module 'A' (most likely due to a circular import) -If those cycles do become a problem when running your program, -there's a trick: if the import is only needed for type annotations in -forward references (string literals) or comments, you can write the -imports inside ``if TYPE_CHECKING:`` so that they are not executed at runtime. -Example: +If those cycles do become a problem when running your program, there's a trick: +if the import is only needed for type annotations and you're using a) the +:ref:`future annotations import`, or b) string literals or type +comments for the relevant annotations, you can write the imports inside ``if +TYPE_CHECKING:`` so that they are not executed at runtime. Example: File ``foo.py``: @@ -224,7 +224,7 @@ complicated and you need to use :ref:`typing.TYPE_CHECKING task_queue: Tasks reveal_type(task_queue.get()) # Reveals str -If your subclass is also generic, you can using something like the following: +If your subclass is also generic, you can use the following: .. code-block:: python From 8e66ca38a8189629abba517283c44f91d93e2a18 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 27 Nov 2020 16:46:41 -0800 Subject: [PATCH 3/9] more edits --- docs/source/kinds_of_types.rst | 4 ++-- docs/source/runtime_troubles.rst | 27 ++++++++++++++------------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/source/kinds_of_types.rst b/docs/source/kinds_of_types.rst index ccb843325775..9000b6aa238f 100644 --- a/docs/source/kinds_of_types.rst +++ b/docs/source/kinds_of_types.rst @@ -247,8 +247,8 @@ X | Y syntax for Unions ----------------------- :pep:`604` introduced an alternative way for spelling union types. In Python -3.10 and later, it is possible to write ``Union[int, str]`` as ``int | str``. It -is possible to use this syntax in versions of Python where it isn't supported by +3.10 and later, you can write ``Union[int, str]`` as ``int | str``. It is +possible to use this syntax in versions of Python where it isn't supported by the runtime with some limitations, see :ref:`runtime_troubles`. .. code-block:: python diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index e3536ac49de3..2f5e7ea81a56 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -43,8 +43,8 @@ with import cycles, see :ref:`import-cycles`.) .. _future-annotations: -Annotations import from future (PEP 563) ----------------------------------------- +Future annotations import (PEP 563) +----------------------------------- Many of the issues described here are caused by Python trying to evaluate annotations. From Python 3.10 on, Python will no longer attempt to evaluate @@ -275,24 +275,25 @@ the built-in collections or those from :py:mod:`collections.abc` y: dict[int, str] z: Sequence[str] = x +There is limited support for using this syntax in Python 3.7 and later as well. If you use ``from __future__ import annotations``, mypy will understand this -syntax in Python 3.7 and later. However, since this will not be supported by the -Python interpreter at runtime, make sure you're aware of the caveats mentioned -in the notes at :ref:`future annotations import`. +syntax in annotations. However, since this will not be supported by the Python +interpreter at runtime, make sure you're aware of the caveats mentioned in the +notes at :ref:`future annotations import`. -Using new syntax ----------------- +Using X | Y syntax for Unions +----------------------------- Starting with Python 3.10 (:pep:`604`), you can spell union types as ``x: int | str``, instead of ``x: typing.Union[int, str]``. +There is limited support for using this syntax in Python 3.7 and later as well. If you use ``from __future__ import annotations``, mypy will understand this -syntax in Python 3.7 and later. However, since this will not be supported by the -Python interpreter at runtime (if evaluated, ``int | str`` will raise -``TypeError: unsupported operand type(s) for |: 'type' and 'type'``), make sure -you're aware of the caveats mentioned in the notes at :ref:`future annotations -import`. You can also use the new syntax in string literal -types or type comments. +syntax in annotations, string literal types, type comments and stub files. +However, since this will not be supported by the Python interpreter at runtime +(if evaluated, ``int | str`` will raise ``TypeError: unsupported operand type(s) +for |: 'type' and 'type'``), make sure you're aware of the caveats mentioned in +the notes at :ref:`future annotations import`. Using new additions to the typing module ---------------------------------------- From 218a46e41410612f4e4ef18ecc82b4a1a838616b Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 11 Dec 2020 12:20:23 -0800 Subject: [PATCH 4/9] make string literal types use with future annotations more explicit --- docs/source/runtime_troubles.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index 2f5e7ea81a56..a0d8cf22c4c9 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -69,7 +69,8 @@ required to be valid Python syntax. For more details, see :pep:`563`. .. code-block:: python # base class example - class A(Tuple['B', 'C']): ... # OK + from __future__ import annotations + class A(Tuple['B', 'C']): ... # String literal types needed here class B: ... class C: ... From 39e5ff9273583b7c9b5121a109dba159b9f95840 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 11 Dec 2020 12:21:18 -0800 Subject: [PATCH 5/9] cross reference generic classes --- docs/source/runtime_troubles.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index a0d8cf22c4c9..0800ca344cc5 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -186,7 +186,8 @@ File ``bar.py``: Using classes that are generic in stubs but not at runtime ---------------------------------------------------------- -Some classes are declared as generic in stubs, but not at runtime. +Some classes are declared as :ref:`generic` in stubs, but not +at runtime. In Python 3.8 and lower, there are several examples within the standard library, for instance, :py:class:`os.PathLike` and :py:class:`queue.Queue`. Subscripting From c0c14b45e105a5169896c4b32cbafeaea074b500 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 11 Dec 2020 12:22:55 -0800 Subject: [PATCH 6/9] fix nits --- docs/source/runtime_troubles.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index 0800ca344cc5..0f61ad1e2b33 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -189,7 +189,7 @@ Using classes that are generic in stubs but not at runtime Some classes are declared as :ref:`generic` in stubs, but not at runtime. -In Python 3.8 and lower, there are several examples within the standard library, +In Python 3.8 and earlier, there are several examples within the standard library, for instance, :py:class:`os.PathLike` and :py:class:`queue.Queue`. Subscripting such a class will result in a runtime error: @@ -268,7 +268,7 @@ Using generic builtins Starting with Python 3.9 (:pep:`585`), the type objects of many collections in the standard library support subscription at runtime. This means that you no longer have to import the equivalents from :py:mod:`typing`; you can simply use -the built-in collections or those from :py:mod:`collections.abc` +the built-in collections or those from :py:mod:`collections.abc`: .. code-block:: python @@ -305,7 +305,7 @@ module in earlier versions of Python than the addition, for example, using any of ``Literal``, ``Protocol``, ``TypedDict`` with Python 3.6. The easiest way to do this is to install and use the ``typing_extensions`` -package for the relevant imports, for example: +package from PyPI for the relevant imports, for example: .. code-block:: python From fb16a12fc1f24d03ebaf0fd3566c03361f915ecb Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 11 Dec 2020 12:28:47 -0800 Subject: [PATCH 7/9] reference previous discussion in forward references --- docs/source/runtime_troubles.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index 0f61ad1e2b33..42da1acf50d4 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -111,8 +111,8 @@ defined (aka forward reference). Thus this code does not work as expected: def f(x: A) -> None: ... # NameError: name 'A' is not defined class A: ... -Starting from Python 3.7, you can add the special import ``from __future__ import annotations``, -which makes the use of string literals in annotations unnecessary: +Starting from Python 3.7, you can add ``from __future__ import annotations`` to +resolve this, as discussed earlier: .. code-block:: python @@ -133,9 +133,9 @@ For Python 3.6 and below, you can enter the type as a string literal or type com class A: ... -Of course, instead of using a string literal type or special import, you could move the -function definition after the class definition. This is not always -desirable or even possible, though. +Of course, instead of using future annotations import or string literal types, +you could move the function definition after the class definition. This is not +always desirable or even possible, though. .. _import-cycles: From e5d098d1511d01f348e4477ff3f7624a3dea97ec Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Fri, 11 Dec 2020 12:35:14 -0800 Subject: [PATCH 8/9] mention py39 in runtime generics section --- docs/source/runtime_troubles.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index 42da1acf50d4..f90ecd0cff8a 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -244,6 +244,9 @@ If your subclass is also generic, you can use the following: task_queue: MyQueue[str] reveal_type(task_queue.get()) # Reveals str +In Python 3.9, we can just inherit directly from ``Queue[str]`` or ``Queue[T]`` +since its :py:class:`queue.Queue` implements :py:meth:`__class_getitem__`, so +the class object can be subscripted at runtime without issue. Using types defined in stubs but not at runtime ----------------------------------------------- From b3d6644422a74471fb4f9f2d9f342c927ca30603 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Tue, 29 Dec 2020 14:46:43 -0800 Subject: [PATCH 9/9] Mention conditional import of typing_extensions Resolves #9856 --- docs/source/runtime_troubles.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst index f90ecd0cff8a..809c7dac1bb8 100644 --- a/docs/source/runtime_troubles.rst +++ b/docs/source/runtime_troubles.rst @@ -314,3 +314,19 @@ package from PyPI for the relevant imports, for example: from typing_extensions import Literal x: Literal["open", "close"] + +If you don't want to rely on ``typing_extensions`` being installed on newer +Pythons, you could alternatively use: + +.. code-block:: python + + import sys + if sys.version_info >= (3, 8): + from typing import Literal + else: + from typing_extensions import Literal + + x: Literal["open", "close"] + +This plays nicely well with following :pep:`508` dependency specification: +``typing_extensions; python_version<"3.8"``