diff --git a/docs/source/builtin_types.rst b/docs/source/builtin_types.rst
index 7ad5fbcfa2cc..5aa84922307d 100644
--- a/docs/source/builtin_types.rst
+++ b/docs/source/builtin_types.rst
@@ -38,3 +38,8 @@ though they are similar to abstract base classes defined in
:py:mod:`collections.abc` (formerly ``collections``), they are not identical. In
particular, prior to Python 3.9, the built-in collection type objects do not
support indexing.
+
+In Python 3.9 and later, built-in collection type objects support indexing. This
+means that you can use built-in classes or those from :py:mod:`collections.abc`
+instead of importing from :py:mod:`typing`. See :ref:`generic-builtins` for more
+details.
diff --git a/docs/source/common_issues.rst b/docs/source/common_issues.rst
index 3867e168bd6a..79c94ac2baff 100644
--- a/docs/source/common_issues.rst
+++ b/docs/source/common_issues.rst
@@ -210,6 +210,21 @@ checking would require a large number of ``assert foo is not None``
checks to be inserted, and you want to minimize the number
of code changes required to get a clean mypy run.
+Issues with code at runtime
+---------------------------
+
+Idiomatic use of type annotations can sometimes run up against what a given
+version of Python considers legal code. These can result in some of the
+following errors when trying to run your code:
+
+* ``ImportError`` from circular imports
+* ``NameError: name 'X' is not defined`` from forward references
+* ``TypeError: 'type' object is not subscriptable`` from types that are not generic at runtime
+* ``ImportError`` or ``ModuleNotFoundError`` from use of stub definitions not available at runtime
+* ``TypeError: unsupported operand type(s) for |: 'type' and 'type'`` from use of new syntax
+
+For dealing with these, see :ref:`runtime_troubles`.
+
Mypy runs are slow
------------------
@@ -499,112 +514,6 @@ to see the types of all local variables at once. Example:
run your code. Both are always available and you don't need to import
them.
-
-.. _import-cycles:
-
-Import cycles
--------------
-
-An import cycle occurs where module A imports module B and module B
-imports module A (perhaps indirectly, e.g. ``A -> B -> C -> A``).
-Sometimes in order to add type annotations you have to add extra
-imports to a module and those imports cause cycles that didn't exist
-before. If those cycles become a problem when running your program,
-there's a trick: if the import is only needed for type annotations in
-forward references (string literals) or comments, you can write the
-imports inside ``if TYPE_CHECKING:`` so that they are not executed at runtime.
-Example:
-
-File ``foo.py``:
-
-.. code-block:: python
-
- from typing import List, TYPE_CHECKING
-
- if TYPE_CHECKING:
- import bar
-
- def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]':
- return [arg]
-
-File ``bar.py``:
-
-.. code-block:: python
-
- from typing import List
- from foo import listify
-
- class BarClass:
- def listifyme(self) -> 'List[BarClass]':
- return listify(self)
-
-.. note::
-
- The :py:data:`~typing.TYPE_CHECKING` constant defined by the :py:mod:`typing` module
- is ``False`` at runtime but ``True`` while type checking.
-
-Python 3.5.1 doesn't have :py:data:`~typing.TYPE_CHECKING`. An alternative is
-to define a constant named ``MYPY`` that has the value ``False``
-at runtime. Mypy considers it to be ``True`` when type checking.
-Here's the above example modified to use ``MYPY``:
-
-.. code-block:: python
-
- from typing import List
-
- MYPY = False
- if MYPY:
- import bar
-
- def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]':
- return [arg]
-
-.. _not-generic-runtime:
-
-Using classes that are generic in stubs but not at runtime
-----------------------------------------------------------
-
-Some classes are declared as generic in stubs, but not at runtime. Examples
-in the standard library include :py:class:`os.PathLike` and :py:class:`queue.Queue`.
-Subscripting such a class will result in a runtime error:
-
-.. code-block:: python
-
- from queue import Queue
-
- class Tasks(Queue[str]): # TypeError: 'type' object is not subscriptable
- ...
-
- results: Queue[int] = Queue() # TypeError: 'type' object is not subscriptable
-
-To avoid these errors while still having precise types you can either use
-string literal types or :py:data:`~typing.TYPE_CHECKING`:
-
-.. code-block:: python
-
- from queue import Queue
- from typing import TYPE_CHECKING
-
- if TYPE_CHECKING:
- BaseQueue = Queue[str] # this is only processed by mypy
- else:
- BaseQueue = Queue # this is not seen by mypy but will be executed at runtime.
-
- class Tasks(BaseQueue): # OK
- ...
-
- results: 'Queue[int]' = Queue() # OK
-
-If you are running Python 3.7+ you can use ``from __future__ import annotations``
-as a (nicer) alternative to string quotes, read more in :pep:`563`. For example:
-
-.. code-block:: python
-
- from __future__ import annotations
- from queue import Queue
-
- results: Queue[int] = Queue() # This works at runtime
-
.. _silencing-linters:
Silencing linters
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 4a8cb59cf1a9..3295f9ca38cc 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -35,6 +35,7 @@ Mypy is a static type checker for Python 3 and Python 2.7.
type_inference_and_annotations
kinds_of_types
class_basics
+ runtime_troubles
protocols
python2
dynamic_typing
diff --git a/docs/source/kinds_of_types.rst b/docs/source/kinds_of_types.rst
index facc5da5a64c..9000b6aa238f 100644
--- a/docs/source/kinds_of_types.rst
+++ b/docs/source/kinds_of_types.rst
@@ -243,93 +243,24 @@ more specific type:
.. _alternative_union_syntax:
-Alternative union syntax
-------------------------
+X | Y syntax for Unions
+-----------------------
-`PEP 604 `_ introduced an alternative way
-for writing union types. Starting with **Python 3.10** it is possible to write
-``Union[int, str]`` as ``int | str``. Any of the following options is possible
+:pep:`604` introduced an alternative way for spelling union types. In Python
+3.10 and later, you can write ``Union[int, str]`` as ``int | str``. It is
+possible to use this syntax in versions of Python where it isn't supported by
+the runtime with some limitations, see :ref:`runtime_troubles`.
.. code-block:: python
from typing import List
- # Use as Union
t1: int | str # equivalent to Union[int, str]
- # Use as Optional
t2: int | None # equivalent to Optional[int]
- # Use in generics
- t3: List[int | str] # equivalent to List[Union[int, str]]
-
- # Use in type aliases
- T4 = int | None
- x: T4
-
- # Quoted variable annotations
- t5: "int | str"
-
- # Quoted function annotations
- def f(t6: "int | str") -> None: ...
-
- # Type comments
- t6 = 42 # type: int | str
-
-It is possible to use most of these even for earlier versions. However there are some
-limitations to be aware of.
-
-.. _alternative_union_syntax_stub_files:
-
-Stub files
-""""""""""
-
-All options are supported, regardless of the Python version the project uses.
-
-.. _alternative_union_syntax_37:
-
-Python 3.7 - 3.9
-""""""""""""""""
-
-It is necessary to add ``from __future__ import annotations`` to delay the evaluation
-of type annotations. Not using it would result in a ``TypeError``.
-This does not apply for **type comments**, **quoted function** and **quoted variable** annotations,
-as those also work for earlier versions, see :ref:`below `.
-
-.. warning::
-
- Type aliases are **NOT** supported! Those result in a ``TypeError`` regardless
- if the evaluation of type annotations is delayed.
-
- Dynamic evaluation of annotations is **NOT** possible (e.g. ``typing.get_type_hints`` and ``eval``).
- See `note PEP 604 `_.
- Use ``typing.Union`` or **Python 3.10** instead if you need those!
-
-.. code-block:: python
-
- from __future__ import annotations
-
- t1: int | None
-
- # Type aliases
- T2 = int | None # TypeError!
-
-.. _alternative_union_syntax_older_version:
-
-Older versions
-""""""""""""""
-
-+------------------------------------------+-----------+-----------+-----------+
-| Python Version | 3.6 | 3.0 - 3.5 | 2.7 |
-+==========================================+===========+===========+===========+
-| Type comments | yes | yes | yes |
-+------------------------------------------+-----------+-----------+-----------+
-| Quoted function annotations | yes | yes | |
-+------------------------------------------+-----------+-----------+-----------+
-| Quoted variable annotations | yes | | |
-+------------------------------------------+-----------+-----------+-----------+
-| Everything else | | | |
-+------------------------------------------+-----------+-----------+-----------+
+ # Usable in type comments
+ t3 = 42 # type: int | str
.. _strict_optional:
@@ -565,82 +496,6 @@ valid for any type, but it's much more
useful for a programmer who is reading the code. This also makes
it easier to migrate to strict ``None`` checking in the future.
-Class name forward references
-*****************************
-
-Python does not allow references to a class object before the class is
-defined. Thus this code does not work as expected:
-
-.. code-block:: python
-
- def f(x: A) -> None: # Error: Name A not defined
- ...
-
- class A:
- ...
-
-In cases like these you can enter the type as a string literal — this
-is a *forward reference*:
-
-.. code-block:: python
-
- def f(x: 'A') -> None: # OK
- ...
-
- class A:
- ...
-
-Starting from Python 3.7 (:pep:`563`), you can add the special import ``from __future__ import annotations``,
-which makes the use of string literals in annotations unnecessary:
-
-.. code-block:: python
-
- from __future__ import annotations
-
- def f(x: A) -> None: # OK
- ...
-
- class A:
- ...
-
-.. note::
-
- Even with the ``__future__`` import, there are some scenarios that could still
- require string literals, typically involving use of forward references or generics in:
-
- * :ref:`type aliases `;
- * :ref:`casts `;
- * type definitions (see :py:class:`~typing.TypeVar`, :py:func:`~typing.NewType`, :py:class:`~typing.NamedTuple`);
- * base classes.
-
- .. code-block:: python
-
- # base class example
- class A(Tuple['B', 'C']): ... # OK
- class B: ...
- class C: ...
-
-Of course, instead of using a string literal type or special import, you could move the
-function definition after the class definition. This is not always
-desirable or even possible, though.
-
-Any type can be entered as a string literal, and you can combine
-string-literal types with non-string-literal types freely:
-
-.. code-block:: python
-
- def f(a: List['A']) -> None: ... # OK
- def g(n: 'int') -> None: ... # OK, though not useful
-
- class A: pass
-
-String literal types are never needed in ``# type:`` comments and :ref:`stub files `.
-
-String literal types must be defined (or imported) later *in the same
-module*. They cannot be used to leave cross-module references
-unresolved. (For dealing with import cycles, see
-:ref:`import-cycles`.)
-
.. _type-aliases:
Type aliases
diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst
new file mode 100644
index 000000000000..809c7dac1bb8
--- /dev/null
+++ b/docs/source/runtime_troubles.rst
@@ -0,0 +1,332 @@
+.. _runtime_troubles:
+
+Annotation issues at runtime
+============================
+
+Idiomatic use of type annotations can sometimes run up against what a given
+version of Python considers legal code. This section describes these scenarios
+and explains how to get your code running again. Generally speaking, we have
+three tools at our disposal:
+
+* For Python 3.7 through 3.9, use of ``from __future__ import annotations``
+ (:pep:`563`), made the default in Python 3.10 and later
+* Use of string literal types or type comments
+* Use of ``typing.TYPE_CHECKING``
+
+We provide a description of these before moving onto discussion of specific
+problems you may encounter.
+
+.. _string-literal-types:
+
+String literal types
+--------------------
+
+Type comments can't cause runtime errors because comments are not evaluated by
+Python. In a similar way, using string literal types sidesteps the problem of
+annotations that would cause runtime errors.
+
+Any type can be entered as a string literal, and you can combine
+string-literal types with non-string-literal types freely:
+
+.. code-block:: python
+
+ def f(a: List['A']) -> None: ... # OK
+ def g(n: 'int') -> None: ... # OK, though not useful
+
+ class A: pass
+
+String literal types are never needed in ``# type:`` comments and :ref:`stub files `.
+
+String literal types must be defined (or imported) later *in the same module*.
+They cannot be used to leave cross-module references unresolved. (For dealing
+with import cycles, see :ref:`import-cycles`.)
+
+.. _future-annotations:
+
+Future annotations import (PEP 563)
+-----------------------------------
+
+Many of the issues described here are caused by Python trying to evaluate
+annotations. From Python 3.10 on, Python will no longer attempt to evaluate
+function and variable annotations. This behaviour is made available in Python
+3.7 and later through the use of ``from __future__ import annotations``.
+
+This can be thought of as automatic string literal-ification of all function and
+variable annotations. Note that function and variable annotations are still
+required to be valid Python syntax. For more details, see :pep:`563`.
+
+.. note::
+
+ Even with the ``__future__`` import, there are some scenarios that could
+ still require string literals or result in errors, typically involving use
+ of forward references or generics in:
+
+ * :ref:`type aliases `;
+ * :ref:`casts `;
+ * type definitions (see :py:class:`~typing.TypeVar`, :py:func:`~typing.NewType`, :py:class:`~typing.NamedTuple`);
+ * base classes.
+
+ .. code-block:: python
+
+ # base class example
+ from __future__ import annotations
+ class A(Tuple['B', 'C']): ... # String literal types needed here
+ class B: ...
+ class C: ...
+
+.. note::
+
+ Some libraries may have use cases for dynamic evaluation of annotations, for
+ instance, through use of ``typing.get_type_hints`` or ``eval``. If your
+ annotation would raise an error when evaluated (say by using :pep:`604`
+ syntax with Python 3.9), you may need to be careful when using such
+ libraries.
+
+.. _typing-type-checking:
+
+typing.TYPE_CHECKING
+--------------------
+
+The :py:mod:`typing` module defines a :py:data:`~typing.TYPE_CHECKING` constant
+that is ``False`` at runtime but treated as ``True`` while type checking.
+
+Since code inside ``if TYPE_CHECKING:`` is not executed at runtime, it provides
+a convenient way to tell mypy something without the code being evaluated at
+runtime. This is most useful for resolving :ref:`import cycles `.
+
+.. note::
+
+ Python 3.5.1 and below don't have :py:data:`~typing.TYPE_CHECKING`. An
+ alternative is to define a constant named ``MYPY`` that has the value
+ ``False`` at runtime. Mypy considers it to be ``True`` when type checking.
+
+Class name forward references
+-----------------------------
+
+Python does not allow references to a class object before the class is
+defined (aka forward reference). Thus this code does not work as expected:
+
+.. code-block:: python
+
+ def f(x: A) -> None: ... # NameError: name 'A' is not defined
+ class A: ...
+
+Starting from Python 3.7, you can add ``from __future__ import annotations`` to
+resolve this, as discussed earlier:
+
+.. code-block:: python
+
+ from __future__ import annotations
+
+ def f(x: A) -> None: ... # OK
+ class A: ...
+
+For Python 3.6 and below, you can enter the type as a string literal or type comment:
+
+.. code-block:: python
+
+ def f(x: 'A') -> None: ... # OK
+
+ # Also OK
+ def g(x): # type: (A) -> None
+ ...
+
+ class A: ...
+
+Of course, instead of using future annotations import or string literal types,
+you could move the function definition after the class definition. This is not
+always desirable or even possible, though.
+
+.. _import-cycles:
+
+Import cycles
+-------------
+
+An import cycle occurs where module A imports module B and module B
+imports module A (perhaps indirectly, e.g. ``A -> B -> C -> A``).
+Sometimes in order to add type annotations you have to add extra
+imports to a module and those imports cause cycles that didn't exist
+before. This can lead to errors at runtime like:
+
+.. code-block::
+
+ ImportError: cannot import name 'b' from partially initialized module 'A' (most likely due to a circular import)
+
+If those cycles do become a problem when running your program, there's a trick:
+if the import is only needed for type annotations and you're using a) the
+:ref:`future annotations import`, or b) string literals or type
+comments for the relevant annotations, you can write the imports inside ``if
+TYPE_CHECKING:`` so that they are not executed at runtime. Example:
+
+File ``foo.py``:
+
+.. code-block:: python
+
+ from typing import List, TYPE_CHECKING
+
+ if TYPE_CHECKING:
+ import bar
+
+ def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]':
+ return [arg]
+
+File ``bar.py``:
+
+.. code-block:: python
+
+ from typing import List
+ from foo import listify
+
+ class BarClass:
+ def listifyme(self) -> 'List[BarClass]':
+ return listify(self)
+
+.. _not-generic-runtime:
+
+Using classes that are generic in stubs but not at runtime
+----------------------------------------------------------
+
+Some classes are declared as :ref:`generic` in stubs, but not
+at runtime.
+
+In Python 3.8 and earlier, there are several examples within the standard library,
+for instance, :py:class:`os.PathLike` and :py:class:`queue.Queue`. Subscripting
+such a class will result in a runtime error:
+
+.. code-block:: python
+
+ from queue import Queue
+
+ class Tasks(Queue[str]): # TypeError: 'type' object is not subscriptable
+ ...
+
+ results: Queue[int] = Queue() # TypeError: 'type' object is not subscriptable
+
+To avoid errors from use of these generics in annotations, just use the
+:ref:`future annotations import` (or string literals or type
+comments for Python 3.6 and below).
+
+To avoid errors when inheriting from these classes, things are a little more
+complicated and you need to use :ref:`typing.TYPE_CHECKING
+`:
+
+.. code-block:: python
+
+ from typing import TYPE_CHECKING
+ from queue import Queue
+
+ if TYPE_CHECKING:
+ BaseQueue = Queue[str] # this is only processed by mypy
+ else:
+ BaseQueue = Queue # this is not seen by mypy but will be executed at runtime
+
+ class Tasks(BaseQueue): # OK
+ ...
+
+ task_queue: Tasks
+ reveal_type(task_queue.get()) # Reveals str
+
+If your subclass is also generic, you can use the following:
+
+.. code-block:: python
+
+ from typing import TYPE_CHECKING, TypeVar, Generic
+ from queue import Queue
+
+ _T = TypeVar("_T")
+ if TYPE_CHECKING:
+ class _MyQueueBase(Queue[_T]): pass
+ else:
+ class _MyQueueBase(Generic[_T], Queue): pass
+
+ class MyQueue(_MyQueueBase[_T]): pass
+
+ task_queue: MyQueue[str]
+ reveal_type(task_queue.get()) # Reveals str
+
+In Python 3.9, we can just inherit directly from ``Queue[str]`` or ``Queue[T]``
+since its :py:class:`queue.Queue` implements :py:meth:`__class_getitem__`, so
+the class object can be subscripted at runtime without issue.
+
+Using types defined in stubs but not at runtime
+-----------------------------------------------
+
+Sometimes stubs that you're using may define types you wish to re-use that do
+not exist at runtime. Importing these types naively will cause your code to fail
+at runtime with ``ImportError`` or ``ModuleNotFoundError``. Similar to previous
+sections, these can be dealt with by using :ref:`typing.TYPE_CHECKING
+`:
+
+.. code-block:: python
+
+ from typing import TYPE_CHECKING
+ if TYPE_CHECKING:
+ from _typeshed import SupportsLessThan
+
+.. _generic-builtins:
+
+Using generic builtins
+----------------------
+
+Starting with Python 3.9 (:pep:`585`), the type objects of many collections in
+the standard library support subscription at runtime. This means that you no
+longer have to import the equivalents from :py:mod:`typing`; you can simply use
+the built-in collections or those from :py:mod:`collections.abc`:
+
+.. code-block:: python
+
+ from collections.abc import Sequence
+ x: list[str]
+ y: dict[int, str]
+ z: Sequence[str] = x
+
+There is limited support for using this syntax in Python 3.7 and later as well.
+If you use ``from __future__ import annotations``, mypy will understand this
+syntax in annotations. However, since this will not be supported by the Python
+interpreter at runtime, make sure you're aware of the caveats mentioned in the
+notes at :ref:`future annotations import`.
+
+Using X | Y syntax for Unions
+-----------------------------
+
+Starting with Python 3.10 (:pep:`604`), you can spell union types as ``x: int |
+str``, instead of ``x: typing.Union[int, str]``.
+
+There is limited support for using this syntax in Python 3.7 and later as well.
+If you use ``from __future__ import annotations``, mypy will understand this
+syntax in annotations, string literal types, type comments and stub files.
+However, since this will not be supported by the Python interpreter at runtime
+(if evaluated, ``int | str`` will raise ``TypeError: unsupported operand type(s)
+for |: 'type' and 'type'``), make sure you're aware of the caveats mentioned in
+the notes at :ref:`future annotations import`.
+
+Using new additions to the typing module
+----------------------------------------
+
+You may find yourself wanting to use features added to the :py:mod:`typing`
+module in earlier versions of Python than the addition, for example, using any
+of ``Literal``, ``Protocol``, ``TypedDict`` with Python 3.6.
+
+The easiest way to do this is to install and use the ``typing_extensions``
+package from PyPI for the relevant imports, for example:
+
+.. code-block:: python
+
+ from typing_extensions import Literal
+ x: Literal["open", "close"]
+
+If you don't want to rely on ``typing_extensions`` being installed on newer
+Pythons, you could alternatively use:
+
+.. code-block:: python
+
+ import sys
+ if sys.version_info >= (3, 8):
+ from typing import Literal
+ else:
+ from typing_extensions import Literal
+
+ x: Literal["open", "close"]
+
+This plays nicely well with following :pep:`508` dependency specification:
+``typing_extensions; python_version<"3.8"``