diff --git a/docs/source/builtin_types.rst b/docs/source/builtin_types.rst index 7ad5fbcfa2cc..5aa84922307d 100644 --- a/docs/source/builtin_types.rst +++ b/docs/source/builtin_types.rst @@ -38,3 +38,8 @@ though they are similar to abstract base classes defined in :py:mod:`collections.abc` (formerly ``collections``), they are not identical. In particular, prior to Python 3.9, the built-in collection type objects do not support indexing. + +In Python 3.9 and later, built-in collection type objects support indexing. This +means that you can use built-in classes or those from :py:mod:`collections.abc` +instead of importing from :py:mod:`typing`. See :ref:`generic-builtins` for more +details. diff --git a/docs/source/common_issues.rst b/docs/source/common_issues.rst index 3867e168bd6a..79c94ac2baff 100644 --- a/docs/source/common_issues.rst +++ b/docs/source/common_issues.rst @@ -210,6 +210,21 @@ checking would require a large number of ``assert foo is not None`` checks to be inserted, and you want to minimize the number of code changes required to get a clean mypy run. +Issues with code at runtime +--------------------------- + +Idiomatic use of type annotations can sometimes run up against what a given +version of Python considers legal code. These can result in some of the +following errors when trying to run your code: + +* ``ImportError`` from circular imports +* ``NameError: name 'X' is not defined`` from forward references +* ``TypeError: 'type' object is not subscriptable`` from types that are not generic at runtime +* ``ImportError`` or ``ModuleNotFoundError`` from use of stub definitions not available at runtime +* ``TypeError: unsupported operand type(s) for |: 'type' and 'type'`` from use of new syntax + +For dealing with these, see :ref:`runtime_troubles`. + Mypy runs are slow ------------------ @@ -499,112 +514,6 @@ to see the types of all local variables at once. Example: run your code. Both are always available and you don't need to import them. - -.. _import-cycles: - -Import cycles -------------- - -An import cycle occurs where module A imports module B and module B -imports module A (perhaps indirectly, e.g. ``A -> B -> C -> A``). -Sometimes in order to add type annotations you have to add extra -imports to a module and those imports cause cycles that didn't exist -before. If those cycles become a problem when running your program, -there's a trick: if the import is only needed for type annotations in -forward references (string literals) or comments, you can write the -imports inside ``if TYPE_CHECKING:`` so that they are not executed at runtime. -Example: - -File ``foo.py``: - -.. code-block:: python - - from typing import List, TYPE_CHECKING - - if TYPE_CHECKING: - import bar - - def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]': - return [arg] - -File ``bar.py``: - -.. code-block:: python - - from typing import List - from foo import listify - - class BarClass: - def listifyme(self) -> 'List[BarClass]': - return listify(self) - -.. note:: - - The :py:data:`~typing.TYPE_CHECKING` constant defined by the :py:mod:`typing` module - is ``False`` at runtime but ``True`` while type checking. - -Python 3.5.1 doesn't have :py:data:`~typing.TYPE_CHECKING`. An alternative is -to define a constant named ``MYPY`` that has the value ``False`` -at runtime. Mypy considers it to be ``True`` when type checking. -Here's the above example modified to use ``MYPY``: - -.. code-block:: python - - from typing import List - - MYPY = False - if MYPY: - import bar - - def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]': - return [arg] - -.. _not-generic-runtime: - -Using classes that are generic in stubs but not at runtime ----------------------------------------------------------- - -Some classes are declared as generic in stubs, but not at runtime. Examples -in the standard library include :py:class:`os.PathLike` and :py:class:`queue.Queue`. -Subscripting such a class will result in a runtime error: - -.. code-block:: python - - from queue import Queue - - class Tasks(Queue[str]): # TypeError: 'type' object is not subscriptable - ... - - results: Queue[int] = Queue() # TypeError: 'type' object is not subscriptable - -To avoid these errors while still having precise types you can either use -string literal types or :py:data:`~typing.TYPE_CHECKING`: - -.. code-block:: python - - from queue import Queue - from typing import TYPE_CHECKING - - if TYPE_CHECKING: - BaseQueue = Queue[str] # this is only processed by mypy - else: - BaseQueue = Queue # this is not seen by mypy but will be executed at runtime. - - class Tasks(BaseQueue): # OK - ... - - results: 'Queue[int]' = Queue() # OK - -If you are running Python 3.7+ you can use ``from __future__ import annotations`` -as a (nicer) alternative to string quotes, read more in :pep:`563`. For example: - -.. code-block:: python - - from __future__ import annotations - from queue import Queue - - results: Queue[int] = Queue() # This works at runtime - .. _silencing-linters: Silencing linters diff --git a/docs/source/index.rst b/docs/source/index.rst index 4a8cb59cf1a9..3295f9ca38cc 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -35,6 +35,7 @@ Mypy is a static type checker for Python 3 and Python 2.7. type_inference_and_annotations kinds_of_types class_basics + runtime_troubles protocols python2 dynamic_typing diff --git a/docs/source/kinds_of_types.rst b/docs/source/kinds_of_types.rst index facc5da5a64c..9000b6aa238f 100644 --- a/docs/source/kinds_of_types.rst +++ b/docs/source/kinds_of_types.rst @@ -243,93 +243,24 @@ more specific type: .. _alternative_union_syntax: -Alternative union syntax ------------------------- +X | Y syntax for Unions +----------------------- -`PEP 604 `_ introduced an alternative way -for writing union types. Starting with **Python 3.10** it is possible to write -``Union[int, str]`` as ``int | str``. Any of the following options is possible +:pep:`604` introduced an alternative way for spelling union types. In Python +3.10 and later, you can write ``Union[int, str]`` as ``int | str``. It is +possible to use this syntax in versions of Python where it isn't supported by +the runtime with some limitations, see :ref:`runtime_troubles`. .. code-block:: python from typing import List - # Use as Union t1: int | str # equivalent to Union[int, str] - # Use as Optional t2: int | None # equivalent to Optional[int] - # Use in generics - t3: List[int | str] # equivalent to List[Union[int, str]] - - # Use in type aliases - T4 = int | None - x: T4 - - # Quoted variable annotations - t5: "int | str" - - # Quoted function annotations - def f(t6: "int | str") -> None: ... - - # Type comments - t6 = 42 # type: int | str - -It is possible to use most of these even for earlier versions. However there are some -limitations to be aware of. - -.. _alternative_union_syntax_stub_files: - -Stub files -"""""""""" - -All options are supported, regardless of the Python version the project uses. - -.. _alternative_union_syntax_37: - -Python 3.7 - 3.9 -"""""""""""""""" - -It is necessary to add ``from __future__ import annotations`` to delay the evaluation -of type annotations. Not using it would result in a ``TypeError``. -This does not apply for **type comments**, **quoted function** and **quoted variable** annotations, -as those also work for earlier versions, see :ref:`below `. - -.. warning:: - - Type aliases are **NOT** supported! Those result in a ``TypeError`` regardless - if the evaluation of type annotations is delayed. - - Dynamic evaluation of annotations is **NOT** possible (e.g. ``typing.get_type_hints`` and ``eval``). - See `note PEP 604 `_. - Use ``typing.Union`` or **Python 3.10** instead if you need those! - -.. code-block:: python - - from __future__ import annotations - - t1: int | None - - # Type aliases - T2 = int | None # TypeError! - -.. _alternative_union_syntax_older_version: - -Older versions -"""""""""""""" - -+------------------------------------------+-----------+-----------+-----------+ -| Python Version | 3.6 | 3.0 - 3.5 | 2.7 | -+==========================================+===========+===========+===========+ -| Type comments | yes | yes | yes | -+------------------------------------------+-----------+-----------+-----------+ -| Quoted function annotations | yes | yes | | -+------------------------------------------+-----------+-----------+-----------+ -| Quoted variable annotations | yes | | | -+------------------------------------------+-----------+-----------+-----------+ -| Everything else | | | | -+------------------------------------------+-----------+-----------+-----------+ + # Usable in type comments + t3 = 42 # type: int | str .. _strict_optional: @@ -565,82 +496,6 @@ valid for any type, but it's much more useful for a programmer who is reading the code. This also makes it easier to migrate to strict ``None`` checking in the future. -Class name forward references -***************************** - -Python does not allow references to a class object before the class is -defined. Thus this code does not work as expected: - -.. code-block:: python - - def f(x: A) -> None: # Error: Name A not defined - ... - - class A: - ... - -In cases like these you can enter the type as a string literal — this -is a *forward reference*: - -.. code-block:: python - - def f(x: 'A') -> None: # OK - ... - - class A: - ... - -Starting from Python 3.7 (:pep:`563`), you can add the special import ``from __future__ import annotations``, -which makes the use of string literals in annotations unnecessary: - -.. code-block:: python - - from __future__ import annotations - - def f(x: A) -> None: # OK - ... - - class A: - ... - -.. note:: - - Even with the ``__future__`` import, there are some scenarios that could still - require string literals, typically involving use of forward references or generics in: - - * :ref:`type aliases `; - * :ref:`casts `; - * type definitions (see :py:class:`~typing.TypeVar`, :py:func:`~typing.NewType`, :py:class:`~typing.NamedTuple`); - * base classes. - - .. code-block:: python - - # base class example - class A(Tuple['B', 'C']): ... # OK - class B: ... - class C: ... - -Of course, instead of using a string literal type or special import, you could move the -function definition after the class definition. This is not always -desirable or even possible, though. - -Any type can be entered as a string literal, and you can combine -string-literal types with non-string-literal types freely: - -.. code-block:: python - - def f(a: List['A']) -> None: ... # OK - def g(n: 'int') -> None: ... # OK, though not useful - - class A: pass - -String literal types are never needed in ``# type:`` comments and :ref:`stub files `. - -String literal types must be defined (or imported) later *in the same -module*. They cannot be used to leave cross-module references -unresolved. (For dealing with import cycles, see -:ref:`import-cycles`.) - .. _type-aliases: Type aliases diff --git a/docs/source/runtime_troubles.rst b/docs/source/runtime_troubles.rst new file mode 100644 index 000000000000..809c7dac1bb8 --- /dev/null +++ b/docs/source/runtime_troubles.rst @@ -0,0 +1,332 @@ +.. _runtime_troubles: + +Annotation issues at runtime +============================ + +Idiomatic use of type annotations can sometimes run up against what a given +version of Python considers legal code. This section describes these scenarios +and explains how to get your code running again. Generally speaking, we have +three tools at our disposal: + +* For Python 3.7 through 3.9, use of ``from __future__ import annotations`` + (:pep:`563`), made the default in Python 3.10 and later +* Use of string literal types or type comments +* Use of ``typing.TYPE_CHECKING`` + +We provide a description of these before moving onto discussion of specific +problems you may encounter. + +.. _string-literal-types: + +String literal types +-------------------- + +Type comments can't cause runtime errors because comments are not evaluated by +Python. In a similar way, using string literal types sidesteps the problem of +annotations that would cause runtime errors. + +Any type can be entered as a string literal, and you can combine +string-literal types with non-string-literal types freely: + +.. code-block:: python + + def f(a: List['A']) -> None: ... # OK + def g(n: 'int') -> None: ... # OK, though not useful + + class A: pass + +String literal types are never needed in ``# type:`` comments and :ref:`stub files `. + +String literal types must be defined (or imported) later *in the same module*. +They cannot be used to leave cross-module references unresolved. (For dealing +with import cycles, see :ref:`import-cycles`.) + +.. _future-annotations: + +Future annotations import (PEP 563) +----------------------------------- + +Many of the issues described here are caused by Python trying to evaluate +annotations. From Python 3.10 on, Python will no longer attempt to evaluate +function and variable annotations. This behaviour is made available in Python +3.7 and later through the use of ``from __future__ import annotations``. + +This can be thought of as automatic string literal-ification of all function and +variable annotations. Note that function and variable annotations are still +required to be valid Python syntax. For more details, see :pep:`563`. + +.. note:: + + Even with the ``__future__`` import, there are some scenarios that could + still require string literals or result in errors, typically involving use + of forward references or generics in: + + * :ref:`type aliases `; + * :ref:`casts `; + * type definitions (see :py:class:`~typing.TypeVar`, :py:func:`~typing.NewType`, :py:class:`~typing.NamedTuple`); + * base classes. + + .. code-block:: python + + # base class example + from __future__ import annotations + class A(Tuple['B', 'C']): ... # String literal types needed here + class B: ... + class C: ... + +.. note:: + + Some libraries may have use cases for dynamic evaluation of annotations, for + instance, through use of ``typing.get_type_hints`` or ``eval``. If your + annotation would raise an error when evaluated (say by using :pep:`604` + syntax with Python 3.9), you may need to be careful when using such + libraries. + +.. _typing-type-checking: + +typing.TYPE_CHECKING +-------------------- + +The :py:mod:`typing` module defines a :py:data:`~typing.TYPE_CHECKING` constant +that is ``False`` at runtime but treated as ``True`` while type checking. + +Since code inside ``if TYPE_CHECKING:`` is not executed at runtime, it provides +a convenient way to tell mypy something without the code being evaluated at +runtime. This is most useful for resolving :ref:`import cycles `. + +.. note:: + + Python 3.5.1 and below don't have :py:data:`~typing.TYPE_CHECKING`. An + alternative is to define a constant named ``MYPY`` that has the value + ``False`` at runtime. Mypy considers it to be ``True`` when type checking. + +Class name forward references +----------------------------- + +Python does not allow references to a class object before the class is +defined (aka forward reference). Thus this code does not work as expected: + +.. code-block:: python + + def f(x: A) -> None: ... # NameError: name 'A' is not defined + class A: ... + +Starting from Python 3.7, you can add ``from __future__ import annotations`` to +resolve this, as discussed earlier: + +.. code-block:: python + + from __future__ import annotations + + def f(x: A) -> None: ... # OK + class A: ... + +For Python 3.6 and below, you can enter the type as a string literal or type comment: + +.. code-block:: python + + def f(x: 'A') -> None: ... # OK + + # Also OK + def g(x): # type: (A) -> None + ... + + class A: ... + +Of course, instead of using future annotations import or string literal types, +you could move the function definition after the class definition. This is not +always desirable or even possible, though. + +.. _import-cycles: + +Import cycles +------------- + +An import cycle occurs where module A imports module B and module B +imports module A (perhaps indirectly, e.g. ``A -> B -> C -> A``). +Sometimes in order to add type annotations you have to add extra +imports to a module and those imports cause cycles that didn't exist +before. This can lead to errors at runtime like: + +.. code-block:: + + ImportError: cannot import name 'b' from partially initialized module 'A' (most likely due to a circular import) + +If those cycles do become a problem when running your program, there's a trick: +if the import is only needed for type annotations and you're using a) the +:ref:`future annotations import`, or b) string literals or type +comments for the relevant annotations, you can write the imports inside ``if +TYPE_CHECKING:`` so that they are not executed at runtime. Example: + +File ``foo.py``: + +.. code-block:: python + + from typing import List, TYPE_CHECKING + + if TYPE_CHECKING: + import bar + + def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]': + return [arg] + +File ``bar.py``: + +.. code-block:: python + + from typing import List + from foo import listify + + class BarClass: + def listifyme(self) -> 'List[BarClass]': + return listify(self) + +.. _not-generic-runtime: + +Using classes that are generic in stubs but not at runtime +---------------------------------------------------------- + +Some classes are declared as :ref:`generic` in stubs, but not +at runtime. + +In Python 3.8 and earlier, there are several examples within the standard library, +for instance, :py:class:`os.PathLike` and :py:class:`queue.Queue`. Subscripting +such a class will result in a runtime error: + +.. code-block:: python + + from queue import Queue + + class Tasks(Queue[str]): # TypeError: 'type' object is not subscriptable + ... + + results: Queue[int] = Queue() # TypeError: 'type' object is not subscriptable + +To avoid errors from use of these generics in annotations, just use the +:ref:`future annotations import` (or string literals or type +comments for Python 3.6 and below). + +To avoid errors when inheriting from these classes, things are a little more +complicated and you need to use :ref:`typing.TYPE_CHECKING +`: + +.. code-block:: python + + from typing import TYPE_CHECKING + from queue import Queue + + if TYPE_CHECKING: + BaseQueue = Queue[str] # this is only processed by mypy + else: + BaseQueue = Queue # this is not seen by mypy but will be executed at runtime + + class Tasks(BaseQueue): # OK + ... + + task_queue: Tasks + reveal_type(task_queue.get()) # Reveals str + +If your subclass is also generic, you can use the following: + +.. code-block:: python + + from typing import TYPE_CHECKING, TypeVar, Generic + from queue import Queue + + _T = TypeVar("_T") + if TYPE_CHECKING: + class _MyQueueBase(Queue[_T]): pass + else: + class _MyQueueBase(Generic[_T], Queue): pass + + class MyQueue(_MyQueueBase[_T]): pass + + task_queue: MyQueue[str] + reveal_type(task_queue.get()) # Reveals str + +In Python 3.9, we can just inherit directly from ``Queue[str]`` or ``Queue[T]`` +since its :py:class:`queue.Queue` implements :py:meth:`__class_getitem__`, so +the class object can be subscripted at runtime without issue. + +Using types defined in stubs but not at runtime +----------------------------------------------- + +Sometimes stubs that you're using may define types you wish to re-use that do +not exist at runtime. Importing these types naively will cause your code to fail +at runtime with ``ImportError`` or ``ModuleNotFoundError``. Similar to previous +sections, these can be dealt with by using :ref:`typing.TYPE_CHECKING +`: + +.. code-block:: python + + from typing import TYPE_CHECKING + if TYPE_CHECKING: + from _typeshed import SupportsLessThan + +.. _generic-builtins: + +Using generic builtins +---------------------- + +Starting with Python 3.9 (:pep:`585`), the type objects of many collections in +the standard library support subscription at runtime. This means that you no +longer have to import the equivalents from :py:mod:`typing`; you can simply use +the built-in collections or those from :py:mod:`collections.abc`: + +.. code-block:: python + + from collections.abc import Sequence + x: list[str] + y: dict[int, str] + z: Sequence[str] = x + +There is limited support for using this syntax in Python 3.7 and later as well. +If you use ``from __future__ import annotations``, mypy will understand this +syntax in annotations. However, since this will not be supported by the Python +interpreter at runtime, make sure you're aware of the caveats mentioned in the +notes at :ref:`future annotations import`. + +Using X | Y syntax for Unions +----------------------------- + +Starting with Python 3.10 (:pep:`604`), you can spell union types as ``x: int | +str``, instead of ``x: typing.Union[int, str]``. + +There is limited support for using this syntax in Python 3.7 and later as well. +If you use ``from __future__ import annotations``, mypy will understand this +syntax in annotations, string literal types, type comments and stub files. +However, since this will not be supported by the Python interpreter at runtime +(if evaluated, ``int | str`` will raise ``TypeError: unsupported operand type(s) +for |: 'type' and 'type'``), make sure you're aware of the caveats mentioned in +the notes at :ref:`future annotations import`. + +Using new additions to the typing module +---------------------------------------- + +You may find yourself wanting to use features added to the :py:mod:`typing` +module in earlier versions of Python than the addition, for example, using any +of ``Literal``, ``Protocol``, ``TypedDict`` with Python 3.6. + +The easiest way to do this is to install and use the ``typing_extensions`` +package from PyPI for the relevant imports, for example: + +.. code-block:: python + + from typing_extensions import Literal + x: Literal["open", "close"] + +If you don't want to rely on ``typing_extensions`` being installed on newer +Pythons, you could alternatively use: + +.. code-block:: python + + import sys + if sys.version_info >= (3, 8): + from typing import Literal + else: + from typing_extensions import Literal + + x: Literal["open", "close"] + +This plays nicely well with following :pep:`508` dependency specification: +``typing_extensions; python_version<"3.8"``