From 40e92a2ff49e0e8a4ec27044e60b17b0dd1a56e4 Mon Sep 17 00:00:00 2001 From: Shantanu <12621235+hauntsaninja@users.noreply.github.com> Date: Tue, 19 Jan 2021 11:01:34 -0800 Subject: [PATCH] Document new source finding behaviour (#9923) This should cover the current state on master, as previously discussed / implemented across #9742, #9683, #9632, #9616, #9614, etc. This will need to be changed if we can make `--namespace-packages` the default (#9636). I haven't documented some of the finer points of the changes, since it felt like an inappropriate level of detail (e.g. using absolute paths when crawling, how directories with invalid package names affect crawling, etc) Co-authored-by: hauntsaninja <> --- docs/source/command_line.rst | 16 ++++-- docs/source/running_mypy.rst | 99 +++++++++++++++++++++--------------- 2 files changed, 71 insertions(+), 44 deletions(-) diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index 53fad0566bfd..40df775742a6 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -110,10 +110,18 @@ imports. passed on the command line, the ``MYPYPATH`` environment variable, and the :confval:`mypy_path` config option. - Note that this only affects import discovery -- for modules and - packages explicitly passed on the command line, mypy still - searches for ``__init__.py[i]`` files in order to determine the - fully-qualified module/package name. + This flag affects how mypy finds modules and packages explicitly passed on + the command line. It also affects how mypy determines fully qualified module + names for files passed on the command line. See :ref:`Mapping file paths to + modules ` for details. + +.. option:: --explicit-package-bases + + This flag tells mypy that top-level packages will be based in either the + current directory, or a member of the ``MYPYPATH`` environment variable or + :confval:`mypy_path` config option. This option is only useful in + conjunction with :option:`--namespace-packages`. See :ref:`Mapping file + paths to modules ` for details. .. option:: --ignore-missing-imports diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 9c0e9bc1a03f..866b57cf91df 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -24,8 +24,11 @@ actual way mypy type checks your code, see our Specifying code to be checked ***************************** -Mypy lets you specify what files it should type check in several -different ways. +Mypy lets you specify what files it should type check in several different ways. + +Note that if you use namespace packages (in particular, packages without +``__init__.py``), you'll need to specify :option:`--namespace-packages `. 1. First, you can pass in paths to Python files and directories you want to type check. For example:: @@ -336,58 +339,79 @@ while this option can be quite powerful, it can also cause many hard-to-debug errors. - .. _mapping-paths-to-modules: Mapping file paths to modules ***************************** -One of the main ways you can tell mypy what files to type check -is by providing mypy the paths to those files. For example:: +One of the main ways you can tell mypy what to type check +is by providing mypy a list of paths. For example:: $ mypy file_1.py foo/file_2.py file_3.pyi some/directory This section describes how exactly mypy maps the provided paths to modules to type check. -- Files ending in ``.py`` (and stub files ending in ``.pyi``) are - checked as Python modules. +- Mypy will check all paths provided that correspond to files. + +- Mypy will recursively discover and check all files ending in ``.py`` or + ``.pyi`` in directory paths provided. + +- For each file to be checked, mypy will attempt to associate the file (e.g. + ``project/foo/bar/baz.py``) with a fully qualified module name (e.g. + ``foo.bar.baz``). The directory the package is in (``project``) is then + added to mypy's module search paths. -- Files not ending in ``.py`` or ``.pyi`` are assumed to be Python - scripts and checked as such. +How mypy determines fully qualified module names depends on if the options +:option:`--namespace-packages ` and +:option:`--explicit-package-bases ` are set. -- Directories representing Python packages (i.e. containing a - ``__init__.py[i]`` file) are checked as Python packages; all - submodules and subpackages will be checked (subpackages must - themselves have a ``__init__.py[i]`` file). +1. If :option:`--namespace-packages ` is off, + mypy will rely solely upon the presence of ``__init__.py[i]`` files to + determine the fully qualified module name. That is, mypy will crawl up the + directory tree for as long as it continues to find ``__init__.py`` (or + ``__init__.pyi``) files. -- Directories that don't represent Python packages (i.e. not directly - containing an ``__init__.py[i]`` file) are checked as follows: + For example, if your directory tree consists of ``pkg/subpkg/mod.py``, mypy + would require ``pkg/__init__.py`` and ``pkg/subpkg/__init__.py`` to exist in + order correctly associate ``mod.py`` with ``pkg.subpkg.mod`` - - All ``*.py[i]`` files contained directly therein are checked as - toplevel Python modules; +2. If :option:`--namespace-packages ` is on, but + :option:`--explicit-package-bases ` is off, + mypy will allow for the possibility that directories without + ``__init__.py[i]`` are packages. Specifically, mypy will look at all parent + directories of the file and use the location of the highest + ``__init__.py[i]`` in the directory tree to determine the top-level package. - - All packages contained directly therein (i.e. immediate - subdirectories with an ``__init__.py[i]`` file) are checked as - toplevel Python packages. + For example, say your directory tree consists solely of ``pkg/__init__.py`` + and ``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified + module name, mypy will look at ``pkg/__init__.py`` and conclude that the + associated module name is ``pkg.a.b.c.d.mod``. -One more thing about checking modules and packages: if the directory -*containing* a module or package specified on the command line has an -``__init__.py[i]`` file, mypy assigns these an absolute module name by -crawling up the path until no ``__init__.py[i]`` file is found. +3. You'll notice that the above case still relies on ``__init__.py``. If + you can't put an ``__init__.py`` in your top-level package, but still wish to + pass paths (as opposed to packages or modules using the ``-p`` or ``-m`` + flags), :option:`--explicit-package-bases ` + provides a solution. -For example, suppose we run the command ``mypy foo/bar/baz.py`` where -``foo/bar/__init__.py`` exists but ``foo/__init__.py`` does not. Then -the module name assumed is ``bar.baz`` and the directory ``foo`` is -added to mypy's module search path. + With :option:`--explicit-package-bases `, mypy + will locate the nearest parent directory that is a member of the ``MYPYPATH`` + environment variable, the :confval:`mypy_path` config or is the current + working directory. mypy will then use the relative path to determine the + fully qualified module name. -On the other hand, if ``foo/bar/__init__.py`` did not exist, ``foo/bar`` -would be added to the module search path instead, and the module name -assumed is just ``baz``. + For example, say your directory tree consists solely of + ``src/namespace_pkg/mod.py``. If you run the command following command, mypy + will correctly associate ``mod.py`` with ``namespace_pkg.mod``:: -If a script (a file not ending in ``.py[i]``) is processed, the module -name assumed is ``__main__`` (matching the behavior of the -Python interpreter), unless :option:`--scripts-are-modules ` is passed. + $ MYPYPATH=src mypy --namespace-packages --explicit-package-bases . + +If you pass a file not ending in ``.py[i]``, the module name assumed is +``__main__`` (matching the behavior of the Python interpreter), unless +:option:`--scripts-are-modules ` is passed. + +Passing :option:`-v ` will show you the files and associated module +names that mypy will check. .. _finding-imports: @@ -407,7 +431,7 @@ This is computed from the following items: (a colon-separated list of directories). - The :confval:`mypy_path` config file option. - The directories containing the sources given on the command line - (see below). + (see :ref:`Mapping file paths to modules `). - The installed packages marked as safe for type checking (see :ref:`PEP 561 support `) - The relevant directories of the @@ -418,11 +442,6 @@ This is computed from the following items: You cannot point to a :pep:`561` package via the ``MYPYPATH``, it must be installed (see :ref:`PEP 561 support `) -For sources given on the command line, the path is adjusted by crawling -up from the given file or package to the nearest directory that does not -contain an ``__init__.py`` or ``__init__.pyi`` file. If the given path -is relative, it will only crawl as far as the current working directory. - Second, mypy searches for stub files in addition to regular Python files and packages. The rules for searching for a module ``foo`` are as follows: