From 44fb9f56eabcd4fd69e90d6e3b4e428e47635c65 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Sun, 17 Jan 2021 23:48:48 -0800 Subject: [PATCH 1/4] Document new source finding behaviour This should cover the current state on master, as previously discussed / implemented across #9742, #9683, #9632, #9616, #9614, etc. This will need to be changed if we can make `--namespace-packages` the default (#9636). I haven't documented some of the finer points of the changes, since it felt like an inappropriate level of detail (e.g. using absolute paths when crawling, how directories with invalid package names affect crawling, etc) --- docs/source/command_line.rst | 15 ++++-- docs/source/running_mypy.rst | 94 +++++++++++++++++++++--------------- 2 files changed, 65 insertions(+), 44 deletions(-) diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index 53fad0566bfd..91f91bd9bbd4 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -110,10 +110,17 @@ imports. passed on the command line, the ``MYPYPATH`` environment variable, and the :confval:`mypy_path` config option. - Note that this only affects import discovery -- for modules and - packages explicitly passed on the command line, mypy still - searches for ``__init__.py[i]`` files in order to determine the - fully-qualified module/package name. + This flag also affects how mypy determines fully qualified module names for + files, modules and packages explicitly passed on the command line. See + :ref:`Mapping file paths to modules ` for details. + +.. option:: --explicit-package-bases + + This flag tells mypy to that the current directory, the ``MYPYPATH`` + environment variable, and the :confval:`mypy_path` config option as the + directories in which top-level packages are located. This option is only + useful in conjunction with :option:`--namespace-packages`. See :ref:`Mapping + file paths to modules ` for details. .. option:: --ignore-missing-imports diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 9c0e9bc1a03f..32060d4ba0d3 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -24,8 +24,11 @@ actual way mypy type checks your code, see our Specifying code to be checked ***************************** -Mypy lets you specify what files it should type check in several -different ways. +Mypy lets you specify what files it should type check in several different ways. + +Note that if you use namespace packages (in particular, packages without +``__init__.py``), you'll need to specify :option:`--namespace-packages `. 1. First, you can pass in paths to Python files and directories you want to type check. For example:: @@ -336,58 +339,74 @@ while this option can be quite powerful, it can also cause many hard-to-debug errors. - .. _mapping-paths-to-modules: Mapping file paths to modules ***************************** -One of the main ways you can tell mypy what files to type check -is by providing mypy the paths to those files. For example:: +One of the main ways you can tell mypy what to type check +is by providing mypy a list of paths. For example:: $ mypy file_1.py foo/file_2.py file_3.pyi some/directory This section describes how exactly mypy maps the provided paths to modules to type check. -- Files ending in ``.py`` (and stub files ending in ``.pyi``) are - checked as Python modules. +- mypy will check all paths provided that correspond to files. + +- mypy will recursively discover and check all files ending in ``.py`` or + ``.pyi`` in directory paths provided. -- Files not ending in ``.py`` or ``.pyi`` are assumed to be Python - scripts and checked as such. +- For each file to be checked, mypy will attempt to associate the file (e.g. + ``project/foo/bar/baz.py``) with a fully qualified module name (e.g. + ``foo.bar.baz``). The directory the package is in (``project``) is then + add to mypy's module search paths. -- Directories representing Python packages (i.e. containing a - ``__init__.py[i]`` file) are checked as Python packages; all - submodules and subpackages will be checked (subpackages must - themselves have a ``__init__.py[i]`` file). +How mypy determines fully qualified module names depends on if the options +:option:`--namespace-packages ` and +:option:`--explicit-package-bases ` are set. -- Directories that don't represent Python packages (i.e. not directly - containing an ``__init__.py[i]`` file) are checked as follows: +First, if :option:`--namespace-packages ` is off, +mypy will rely solely upon the presence of ``__init__.py[i]`` files to determine +the fully qualified module name. That is, mypy will crawl up the directory tree +for as long as it continues to find ``__init__.py`` (or ``__init__.pyi``) files. - - All ``*.py[i]`` files contained directly therein are checked as - toplevel Python modules; +Second, if :option:`--namespace-packages ` is on, but +:option:`--explicit-package-bases ` is off, mypy +will allow for the possibility that directories without ``__init__.py[i]`` are +packages. Specifically, mypy will look at all parent directories of the file and +use the location of the highest ``__init__.py[i]`` in the directory tree to +determine the top-level package. - - All packages contained directly therein (i.e. immediate - subdirectories with an ``__init__.py[i]`` file) are checked as - toplevel Python packages. +For example, say your directory tree consists solely of ``pkg/__init__.py`` and +``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified module +name, mypy will look at ``pkg/__init__.py`` and conclude that the associated +module name is ``pkg.a.b.c.d.mod``. -One more thing about checking modules and packages: if the directory -*containing* a module or package specified on the command line has an -``__init__.py[i]`` file, mypy assigns these an absolute module name by -crawling up the path until no ``__init__.py[i]`` file is found. +You'll notice that that method still relies on ``__init__.py``. If you can't put +an ``__init__.py`` in your top-level package, but still wish to pass paths (as +opposed to packages or modules using the ``-p`` or ``-m`` flags), +:option:`--explicit-package-bases ` provides a +solution. -For example, suppose we run the command ``mypy foo/bar/baz.py`` where -``foo/bar/__init__.py`` exists but ``foo/__init__.py`` does not. Then -the module name assumed is ``bar.baz`` and the directory ``foo`` is -added to mypy's module search path. +Third, with :option:`--explicit-package-bases `, +mypy will locate the nearest parent directory that is a member of the +``MYPYPATH`` environment variable, the :confval:`mypy_path` config or is the +current working directory. mypy will then use the relative path to determine the +fully qualified module name. -On the other hand, if ``foo/bar/__init__.py`` did not exist, ``foo/bar`` -would be added to the module search path instead, and the module name -assumed is just ``baz``. +For example, say your directory tree consists solely of +``src/namespace_pkg/mod.py``. If you run the command following command, mypy +will correctly associate ``mod.py`` with ``namespace_pkg.mod``:: -If a script (a file not ending in ``.py[i]``) is processed, the module -name assumed is ``__main__`` (matching the behavior of the -Python interpreter), unless :option:`--scripts-are-modules ` is passed. + $ MYPYPATH=src mypy --namespace-packages --explicit-package-bases . + +If you pass a file not ending in ``.py[i]``, the module name assumed is +``__main__`` (matching the behavior of the Python interpreter), unless +:option:`--scripts-are-modules ` is passed. + +Passing :option:`-v ` will show you the files and associated module +names that mypy will check. .. _finding-imports: @@ -407,7 +426,7 @@ This is computed from the following items: (a colon-separated list of directories). - The :confval:`mypy_path` config file option. - The directories containing the sources given on the command line - (see below). + (see :ref:`Mapping file paths to modules `). - The installed packages marked as safe for type checking (see :ref:`PEP 561 support `) - The relevant directories of the @@ -418,11 +437,6 @@ This is computed from the following items: You cannot point to a :pep:`561` package via the ``MYPYPATH``, it must be installed (see :ref:`PEP 561 support `) -For sources given on the command line, the path is adjusted by crawling -up from the given file or package to the nearest directory that does not -contain an ``__init__.py`` or ``__init__.pyi`` file. If the given path -is relative, it will only crawl as far as the current working directory. - Second, mypy searches for stub files in addition to regular Python files and packages. The rules for searching for a module ``foo`` are as follows: From 01c90a77a8a7da0229a78fbe37d2ecb5998722ff Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Mon, 18 Jan 2021 11:35:28 -0800 Subject: [PATCH 2/4] edits --- docs/source/command_line.rst | 7 ++-- docs/source/running_mypy.rst | 73 +++++++++++++++++++----------------- 2 files changed, 43 insertions(+), 37 deletions(-) diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index 91f91bd9bbd4..68e6411bdef9 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -110,9 +110,10 @@ imports. passed on the command line, the ``MYPYPATH`` environment variable, and the :confval:`mypy_path` config option. - This flag also affects how mypy determines fully qualified module names for - files, modules and packages explicitly passed on the command line. See - :ref:`Mapping file paths to modules ` for details. + This flag affects how mypy finds modules and packages explicitly passed on + the command line. It also affects how mypy determines fully qualified module + names for files passed on the command line. See :ref:`Mapping file paths to + modules ` for details. .. option:: --explicit-package-bases diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 32060d4ba0d3..84a7b22d3f0e 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -366,40 +366,45 @@ How mypy determines fully qualified module names depends on if the options :option:`--namespace-packages ` and :option:`--explicit-package-bases ` are set. -First, if :option:`--namespace-packages ` is off, -mypy will rely solely upon the presence of ``__init__.py[i]`` files to determine -the fully qualified module name. That is, mypy will crawl up the directory tree -for as long as it continues to find ``__init__.py`` (or ``__init__.pyi``) files. - -Second, if :option:`--namespace-packages ` is on, but -:option:`--explicit-package-bases ` is off, mypy -will allow for the possibility that directories without ``__init__.py[i]`` are -packages. Specifically, mypy will look at all parent directories of the file and -use the location of the highest ``__init__.py[i]`` in the directory tree to -determine the top-level package. - -For example, say your directory tree consists solely of ``pkg/__init__.py`` and -``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified module -name, mypy will look at ``pkg/__init__.py`` and conclude that the associated -module name is ``pkg.a.b.c.d.mod``. - -You'll notice that that method still relies on ``__init__.py``. If you can't put -an ``__init__.py`` in your top-level package, but still wish to pass paths (as -opposed to packages or modules using the ``-p`` or ``-m`` flags), -:option:`--explicit-package-bases ` provides a -solution. - -Third, with :option:`--explicit-package-bases `, -mypy will locate the nearest parent directory that is a member of the -``MYPYPATH`` environment variable, the :confval:`mypy_path` config or is the -current working directory. mypy will then use the relative path to determine the -fully qualified module name. - -For example, say your directory tree consists solely of -``src/namespace_pkg/mod.py``. If you run the command following command, mypy -will correctly associate ``mod.py`` with ``namespace_pkg.mod``:: - - $ MYPYPATH=src mypy --namespace-packages --explicit-package-bases . +1. If :option:`--namespace-packages ` is off, + mypy will rely solely upon the presence of ``__init__.py[i]`` files to + determine the fully qualified module name. That is, mypy will crawl up the + directory tree for as long as it continues to find ``__init__.py`` (or + ``__init__.pyi``) files. + + For example, if your directory tree consists of ``pkg/subpkg/mod.py``, mypy + would require ``pkg/__init__.py`` and ``pkg/subpkg/__init__.py`` to exist in + order correctly associate ``mod.py`` with ``pkg.subpkg.mod`` + +2. If :option:`--namespace-packages ` is on, but + :option:`--explicit-package-bases ` is off, + mypy will allow for the possibility that directories without + ``__init__.py[i]`` are packages. Specifically, mypy will look at all parent + directories of the file and use the location of the highest + ``__init__.py[i]`` in the directory tree to determine the top-level package. + + For example, say your directory tree consists solely of ``pkg/__init__.py`` + and ``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified + module name, mypy will look at ``pkg/__init__.py`` and conclude that the + associated module name is ``pkg.a.b.c.d.mod``. + +3. You'll notice that the above case still relies on ``__init__.py``. If + you can't put an ``__init__.py`` in your top-level package, but still wish to + pass paths (as opposed to packages or modules using the ``-p`` or ``-m`` + flags), :option:`--explicit-package-bases ` + provides a solution. + + With :option:`--explicit-package-bases `, mypy + will locate the nearest parent directory that is a member of the ``MYPYPATH`` + environment variable, the :confval:`mypy_path` config or is the current + working directory. mypy will then use the relative path to determine the + fully qualified module name. + + For example, say your directory tree consists solely of + ``src/namespace_pkg/mod.py``. If you run the command following command, mypy + will correctly associate ``mod.py`` with ``namespace_pkg.mod``:: + + $ MYPYPATH=src mypy --namespace-packages --explicit-package-bases . If you pass a file not ending in ``.py[i]``, the module name assumed is ``__main__`` (matching the behavior of the Python interpreter), unless From c2ec15eb55e045686c334bb47913a9b16f031814 Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Mon, 18 Jan 2021 11:40:02 -0800 Subject: [PATCH 3/4] more edit --- docs/source/command_line.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index 68e6411bdef9..40df775742a6 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -117,11 +117,11 @@ imports. .. option:: --explicit-package-bases - This flag tells mypy to that the current directory, the ``MYPYPATH`` - environment variable, and the :confval:`mypy_path` config option as the - directories in which top-level packages are located. This option is only - useful in conjunction with :option:`--namespace-packages`. See :ref:`Mapping - file paths to modules ` for details. + This flag tells mypy that top-level packages will be based in either the + current directory, or a member of the ``MYPYPATH`` environment variable or + :confval:`mypy_path` config option. This option is only useful in + conjunction with :option:`--namespace-packages`. See :ref:`Mapping file + paths to modules ` for details. .. option:: --ignore-missing-imports From 44fb79a037b4691bbc18ac37dc592f10fd33ac3f Mon Sep 17 00:00:00 2001 From: hauntsaninja <> Date: Tue, 19 Jan 2021 10:35:55 -0800 Subject: [PATCH 4/4] code review --- docs/source/running_mypy.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 84a7b22d3f0e..866b57cf91df 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -352,15 +352,15 @@ is by providing mypy a list of paths. For example:: This section describes how exactly mypy maps the provided paths to modules to type check. -- mypy will check all paths provided that correspond to files. +- Mypy will check all paths provided that correspond to files. -- mypy will recursively discover and check all files ending in ``.py`` or +- Mypy will recursively discover and check all files ending in ``.py`` or ``.pyi`` in directory paths provided. - For each file to be checked, mypy will attempt to associate the file (e.g. ``project/foo/bar/baz.py``) with a fully qualified module name (e.g. ``foo.bar.baz``). The directory the package is in (``project``) is then - add to mypy's module search paths. + added to mypy's module search paths. How mypy determines fully qualified module names depends on if the options :option:`--namespace-packages ` and