[BUG] Realtive import of nonexistent package with same name throws RecursionError

[0x2b3bfa0]

setuptools version

setuptools 64.0.0

Python version

Python 3.10

OS

Ubuntu 22.04

Additional environment information

Seen in the wild at https://github.com/AMYPAD/CuVec, had to pin setuptools in AMYPAD/CuVec#25 to fix this issue.

Description

Introduced with #3488, throws RecursionError instead of ModuleNotFoundError when importing an editable package with an __init__.py containing an import in the form from .example import anything where example is also the package name.

Expected behavior

In this specific case, the expected outcome is ModuleNotFoundError: No module named 'test.test' like when pinning setuptools in the minimal reproducible example below to any versions lower than 64.

$ docker build .
[+] Building 7.3s (7/7) FINISHED                                                                                                                                                                                                                                                                     
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                            0.1s
 => => transferring dockerfile: 337B                                                                                                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                               0.2s
 => => transferring context: 2B                                                                                                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/python:3                                                                                                                                                                                                                                     0.9s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                                                                                                                   0.0s
 => CACHED [1/3] FROM docker.io/library/python:3@sha256:cbf49327fae903d64ab28251912fc00faea2c1baee493d347a07973a2cb50f98                                                                                                                                                                        0.0s
 => [2/3] RUN mkdir test && echo 'build-system.requires = ["setuptools<64"]' >pyproject.toml && echo 'from distutils.core import setup; setup(packages=["test"])' >setup.py && echo 'from .test import nonexistent' >test/__init__.py                                                           0.6s
 => ERROR [3/3] RUN pip install -e . && python -c 'import test'                                                                                                                                                                                                                                 5.5s
------                                                                                                                                                                                                                                                                                               
 > [3/3] RUN pip install -e . && python -c 'import test':                                                                                                                                                                                                                                            
#7 2.005 Obtaining file:///                                                                                                                                                                                                                                                                          
#7 2.008   Installing build dependencies: started                                                                                                                                                                                                                                                    
#7 3.556   Installing build dependencies: finished with status 'done'                                                                                                                                                                                                                                
#7 3.559   WARNING: Missing build requirements in pyproject.toml for file:///.                                                                                                                                                                                                                       
#7 3.559   WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
#7 3.560   Checking if build backend supports build_editable: started
#7 3.790   Checking if build backend supports build_editable: finished with status 'done'
#7 3.791   Getting requirements to build wheel: started
#7 4.020   Getting requirements to build wheel: finished with status 'done'
#7 4.022   Installing backend dependencies: started
#7 4.715   Installing backend dependencies: finished with status 'done'
#7 4.717   Preparing metadata (pyproject.toml): started
#7 4.967   Preparing metadata (pyproject.toml): finished with status 'done'
#7 5.007 Installing collected packages: test
#7 5.008   Running setup.py develop for test
#7 5.291 Successfully installed test-0.0.0
#7 5.291 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#7 5.296 
#7 5.296 [notice] A new release of pip available: 22.2.1 -> 22.2.2
#7 5.296 [notice] To update, run: pip install --upgrade pip
#7 5.409 Traceback (most recent call last):
#7 5.409   File "<string>", line 1, in <module>
#7 5.409   File "/test/__init__.py", line 1, in <module>
#7 5.410     from .test import nonexistent
#7 5.410 ModuleNotFoundError: No module named 'test.test'
------
executor failed running [/bin/sh -c pip install -e . && python -c 'import test']: exit code: 1

How to Reproduce

Create a file named Dockerfile with the contents below and run docker build . to reproduce the issue.

Dockerfile

FROM python:3

RUN mkdir test\
 && echo 'build-system.requires = ["setuptools==64.0.0"]' >pyproject.toml\
 && echo 'from distutils.core import setup; setup(packages=["test"])' >setup.py\
 && echo 'from .test import nonexistent' >test/__init__.py

RUN pip install -e .\
 && python -c 'import test'

Output

$ docker build .
[+] Building 7.5s (6/6) FINISHED                                                                                                                                                                                                                                                                     
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                            0.7s
 => => transferring dockerfile: 38B                                                                                                                                                                                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                               0.8s
 => => transferring context: 2B                                                                                                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/python:3                                                                                                                                                                                                                                     0.7s
 => [1/3] FROM docker.io/library/python:3@sha256:cbf49327fae903d64ab28251912fc00faea2c1baee493d347a07973a2cb50f98                                                                                                                                                                               0.0s
 => CACHED [2/3] RUN mkdir test && echo 'build-system.requires = ["setuptools==64.0.0"]' >pyproject.toml && echo 'from distutils.core import setup; setup(packages=["test"])' >setup.py && echo 'from .test import nonexistent' >test/__init__.py                                               0.0s
 => ERROR [3/3] RUN pip install -e . && python -c 'import test'                                                                                                                                                                                                                                 6.0s
------                                                                                                                                                                                                                                                                                               
 > [3/3] RUN pip install -e . && python -c 'import test':                                                                                                                                                                                                                                            
#6 1.921 Obtaining file:///                                                                                                                                                                                                                                                                          
#6 1.923   Installing build dependencies: started                                                                                                                                                                                                                                                    
#6 3.514   Installing build dependencies: finished with status 'done'                                                                                                                                                                                                                                
#6 3.517   WARNING: Missing build requirements in pyproject.toml for file:///.                                                                                                                                                                                                                       
#6 3.517   WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
#6 3.518   Checking if build backend supports build_editable: started
#6 3.755   Checking if build backend supports build_editable: finished with status 'done'
#6 3.756   Getting requirements to build editable: started
#6 3.995   Getting requirements to build editable: finished with status 'done'
#6 3.997   Installing backend dependencies: started
#6 4.945   Installing backend dependencies: finished with status 'done'
#6 4.946   Preparing editable metadata (pyproject.toml): started
#6 5.196   Preparing editable metadata (pyproject.toml): finished with status 'done'
#6 5.201 Building wheels for collected packages: test
#6 5.202   Building editable for test (pyproject.toml): started
#6 5.521   Building editable for test (pyproject.toml): finished with status 'done'
#6 5.522   Created wheel for test: filename=test-0.0.0-0.editable-cp310-cp310-linux_x86_64.whl size=2179 sha256=f590bda7d3eba788bb56059a8bf19d008194c7df0d0ac2c7ef90286629c3c38a
#6 5.523   Stored in directory: /tmp/pip-ephem-wheel-cache-rpsc66p0/wheels/31/76/a9/e09ab97e8ad0c49e207a3bddaf35a323dc6d5d03bb95d5511f
#6 5.527 Successfully built test
#6 5.566 Installing collected packages: test
#6 5.625 Successfully installed test-0.0.0
#6 5.626 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#6 5.631 
#6 5.631 [notice] A new release of pip available: 22.2.1 -> 22.2.2
#6 5.631 [notice] To update, run: pip install --upgrade pip
#6 5.795 Traceback (most recent call last):
#6 5.795   File "/usr/local/lib/python3.10/pathlib.py", line 625, in __str__
#6 5.795     return self._str
#6 5.795 AttributeError: 'PosixPath' object has no attribute '_str'
#6 5.795 
#6 5.795 During handling of the above exception, another exception occurred:
#6 5.795 
#6 5.795 Traceback (most recent call last):
#6 5.795   File "<string>", line 1, in <module>
#6 5.795   File "/test/__init__.py", line 1, in <module>
#6 5.796     from .test import nonexistent
#6 5.796   File "/test/__init__.py", line 1, in <module>
#6 5.796     from .test import nonexistent
#6 5.796   File "/test/__init__.py", line 1, in <module>
#6 5.796     from .test import nonexistent

[OMITTED ~270 SIMILAR LINES]

#6 5.805   File "/test/__init__.py", line 1, in <module>
#6 5.805     from .test import nonexistent
#6 5.805   File "/test/__init__.py", line 1, in <module>
#6 5.805     from .test import nonexistent
#6 5.805   File "/test/__init__.py", line 1, in <module>
#6 5.805     from .test import nonexistent
#6 5.805   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
#6 5.805   File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
#6 5.806   File "<frozen importlib._bootstrap>", line 945, in _find_spec
#6 5.806   File "/usr/local/lib/python3.10/site-packages/__editable___test_0_0_0_finder.py", line 19, in find_spec
#6 5.806     return cls._find_spec(fullname, Path(pkg_path, *rest))
#6 5.806   File "/usr/local/lib/python3.10/site-packages/__editable___test_0_0_0_finder.py", line 28, in _find_spec
#6 5.806     if candidate.exists():
#6 5.806   File "/usr/local/lib/python3.10/pathlib.py", line 1290, in exists
#6 5.806     self.stat()
#6 5.806   File "/usr/local/lib/python3.10/pathlib.py", line 1097, in stat
#6 5.806     return self._accessor.stat(self, follow_symlinks=follow_symlinks)
#6 5.806   File "/usr/local/lib/python3.10/pathlib.py", line 632, in __fspath__
#6 5.806     return str(self)
#6 5.806   File "/usr/local/lib/python3.10/pathlib.py", line 627, in __str__
#6 5.807     self._str = self._format_parsed_parts(self._drv, self._root,
#6 5.807 RecursionError: maximum recursion depth exceeded
------
executor failed running [/bin/sh -c pip install -e . && python -c 'import test']: exit code: 1

[abravalheri]

Hi @0x2b3bfa0, thank you very much for bringing this issue for discussion.

In your test/__init__.py file you are trying to do a relative import of a module that don't exist...
That is the main reason for the error... Although I have to investigate better the cause of the recursion.

I can demonstrate an error with setuptools v63.4.3, which was the last version of setuptools without PEP 660 support:

FROM python:3

RUN mkdir test\
 && echo 'build-system.requires = ["setuptools==63.4.3"]' >pyproject.toml\
 && echo 'from distutils.core import setup; setup(packages=["test"])' >setup.py\
 && echo 'from .test import nonexistent' >test/__init__.py

RUN pip install -e .\
 && python -c 'import test'

[+] Building 4.9s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 343B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/python:3                                                        0.7s
 => [1/3] FROM docker.io/library/python:3@sha256:cbf49327fae903d64ab28251912fc00faea2c1baee493d347a07973a2cb50f98  0.0s
 => CACHED [2/3] RUN mkdir test && echo 'build-system.requires = ["setuptools==63.4.3"]' >pyproject.toml && echo   0.0s
 => ERROR [3/3] RUN pip install -e . && python -c 'import test'                                                    4.1s
------
 > [3/3] RUN pip install -e . && python -c 'import test':
#6 1.317 Obtaining file:///
#6 1.320   Installing build dependencies: started
#6 2.738   Installing build dependencies: finished with status 'done'
#6 2.740   WARNING: Missing build requirements in pyproject.toml for file:///.
#6 2.740   WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
#6 2.741   Checking if build backend supports build_editable: started
#6 2.893   Checking if build backend supports build_editable: finished with status 'done'
#6 2.894   Getting requirements to build wheel: started
#6 3.041   Getting requirements to build wheel: finished with status 'done'
#6 3.043   Installing backend dependencies: started
#6 3.587   Installing backend dependencies: finished with status 'done'
#6 3.588   Preparing metadata (pyproject.toml): started
#6 3.760   Preparing metadata (pyproject.toml): finished with status 'done'
#6 3.794 Installing collected packages: test
#6 3.795   Running setup.py develop for test
#6 3.979 Successfully installed test-0.0.0
#6 3.980 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#6 3.983
#6 3.983 [notice] A new release of pip available: 22.2.1 -> 22.2.2
#6 3.983 [notice] To update, run: pip install --upgrade pip
#6 4.042 Traceback (most recent call last):
#6 4.042   File "<string>", line 1, in <module>
#6 4.042   File "/test/__init__.py", line 1, in <module>
#6 4.042     from .test import nonexistent
#6 4.042 ModuleNotFoundError: No module named 'test.test'
------
executor failed running [/bin/sh -c pip install -e . && python -c 'import test']: exit code: 1

On the other hand, I can verify version 65.1.0 working normally (the latest version of setuptools, which includes PEP 660 support), by tweaking a little bit the Dockerfile:

FROM python:3

WORKDIR /tmp
# Let's avoid naming the package `test`,
# because there is a `test` in Python's stdlib...
RUN mkdir -p /tmp/pkg_root/pkg \
 && echo 'build-system.requires = ["setuptools==65.1.0"]' > /tmp/pkg_root/pyproject.toml \
 && echo 'from distutils.core import setup; setup(packages=["pkg"])' > /tmp/pkg_root/setup.py \
 && echo 'from . import othermodule' > /tmp/pkg_root/pkg/__init__.py \
 && echo 'print("hello world")' > /tmp/pkg_root/pkg/othermodule.py

RUN pip install -e /tmp/pkg_root
WORKDIR /var
# We change directories here to avoid the side-effect
# of CWD being accidentally added to `sys.path`...
RUN python -c 'import pkg; print(pkg.__path__)'

This Dockerfile is able to build normally. I can also see the import statement working normally by doing the following:

docker build -t testcase .
docker run --rm -it testcase /bin/bash

$ docker build -t testcase .
[+] Building 8.6s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 731B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/python:3                                                        2.9s
 => [1/6] FROM docker.io/library/python:3@sha256:cbf49327fae903d64ab28251912fc00faea2c1baee493d347a07973a2cb50f98  0.0s
 => CACHED [2/6] WORKDIR /tmp                                                                                      0.0s
 => [3/6] RUN mkdir -p /tmp/pkg_root/pkg  && echo 'build-system.requires = ["setuptools==65.1.0"]' > /tmp/pkg_roo  0.4s
 => [4/6] RUN pip install -e /tmp/pkg_root                                                                         4.5s
 => [5/6] WORKDIR /var                                                                                             0.0s
 => [6/6] RUN python -c 'import pkg; print(pkg.__path__)'                                                          0.5s
 => exporting to image                                                                                             0.1s
 => => exporting layers                                                                                            0.1s
 => => writing image sha256:e463267a33fffafbb2c1ec432744b934a8e16e50a2f1166032e251729628c0a4                       0.0s
 => => naming to docker.io/library/testcase                                                                        0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

$ docker run --rm -it testcase /bin/bash
root@6f9db6fe1c05:/var# python -c 'import pkg; print(pkg.__path__)'
hello world
['/tmp/pkg_root/pkg']

[0x2b3bfa0]

Hi @abravalheri! Thank you ver much for the detailed response.

I can demonstrate a very similar error with setuptools v63.4.3, which was the last version of setuptools without PEP 660 support

As pointed out in the “expected behavior” section of the issue, the error is very similar, but is not the same: the RecursionError only happens on setuptools v64 and later.

Still, this issue turned out to be an x-y problem; I tried to demonstrate that packages with native code were causing this issue, and realized that it wasn't the case. 🙈

FROM python:3

RUN echo '#include <Python.h>' > test.c\
 && echo 'PyMODINIT_FUNC PyInit_test() { static struct PyModuleDef test = {PyModuleDef_HEAD_INIT, "test"}; PyObject* module = PyModule_Create(&test); PyModule_AddStringConstant(module, "variable", "value"); return module; }' >> test.c

RUN mkdir test\
 && echo 'build-system.requires = ["setuptools==64.0.0"]' > pyproject.toml\
 && echo 'from setuptools import Extension, setup; setup(packages=["test"], ext_modules=[Extension("test.test", sources=["test.c"])])' > setup.py\
 && echo 'from .test import variable' > test/__init__.py

RUN pip install -e .\
 && python -c 'import test; print(test.variable)'

[0x2b3bfa0]

Note: feel free to reopen if throwing RecursionError instead of ModuleNotFoundError can be considered a valid issue.

[abravalheri]

Thank you very much @0x2b3bfa0, I think I have captured the recursion problem in a test case and I am working on a fix.

This issue report was very helpful. Thank you very much!

[abravalheri]

@0x2b3bfa0 is there any error happening with the second version of the Dockerfile in #3550 (comment)?

I tried to build it in my machine and it builds correctly. But I have not tried to run anything in the container...

[0x2b3bfa0]

No, the example in #3550 (comment) works as intended: I tried to prove the opposite, and failed to do so.

I'll have to keep searching for the cause of the “Y” issue. 😅

[0x2b3bfa0]

Reopening so it's automatically closed when merging #3551 🎉

[casperdcl]

Unfortunately #3551 still doesn't fix editable installs. https://github.com/AMYPAD/CuVec/runs/7937211204?check_suite_focus=true fails on setuptools==65.1.1, with the "error":

ModuleNotFoundError: No module named 'cuvec.cuvec'

However the "error" is not actually accurate - there is indeed a module called cuvec.cuvec, which imports successfully on setuptools<64

[casperdcl]

Ah, perhaps the "real" issue is an upstream package not supporting PEP-660 and therefore cuvec.cuvec no longer exists... scikit-build/scikit-build#740