Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursion error with Pydantic #5131

Closed
1 of 6 tasks
agronholm opened this issue Aug 31, 2020 · 3 comments · Fixed by #5157
Closed
1 of 6 tasks

Recursion error with Pydantic #5131

agronholm opened this issue Aug 31, 2020 · 3 comments · Fixed by #5157

Comments

@agronholm
Copy link

+++ ONLY TEXT +++ DO NOT POST IMAGES +++

Description of the issue

Trying to run pyinstaller against a module that contains just import pydantic fails due to RecursionError: maximum recursion depth exceeded while calling a Python object.

Context information (for bug reports)

  • Output of pyinstaller --version: 4.0

  • Version of Python: 3.8

  • Platform: Fedora 32

  • Did you also try this on another platform? Does it work there? I didn't

  • try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

Make sure everything is packaged correctly

  • start with clean installation
  • use the latest development version
  • Run your frozen program from a command window (shell) — instead of double-clicking on it
  • Package your program in --onedir mode
  • Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
  • Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

import pydantic

Stacktrace / full error message

...
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2907, in _load_package
    self._load_module(fqname, fp, buf, stuff)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2907, in _load_package
    self._load_module(fqname, fp, buf, stuff)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2907, in _load_package
    self._load_module(fqname, fp, buf, stuff)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2088, in _load_module
    m = self._load_package(fqname, pathname, packagepath)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2880, in _load_package
    ns_pkgpath = _namespace_package_path(fqname, pkgpath or [], self.path)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 164, in _namespace_package_path
    working_set = pkg_resources.WorkingSet(path)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2065, in find_on_path
    for dist in factory(fullpath):
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2134, in distributions_from_metadata
    yield Distribution.from_location(
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2590, in from_location
    return cls(
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2994, in _reload_version
    md_version = self._get_version()
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2772, in _get_version
    version = _version_from_file(lines)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2556, in _version_from_file
    line = next(iter(version_lines), '')
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 2767, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1432, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1419, in get_metadata
    path = self._get_metadata_path(name)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1407, in _get_metadata_path
    return self._fn(self.egg_info, name)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1487, in _fn
    self._validate_resource_path(resource_name)
  File "/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pkg_resources/__init__.py", line 1547, in _validate_resource_path
    posixpath.isabs(path) or
  File "/usr/lib64/python3.8/posixpath.py", line 63, in isabs
    sep = _get_sep(s)
  File "/usr/lib64/python3.8/posixpath.py", line 42, in _get_sep
    if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object

Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs
for more about what would use to solve the issue.

@agronholm
Copy link
Author

This may be caused by pydantic having two files for the same module with the same base name. I'm investigating.

@agronholm
Copy link
Author

Here's when things start going wrong:

2297 TRACE: safe_import_module 'test' 'test' None
2297 TRACE: _find_module_path <- 'test' ['/home/alex/workspace/cunit/venv/bin', '/usr/lib64/python38.zip', '/usr/lib64/python3.8', '/usr/lib64/python3.8/lib-dynload', '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages', '/home/alex/workspace/cunit/venv/lib/python3.8/site-packages', '/home/alex/workspace/cunit', '/home/alex/workspace/cunit']
2297 TRACE: _find_module_path -> (<_io.StringIO object at 0x7f7c292d7f70>, '/home/alex/workspace/cunit/test.py', ('.py', 'r', 1))
2297 TRACE: load_module 'test' 'fp' '/home/alex/workspace/cunit/test.py'
2297 TRACE: addNode SourceModule('test',)
2297 TRACE: _safe_import_hook 'pydantic' SourceModule('test', '/home/alex/workspace/cunit/test.py') None 0
2297 TRACE: _import_hook 'pydantic' SourceModule('test', '/home/alex/workspace/cunit/test.py') SourceModule('test', '/home/alex/workspace/cunit/test.py') 0
2297 TRACE: determine_parent SourceModule('test', '/home/alex/workspace/cunit/test.py')
2297 TRACE: determine_parent -> None
2297 TRACE: find_head_package None 'pydantic' 0
2297 TRACE: safe_import_module 'pydantic' 'pydantic' None
2297 TRACE: _find_module_path <- 'pydantic' ['/home/alex/workspace/cunit/venv/bin', '/usr/lib64/python38.zip', '/usr/lib64/python3.8', '/usr/lib64/python3.8/lib-dynload', '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages', '/home/alex/workspace/cunit/venv/lib/python3.8/site-packages', '/home/alex/workspace/cunit', '/home/alex/workspace/cunit']
2297 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
2297 TRACE: load_module 'pydantic' None '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so'
2297 TRACE: load_package 'pydantic' '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so' []
2312 TRACE: addNode Package('pydantic',)
2312 TRACE: find __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2312 TRACE: _find_module_path <- 'pydantic.__init__' ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic']
2312 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
2312 TRACE: load __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2312 TRACE: load_module 'pydantic' None '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so'
2313 TRACE: load_package 'pydantic' '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so' []
2327 TRACE: find __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2327 TRACE: _find_module_path <- 'pydantic.__init__' ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic']
2327 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
2327 TRACE: load __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2327 TRACE: load_module 'pydantic' None '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so'
2327 TRACE: load_package 'pydantic' '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so' []
2342 TRACE: find __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2342 TRACE: _find_module_path <- 'pydantic.__init__' ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic']
2342 TRACE: _find_module_path -> (None, '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so', ('', '', 5))
...

@agronholm
Copy link
Author

Looks like this is a duplicate of #4346.

htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change `_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 12, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 12, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
phil65 pushed a commit to phil65/pyinstaller that referenced this issue Oct 1, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
phil65 pushed a commit to phil65/pyinstaller that referenced this issue Oct 1, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 11, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 11, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 18, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 18, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 18, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit that referenced this issue Oct 18, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See #5131, #4346.
htgoebel added a commit that referenced this issue Oct 18, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see #5131, #4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See #4406, closes #5131, #4346.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant