Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyinstaller with pydantic causes "maximum recursion depth exceeded" #4346

Closed
sindhukothe opened this issue Aug 1, 2019 · 18 comments · May be fixed by #4950
Closed

pyinstaller with pydantic causes "maximum recursion depth exceeded" #4346

sindhukothe opened this issue Aug 1, 2019 · 18 comments · May be fixed by #4950

Comments

@sindhukothe
Copy link

sindhukothe commented Aug 1, 2019

I have the following environment:
Python 3.7.3
pipenv, version 2018.11.26
pyinstaller Version 3.5
pydantic.
When I import pydnatic into my application and try to freeze it with pyinstaller, I get the following error:
RecursionError: maximum recursion depth exceeded while calling a Python object
The partial stack trace is as follows.
........
ile "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 243, in init
self.postinit()
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 158, in postinit
self.assemble()
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 445, in assemble
priority_scripts.append(self.graph.run_script(script))
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/depend/analysis.py", line 201, in run_script
self._top_script_node = super(PyiModuleGraph, self).run_script(pathname)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 1401, in run_script
self._scan_code(m, co, co_ast)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2636, in _scan_code
self._process_imports(module)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2835, in _process_imports
target_module = self._safe_import_hook(*import_info, **kwargs)[0]
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2291, in _safe_import_hook
target_attr_names=None, level=level, edge_attr=edge_attr)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 1474, in import_hook
source_package, target_module_partname, level)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 1631, in _find_head_package
target_module_headname, target_package_name, source_package)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/depend/analysis.py", line 271, in _safe_import_module
module_basename, module_name, parent_package)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2048, in _safe_import_module
module_name, file_handle, pathname, metadata)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2086, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "/home/test/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2905, in _load_package

...... A few more hundred lines like this

Relevant snippet with -d all --log-level TRACE:
5624 INFO: Analyzing /home/nuviso/test/main.py
5626 TRACE: run_script '/home/nuviso/test/main.py'
5626 TRACE: addNode Script('/home/nuviso/test/main.py',)
5627 TRACE: createReference Script('/home/nuviso/test/main.py',) None
5627 TRACE: _safe_import_hook 'pydantic' Script('/home/nuviso/test/main.py',) ['BaseModel'] 0
5627 TRACE: _import_hook 'pydantic' Script('/home/nuviso/test/main.py',) Script('/home/nuviso/test/main.py',) 0
5627 TRACE: determine_parent Script('/home/nuviso/test/main.py',)
5627 TRACE: determine_parent -> None
5627 TRACE: find_head_package None 'pydantic' 0
5628 TRACE: safe_import_module 'pydantic' 'pydantic' None
5628 TRACE: _find_module_path <- 'pydantic' ['/home/nuviso/test', '/home/nuviso/test', '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/bin', '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python37.zip', '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7', '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7', '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages', '/home/nuviso/test', '/home/nuviso/test']
5628 TRACE: _find_module_path -> (None, '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so', ('', '', 5))
5628 TRACE: load_module 'pydantic' None '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so'
5628 TRACE: load_package 'pydantic' '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so' []
5630 TRACE: addNode Package('pydantic',)
5630 TRACE: find init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5630 TRACE: _find_module_path <- 'pydantic.init' ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5631 TRACE: _find_module_path -> (None, '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so', ('', '', 5))
5631 TRACE: load init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5631 TRACE: load_module 'pydantic' None '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so'
5631 TRACE: load_package 'pydantic' '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so' []
5633 TRACE: find init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5633 TRACE: _find_module_path <- 'pydantic.init' ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5633 TRACE: _find_module_path -> (None, '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so', ('', '', 5))
5633 TRACE: load init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5633 TRACE: load_module 'pydantic' None '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so'
5634 TRACE: load_package 'pydantic' '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so' []
5635 TRACE: find init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5635 TRACE: _find_module_path <- 'pydantic.init' ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5635 TRACE: _find_module_path -> (None, '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so', ('', '', 5))
5636 TRACE: load init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5636 TRACE: load_module 'pydantic' None '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so'
5636 TRACE: load_package 'pydantic' '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so' []
5637 TRACE: find init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5638 TRACE: _find_module_path <- 'pydantic.init' ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5638 TRACE: _find_module_path -> (None, '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so', ('', '', 5))
5638 TRACE: load init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5638 TRACE: load_module 'pydantic' None '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so'
5638 TRACE: load_package 'pydantic' '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so' []
5640 TRACE: find init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5641 TRACE: _find_module_path <- 'pydantic.init' ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5641 TRACE: _find_module_path -> (None, '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so', ('', '', 5))
5641 TRACE: load init for ['/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic']
5641 TRACE: load_module 'pydantic' None '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so'
5642 TRACE: load_package 'pydantic' '/home/nuviso/.local/share/virtualenvs/test-avG2MBOP/lib/python3.7/site-packages/pydantic/init.cpython-37m-x86_64-linux-gnu.so' []

@sindhukothe
Copy link
Author

Basically, the modulegraph.py is going into infinite loop with calls between _load_module() and _load_package()
The first line of init.py in pydantic is
from . import dataclasses
Not sure that would cause this infinite loop.
Thanks for your help.

@vinayak618
Copy link

Add below lines at the start spec file:
import sys
sys.setrecursionlimit(2000)

@sindhukothe
Copy link
Author

Add below lines at the start spec file:
import sys
sys.setrecursionlimit(2000)

I tried that. It does not help because this is a never ending infinite loop.

@Hanaasagi
Copy link

@sindhukothe
If you install pydantic from pypi, it was already compiled with cython. Maybe using original source will work.

SKIP_CYTHON=1 pip install git+https://github.com/samuelcolvin/pydantic.git@v0.32

Ref: https://pydantic-docs.helpmanual.io/#install

@sander76
Copy link

@Hanaasagi I have the same issue. Your solution seems to do the trick. Thanks !

I didn't know what to do with SKIP_CYTHON=1, but as I didn't have cython installed I just did the pip install from github and pyinstaller did the job well.

@htgoebel
Copy link
Member

Please help taming the "RecursionError:maximum recursion depth exceeded" errors, I was not yet able to reproduce this on my system, thus you need to provide data so I can try to fix the issue.

Please follow #4406, which (hopefully) provide the required information. Thanks!

@sindhukothe
Copy link
Author

@htgoebel Thanks for your reply. I am not able to reproduce this issue even in my main application as well. I followed your steps in #4406 as well, but not able to reproduce it.
The pydantic library has been updated from 0.31 to no 0.32. Not sure if that is fixed because of the upgrade. I will spend some more time later to see if I can reproduce it, but for now, I will close the issue.
(PS: there is a typo in #4406 (It should be main.py and not main.oy)

@htgoebel
Copy link
Member

@sindhukothe Thanks for your message, It would be really great if you could take your rime and reproduce this issue. It is plopping up every now and then and is hard to reproduce on other environments - let alone to fix :-)

(The typo I just fixed.)

@sindhukothe
Copy link
Author

sindhukothe commented Sep 12, 2019

I spent some more time on this and am able to reproduce this again. I sent you email with the details.
please note that I used pipenv instead of pip as prescribed in #4406
But I am positive it would not matter.
I tried pyinstaller versions of 3.5 as well as latest from github. Able to reproduce with either version

@sindhukothe sindhukothe reopened this Sep 12, 2019
@sindhukothe
Copy link
Author

I tried @Hanaasagi 's solution and it is working. Thank you!

@Legorooj
Copy link
Member

Legorooj commented Feb 24, 2020

Please use #4406 to follow this.

@Tracyrvp
Copy link

Tracyrvp commented Aug 12, 2020

Hi,does This problem fixed?

@Legorooj
Copy link
Member

Legorooj commented Aug 12, 2020

@Tracyrvp no - I'm in the process of patching it though. The fix will be out next release (sometime December probably - please note that's a guess).

This was referenced Aug 31, 2020
@nicobako
Copy link

nicobako commented Sep 3, 2020

@sindhukothe
If you install pydantic from pypi, it was already compiled with cython. Maybe using original source will work.

SKIP_CYTHON=1 pip install git+https://github.com/samuelcolvin/pydantic.git@v0.32

Ref: https://pydantic-docs.helpmanual.io/#install

This advice was wonderful!

Just a note that you could also type pip install pydantic --no-binary pydantic and achieve the same result while getting the latest pydantic.
https://pip.pypa.io/en/stable/reference/pip_install/#install-no-binary

@htgoebel
Copy link
Member

Pointed to this issue by #5131, I learnd: this one is a different class of recursion errors, as it is an infinite loop. (Whereas #4406 is about import-nesting too deep, but no loop.) Sorry for not understanding this issue earlier. Fix is in the pipeline.

htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change `_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 10, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 12, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Sep 12, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
@ENDERZOMBI102
Copy link

ENDERZOMBI102 commented Sep 24, 2020

i have this error too, i'm trying to setup appveyor but pyinstaller cuntinue to give me this error, i already tried sys.setrecursionlimit() but it didn't work, there's another solution in the works?
EDIT:
there's my installed packages:
future
markdown2
pypresence
requests
wxPython
psutil
pydantic
pyinstaller
pyinstaller-hooks-contrib

@mbollemeijer
Copy link

i have this error too, i'm trying to setup appveyor but pyinstaller cuntinue to give me this error, i already tried sys.setrecursionlimit() but it didn't work, there's another solution in the works?
EDIT:
there's my installed packages:
future
markdown2
pypresence
requests
wxPython
psutil
pydantic
pyinstaller
pyinstaller-hooks-contrib

I solved it by installing pyinstaller from @htgoebel fix
pip3 install git+https://github.com/htgoebel/pyinstaller@issue-4158-pydantic

phil65 pushed a commit to phil65/pyinstaller that referenced this issue Oct 1, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
phil65 pushed a commit to phil65/pyinstaller that referenced this issue Oct 1, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 11, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 11, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 18, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 18, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit to htgoebel/pyinstaller that referenced this issue Oct 18, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see pyinstaller#5131, pyinstaller#4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See pyinstaller#4406, closes pyinstaller#5131, pyinstaller#4346.
htgoebel added a commit that referenced this issue Oct 18, 2020
These tests check for "recursion too deep" caused by the `__init__` module
being an extension module.

See #5131, #4346.
htgoebel added a commit that referenced this issue Oct 18, 2020
modulegraph's heritage includes code still based on Python 2.x capabilities on
how to find a module and get its source or code. It also contained a anomaly
regarding packages and their ``__init__``-file: If a package was detected,
it's ``__init__`` file was loaded as a module. This, while being ugly, worked
in most cases, but failed if the ``__init__`` module is an extension
module (see #5131, #4346), ending in an infinite loop. This was caused by
modulegraph distinguishing between the package and its ``__init__`` module.

The solution is to switch to "modern" loaders, both being a loader for a
specific type of modules (source, extension, etc.) and having a package
characteristic (property ``is_package()``)

This commit does the following

- In ``_find_module_path()`` no longer return "metadata" but a loader. This
  also removed huge part of this function, making it much easier to understand.
  As a side effect, this asymmetric closing of a file in a completely other
  part of the code (``_safe_import_module``) could be removed.

- Change ``_load_module`` to use the loaders.

- Merge ``_load_package`` into `__load_module``, getting rid of the anomaly
  described above.

- Adjust the test-cases to the new behavior (esp. loader instead of
  metadata-tuple and filehandle)

Please note: Since we plan to change to modulegraph2 soon anyway, I did not
spend too much time on creating a clean solution.

See #4406, closes #5131, #4346.
@JustinYeoh93
Copy link

@sindhukothe
If you install pydantic from pypi, it was already compiled with cython. Maybe using original source will work.

SKIP_CYTHON=1 pip install git+https://github.com/samuelcolvin/pydantic.git@v0.32

Ref: https://pydantic-docs.helpmanual.io/#install

10/10. My unfamiliarity with Python have definitely cost me.
Thanks for the advice!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.