Maximum recursion bug

[scales11]

hello all.

I am trying to create an executable from a simple python script containing the pandas library, but I think that I have encountered a bug.

If I try to make an executable using a python script:

Import pandas as pd
print("hello")

If use pyinstaller:
pyinstaller --onefile test_script.py

I get the following:
"Recursion error: maximum recursion depth exceeded"

I have tried adding lines such as "sys.setrecursionlimit(5000)" with numbers up to 100000 but the issue remains.

Can anyone confirm the bug or offer some advice?

I also tried the suggestions mentioned here, but did not have any luck.
#3766

Thanks

[JackYangzg]

I think you can try to set hiddenimports include 'cython'

[scales11]

I appologize, I am still new to python. Can you provide an example?

[scales11]

After using debug log trace it looks like I have the issue with modulegraph.py
#2919

How can I ignore that module or make it work with Winpython 3.5.4.1?

Thanks,

[AgentChezni]

I'm also having this issue -- seemingly related to modulegraph.py through urllib3, which in turn is through pandas. Removing the pandas import "fixes" the problem, but of course that's not much of a solution, since I need the pandas module for the script.

[agronholm]

Seems like v4.0 did not fix this. I ran into this with the pydantic import.

[agronholm]

Actually pandas works just fine, pydantic does not. @scales11, @AgentChezni are you still having trouble?

[rokm]

I can confirm that maximum recursion error occurs with pydantic on Fedora 32 with python 3.8.5, clean venv, and pydantic 1.6.1 from pip.

I'll take a look and see if I can track down the cause.

[agronholm]

I opened a new issue already: #5131

[agronholm]

Are there any other libraries in existence that have a C extension module for __init__?

[agronholm]

If I delete pydantic/__init__.cpython-38-x86_64-linux-gnu.so, the operation completes successfully.

[agronholm]

I compared the process to a version where I import pandas instead. This is where they diverge:

2284 TRACE: load __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pandas'] 
2284 TRACE: load_module 'pandas' 'fp' '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pandas/__init__.py' 1

vs.

2426 TRACE: load __init__ for ['/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic'] 
2426 TRACE: load_module 'pydantic' None '/home/alex/workspace/cunit/venv/lib64/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so' 5

I modified the debug logging to output the typ variable in _load_module(). In the pandas version, this is 1, meaning imp.PY_SOURCE. In the pydantic version, it's 5, meaning imp.PKG_DIRECTORY.

[agronholm]

This pydantic error has already been reported: #4346

[htgoebel]

@agronholm First of all, many thanks for tracking this down to pydantic as an example. Based on this I was able to build test-cases replicating the recursion. (In fact these test-cases are easy, now that we know where to look at.) See 0f635a6.

@Legorooj pytest tests/unit/test_modulegraph_more.py -k package_init_is_extension - sorry, this should have not be gone to develop branch. Looks like I pushed wrongly.Thanks for reverting.

The root cause for this issue seems to be that modulegraph "redirects" packages it its __init__ file - this is a behaviour which is striking me since long, but it is build somehow deep into modulegraph. You can observe this easily by setting a breakpoint to https://github.com/pyinstaller/pyinstaller/blob/v4.0/PyInstaller/lib/modulegraph/modulegraph.py#L3091: This will be called twice for pytantic: Once for the package and once for the __init__ file, both as a ExtensionFileLoader.

I hacked a prototype, using the Loader classes, which stops this recursion error. You can find it at https://github.com/htgoebel/pyinstaller/tree/issue-4158-pydantic

[ENDERZOMBI102]

that branch doesn't exist anymore : (
how to do without it? the problem was fixed?

[htgoebel]

The fix is contained in PyInstaller v4.1.