Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repeated processing of extension TOC entries with add_suffix_to_extensions #5851

Closed
rokm opened this issue May 20, 2021 · 0 comments · Fixed by #5852
Closed

Repeated processing of extension TOC entries with add_suffix_to_extensions #5851

rokm opened this issue May 20, 2021 · 0 comments · Fixed by #5852
Assignees
Labels

Comments

@rokm
Copy link
Member

rokm commented May 20, 2021

This issue is a bit easier to spot now that we divert built-in extensions into lib-dynload directory: when building a onedir application on linux and macOS, we end up with two files in the sys._MEIPASS with oddly mangled filename/path:
_struct/cpython-39-x86_64-linux-gnu/sotruct.cpython-39-x86_64-linux-gnu.so
zlib/cpython-39-x86_64-linux-gnu/soib.cpython-39-x86_64-linux-gnu.so.

It turns out that this is caused by these two extension modules being added to list of dependencies early in the build chain (they are listed among bootstrap modules, probably just in case) and consequently ending up being processed by add_suffix_to_extensions() twice. The reconstructed chain of events is as follows:

  • the zlib and _struct extensions are explicitly returned among bootstrap modules by depend.analysis.get_bootstrap_modules(). These are added to PYZ.dependencies.

  • the pyz object is passed over to an EXE, whose TOC is extended with those pyz dependencies. At this point, zlib and _struct entries are still intact.

  • the EXE then instantiates a PKG, and passes it its TOC. PKG.assemble() performs add_suffix_to_extensions() on the TOC - at this point, the entries are turned into '_struct.cpython-39-x86_64-linux-gnu.so' and 'zlib.cpython-39-x86_64-linux-gnu.so', respectively.

  • but in onedir mode, extensions are ignored by PKG, and are instead appended to dependencies.

  • the EXE then takes PKG.dependencies, and makes them its own.

  • then, COLLECT is instantiated, and is passed the exe instance. It takes the exe's dependencies, and merges them into its own TOC.

  • COLLECT.assemble() performs add_suffix_to_extensions() on its TOC, and at this point, the offending two entries have been processed twice, and are now mangled into:
    _struct/cpython-39-x86_64-linux-gnu/sotruct.cpython-39-x86_64-linux-gnu.so
    and
    zlib/cpython-39-x86_64-linux-gnu/soib.cpython-39-x86_64-linux-gnu.so.

So either the assembly pipeline should avoid repeatedly processing entries with add_suffix_to_extensions, or add_suffix_to_extensions should verify that entries do not already have suffix (which is probably easier to do).

In addition, those modules should be explicitly diverted into lib-dynload to follow the behavior from #5604 and avoid creating useless copies.

@rokm rokm self-assigned this May 20, 2021
@rokm rokm added the bug label May 20, 2021
rokm added a commit to rokm/pyinstaller that referenced this issue May 20, 2021
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in pyinstaller#5851).
rokm added a commit to rokm/pyinstaller that referenced this issue May 20, 2021
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in pyinstaller#5851).
rokm added a commit to rokm/pyinstaller that referenced this issue May 23, 2021
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in pyinstaller#5851).
bwoodsend pushed a commit that referenced this issue May 26, 2021
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in #5851).
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant