You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue is a bit easier to spot now that we divert built-in extensions into lib-dynload directory: when building a onedir application on linux and macOS, we end up with two files in the sys._MEIPASS with oddly mangled filename/path: _struct/cpython-39-x86_64-linux-gnu/sotruct.cpython-39-x86_64-linux-gnu.so zlib/cpython-39-x86_64-linux-gnu/soib.cpython-39-x86_64-linux-gnu.so.
It turns out that this is caused by these two extension modules being added to list of dependencies early in the build chain (they are listed among bootstrap modules, probably just in case) and consequently ending up being processed by add_suffix_to_extensions() twice. The reconstructed chain of events is as follows:
the zlib and _struct extensions are explicitly returned among bootstrap modules by depend.analysis.get_bootstrap_modules(). These are added to PYZ.dependencies.
the pyz object is passed over to an EXE, whose TOC is extended with those pyz dependencies. At this point, zlib and _struct entries are still intact.
the EXE then instantiates a PKG, and passes it its TOC. PKG.assemble() performs add_suffix_to_extensions() on the TOC - at this point, the entries are turned into '_struct.cpython-39-x86_64-linux-gnu.so' and 'zlib.cpython-39-x86_64-linux-gnu.so', respectively.
but in onedir mode, extensions are ignored by PKG, and are instead appended to dependencies.
the EXE then takes PKG.dependencies, and makes them its own.
then, COLLECT is instantiated, and is passed the exe instance. It takes the exe's dependencies, and merges them into its own TOC.
COLLECT.assemble() performs add_suffix_to_extensions() on its TOC, and at this point, the offending two entries have been processed twice, and are now mangled into: _struct/cpython-39-x86_64-linux-gnu/sotruct.cpython-39-x86_64-linux-gnu.so
and zlib/cpython-39-x86_64-linux-gnu/soib.cpython-39-x86_64-linux-gnu.so.
So either the assembly pipeline should avoid repeatedly processing entries with add_suffix_to_extensions, or add_suffix_to_extensions should verify that entries do not already have suffix (which is probably easier to do).
In addition, those modules should be explicitly diverted into lib-dynload to follow the behavior from #5604 and avoid creating useless copies.
The text was updated successfully, but these errors were encountered:
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in pyinstaller#5851).
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in pyinstaller#5851).
rokm
added a commit
to rokm/pyinstaller
that referenced
this issue
May 23, 2021
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in pyinstaller#5851).
Detect if EXTENSION entry has already been processed and avoid
processing it again, which would mangle the filename due to
replacement of dots in suffix with os.sep (resulting in #5851).
This issue is a bit easier to spot now that we divert built-in extensions into
lib-dynload
directory: when building aonedir
application on linux and macOS, we end up with two files in thesys._MEIPASS
with oddly mangled filename/path:_struct/cpython-39-x86_64-linux-gnu/sotruct.cpython-39-x86_64-linux-gnu.so
zlib/cpython-39-x86_64-linux-gnu/soib.cpython-39-x86_64-linux-gnu.so
.It turns out that this is caused by these two extension modules being added to list of dependencies early in the build chain (they are listed among bootstrap modules, probably just in case) and consequently ending up being processed by
add_suffix_to_extensions()
twice. The reconstructed chain of events is as follows:the
zlib
and_struct
extensions are explicitly returned among bootstrap modules bydepend.analysis.get_bootstrap_modules()
. These are added toPYZ.dependencies
.the
pyz
object is passed over to anEXE
, whose TOC is extended with thosepyz
dependencies. At this point,zlib
and_struct
entries are still intact.the
EXE
then instantiates aPKG
, and passes it its TOC.PKG.assemble()
performsadd_suffix_to_extensions()
on the TOC - at this point, the entries are turned into '_struct.cpython-39-x86_64-linux-gnu.so' and 'zlib.cpython-39-x86_64-linux-gnu.so', respectively.but in
onedir
mode, extensions are ignored by PKG, and are instead appended todependencies
.the EXE then takes
PKG.dependencies
, and makes them its own.then, COLLECT is instantiated, and is passed the
exe
instance. It takes theexe
's dependencies, and merges them into its own TOC.COLLECT.assemble()
performsadd_suffix_to_extensions()
on its TOC, and at this point, the offending two entries have been processed twice, and are now mangled into:_struct/cpython-39-x86_64-linux-gnu/sotruct.cpython-39-x86_64-linux-gnu.so
and
zlib/cpython-39-x86_64-linux-gnu/soib.cpython-39-x86_64-linux-gnu.so
.So either the assembly pipeline should avoid repeatedly processing entries with
add_suffix_to_extensions
, oradd_suffix_to_extensions
should verify that entries do not already have suffix (which is probably easier to do).In addition, those modules should be explicitly diverted into
lib-dynload
to follow the behavior from #5604 and avoid creating useless copies.The text was updated successfully, but these errors were encountered: