Duplicating binary files

[jawhien]

Description of the issue

Including binaries via "--add-binary" that are in subdirectories and require additional dependencies results in duplication of these dependencies in the application root directory, even though the dependencies are already added via "--add-binary".
This problem started in version 5.10.

Context information (for bug reports)

• Version of Python: 3.8.10

• Platform: Windows 10

• How you installed Python: python.org/downloads

• Did you also try this on another platform? Does it work there?
  Haven't tried it

• try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

• follow all the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

• start with clean installation
• use the latest development version
• Run your frozen program from a command window (shell) — instead of double-clicking on it
• Package your program in --onedir mode
• Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

(paste text here)
“Minimal“ means: remove everything from your code which is not relevant for this bug,
esp. don't use external programs, remote requests, etc.
A very good example is https://gist.github.com/ronen/024cdae9ff2d50488438. This one helped
us reproducing and fixing a quite complex problem within approx 1 hour.

Stacktrace / full error message

(paste text here)

Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs
for more about what would use to solve the issue.

[bwoodsend]

Can you give an example? Depending on whatever this binary is and how it finds its dependencies, there's a pretty good chance that it wouldn't be able to find the dependencies in subdirectories.

[rokm]

PyInstaller's binary dependency analysis has always collected dependencies into top-level application directory, since that's the only location that is always guaranteed to be in the library search path. Only in recent versions, we've begun changing that behavior for binaries that are located in python packages (since their code often depends on the exact location). But for binaries in an arbitrary location, the destination directory will be top-level directory, same as before.

The difference before and after 5.10 is #7522, which puts binaries specified via --add-binaries through binary dependency analysis.

That said, the binary parent path preservation could (and probably should) be extended a bit more; so that on Windows, if we find a dependency next to the analyzed binary, we reuse the analyzed binary's target location (since either its location was preserved, or was explicitly specified via --add-binary (and is not the top-level directory for whatever reason)).

[jawhien]

Well I don't know what's really going on. Now I’ll try to describe it in as much detail as possible.

My application has a tdlib module that contains a wrapper, as well as 4 binary files for its operation:

• libcrypto-3.dll
• libssl-3.dll
• tdjson.dll
• zlib1.dll

All this is located in the tdlib subdirectory and is loaded from there through the CDLL function of the ctypes module.
Then I add the binaries to the build, something like this:

--add-binary tdLib\tdjson.dll;tdLib --add-binary tdLib\libcrypto-3.dll;tdLib --add-binary tdLib\libssl-3.dll;tdLib --add-binary tdLib\zlib1.dll;tdLib

As a result of the build, the tdlib subdirectory contains all 4 specified binaries, but the following binaries are copied to the application root directory, or the "_internal" folder:

• libcrypto-3.dll
• libssl-3.dll
• zlib1.dll

And this happens with some other binaries in other submodules.

[rokm]

Yeah, the described behavior matches what I would expect, as long as the tdLib package is not installed in site-packages (if it were, then directory preservation would also cause binary dependency analysis to put those libs into sub-directory).

With latest PyInstaller versions, adding just --add-binary tdLib\tdjson.dll;tdLib should suffice, and while it would collect dependencies into top-level application directory, you would not have duplicates. (Even collecting tdjson.dll to top-level directory might work due to ctypes override that we have).

But to allow you to explicitly specify all binaries and avoid the duplication, the binary parent path preservation will need to be extended as I outlined earlier.

[jawhien]

But to allow you to explicitly specify all binaries and avoid the duplication, the binary parent path preservation will need to be extended as I outlined earlier.

Could you explain what this means? I don't quite understand what you mean.

[rokm]

But to allow you to explicitly specify all binaries and avoid the duplication, the binary parent path preservation will need to be extended as I outlined earlier.

Could you explain what this means? I don't quite understand what you mean.

It means I need to fix that part of PyInstaller code.

[jawhien]

Understood thanks:)