pyinstaller 4.1 executables have incorrect references to non-extant archives

[tfoote]

Description of the issue

Context information (for bug reports)

• Output of pyinstaller --version: 4.1 and latest (This is a regression since 4.0 which works)

• Version of Python: e.g. 3.7

• Platform: Debian Buster

• Did you also try this on another platform? It fails on Stretch too.

• try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

• follow all the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

• start with clean installation
• use the latest development version
• Run your frozen program from a command window (shell) — instead of double-clicking on it
• [] Package your program in --onedir mode
  • It's only one file using --onefile option
• Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• [] Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.
  • --debug does not appear to be an option pyinstaller will take

A minimal example program which shows the error

FROM python:3 as detector

RUN mkdir -p /tmp/distrovenv
RUN python3 -m venv /tmp/distrovenv
# It works with 4.0, does not work with 4.1 or the latest development archive
# RUN . /tmp/distrovenv/bin/activate && pip install distro pyinstaller==4.0 staticx
RUN . /tmp/distrovenv/bin/activate && pip install distro https://github.com/pyinstaller/pyinstaller/archive/develop.zip staticx
RUN apt-get update && apt-get install patchelf #needed for staticx

RUN echo 'import distro; import sys; output = distro.linux_distribution(); print(output) if output[0] else sys.exit(1)' > /tmp/distrovenv/detect_os.py
RUN . /tmp/distrovenv/bin/activate && pyinstaller --noupx --debug --onefile /tmp/distrovenv/detect_os.py

RUN . /tmp/distrovenv/bin/activate && staticx /dist/detect_os /dist/detect_os_static

CMD /dist/detect_os && /dist/detect_os_static

To build the image to reproduce:

docker build -t pyi_error .

Stacktrace / full error message

To run the results

docker run -ti --rm pyi_error

('Debian GNU/Linux', '10', 'buster')
[11] Cannot open self /tmp/staticx-GdKCGK/detect_os or archive /tmp/staticx-GdKCGK/detect_os.pkg

As you can see this works with the bare executable generated by pyinstaller but fails when it's converted to a static object. This was working until the 4.1 release.

There is an issue with a very similiar backtrace: #5257 but it has an associated fix #5258 which I believe is in the release.

This is explicitly being exposed by the interaction with staticx, and similar issues have been seen before: JonathonReinhart/staticx#71

But because this is a specific regression since the latest release here I'm filing the issue here.

I have worked around this issue on my project by pinning back to 4.0: osrf/rocker#106

[BoboTiG]

Maybe fixed by #5232?

[rokm]

@tfoote can you try with locally rebuilt PyInstaller's bootloader?

[rokm]

Hm, it looks like the pre-compiled bootloaders in the new release were built with PIE enabled. I.e.,

4.0: run: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=4485007e1df54173663f148d5840c963a7e868ee, stripped

vs.

4.1: run: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=3507aa01d32c34dc8e8c6462b764adb90a82768d, stripped

If I rebuild bootloader from develop with default gcc settings, staticx-processed program works correctly. If I rebuild bootloader with PIE enabled, it doesn't.

And it seems staticx behaves differently with non-PIE/PIE executable. With the latter, it appends stuff at the end of the executable, which causes the search for PyInstaller cookie ("MEI\014\013\012\013\016") fail (because it falls outside of the range of last 4096+8 bytes in the file, where we do the scan). This in turn causes error message that is a bit misleading:

[70035] PyInstaller Bootloader 3.x
[70035] LOADER: executable is /tmp/staticx-ipJDdj/program
[70035] LOADER: homepath is /tmp/staticx-ipJDdj
[70035] LOADER: _MEIPASS2 is NULL
[70035] LOADER: archivename is /tmp/staticx-ipJDdj/program
[70035] Loader: Cannot find cookie[70035] LOADER: archivename is /tmp/staticx-ipJDdj/program.pkg
[70035] LOADER: Cannot open archive: /tmp/staticx-ipJDdj/program.pkg
[70035] Cannot open self /tmp/staticx-ipJDdj/program or archive /tmp/staticx-ipJDdj/program.pkg

[tfoote]

@tfoote can you try with locally rebuilt PyInstaller's bootloader?

I tried building the bootloader this way and it didn't make a difference. The below is how I tested it.

FROM python:3 as detector

RUN mkdir -p /tmp/distrovenv
RUN python3 -m venv /tmp/distrovenv
# It works with 4.0, does not work with 4.1 or the latest development archive
# RUN . /tmp/distrovenv/bin/activate && pip install distro pyinstaller==4.0 staticx
RUN . /tmp/distrovenv/bin/activate && pip install distro https://github.com/pyinstaller/pyinstaller/archive/develop.zip staticx
RUN apt-get update && apt-get install patchelf #needed for staticx

RUN echo 'import distro; import sys; output = distro.linux_distribution(); print(output) if output[0] else sys.exit(1)' > /tmp/distrovenv/detect_os.py
RUN wget https://github.com/pyinstaller/pyinstaller/tarball/develop -O /tmp/pyinstaller.tar.gz
WORKDIR /tmp
RUN mkdir pyinstaller
RUN tar -xvzf /tmp/pyinstaller.tar.gz --strip 1 --directory /tmp/pyinstaller
WORKDIR /tmp/pyinstaller/bootloader
RUN python ./waf all


WORKDIR /
RUN . /tmp/distrovenv/bin/activate && pyinstaller --noupx --onefile /tmp/distrovenv/detect_os.py

RUN . /tmp/distrovenv/bin/activate && staticx /dist/detect_os /dist/detect_os_static

CMD /dist/detect_os && /dist/detect_os_static

[tfoote]

Maybe fixed by #5232?

Good eye @BoboTiG I confirmed that this PR does resolve this issue.

For reference I was testing against: a34a2c5

FROM python:3 as detector

RUN mkdir -p /tmp/distrovenv
RUN python3 -m venv /tmp/distrovenv
RUN . /tmp/distrovenv/bin/activate && pip install distro git+https://github.com/rokm/pyinstaller.git@bootloader-process-name staticx
RUN apt-get update && apt-get install patchelf #needed for staticx

RUN echo 'import distro; import sys; output = distro.linux_distribution(); print(output) if output[0] else sys.exit(1)' > /tmp/distrovenv/detect_os.py
RUN wget https://github.com/pyinstaller/pyinstaller/tarball/develop -O /tmp/pyinstaller.tar.gz


WORKDIR /
RUN . /tmp/distrovenv/bin/activate && pyinstaller --noupx --onefile /tmp/distrovenv/detect_os.py

RUN . /tmp/distrovenv/bin/activate && staticx /dist/detect_os /dist/detect_os_static

CMD /dist/detect_os && /dist/detect_os_static

And it passes both

('Debian GNU/Linux', '10', 'buster')
('Debian GNU/Linux', '10', 'buster')

[rokm]

In the first case you are recompiling the bootloader, but you are not installing the updated version. So you're still using the pre-compiled bootloader from the developbranch (which is the new, PIE version).

In the second case, you are not building the bootloader at all, so you're using the pre-compiled version that's part of repository (and corresponds to the old, non-PIE version). So using any commit from before the bootloader was rebuilt for the 4.1 release (3873f04) will seemingly "fix" the issue...

[tfoote]

In the console output I saw it installing some things during the waf build so I though it had done enough:

Waf: Leaving directory `/tmp/pyinstaller/bootloader/build/release'
'build_release' finished successfully (0.237s)
Waf: Entering directory `/tmp/pyinstaller/bootloader/build/debug'
[12/13] Processing build/debug/run_d
+ install /tmp/pyinstaller/PyInstaller/bootloader/Linux-64bit/run_d (from build/debug/run_d)
Waf: Leaving directory `/tmp/pyinstaller/bootloader/build/debug'
'install_debug' finished successfully (0.016s)
Waf: Entering directory `/tmp/pyinstaller/bootloader/build/release'
[12/13] Processing build/release/run
+ install /tmp/pyinstaller/PyInstaller/bootloader/Linux-64bit/run (from build/release/run)
Waf: Leaving directory `/tmp/pyinstaller/bootloader/build/release'
'install_release' finished successfully (0.015s)

How can I install the version? I'm going based on the documentation I found at: https://pyinstaller.readthedocs.io/en/stable/bootloader-building.html

[rokm]

You need to run pip install $pyinstaller_src_dir after compiling the bootloader, so that the new bootloader binaries get installed in your environment.

In your case, that would probably be
RUN . /tmp/distrovenv/bin/activate && pip install /tmp/pyinstaller

[Legorooj]

CC @htgoebel on this one, as he's the one building the bootloaders.

[rokm]

As far as I understand, it's perfectly legal for staticx to append extra ELF sections at the end of the binary (i.e., after the section with archive data), and the problem is that we assume that the archive cookie will be located in the last 4096+sizeof(COOKIE) bytes.

We could expand that search window, but I think it would actually be better to implement full back-to-front scan of the file. This way, we can avoid assumptions about cookie location and also avoid OS-specific parameters; currently on linux, we scan last 4096+sizeof(COOKIE) bytes of file, while on Windows and macOS, we search for the digital signature, and then scan 8+sizeof(COOKIE) bytes that come before the signature or at the end of file if executable is not signed.