Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unreproducible build: base_library.zip contains references to source files #5563

Closed
jdcowf opened this issue Feb 17, 2021 · 1 comment · Fixed by #5564
Closed

Unreproducible build: base_library.zip contains references to source files #5563

jdcowf opened this issue Feb 17, 2021 · 1 comment · Fixed by #5564
Labels

Comments

@jdcowf
Copy link

jdcowf commented Feb 17, 2021

Description of the issue

.pyc files in base_library.zip contain references to their source files. This is similar to #5377.

Context information (for bug reports)

  • Output of pyinstaller --version: 4.2, 5.0.dev0
  • Version of Python: 3.8.1
  • Platform: Win10 En
  • Did you also try this on another platform? Does it work there?
    Did not try on another platform

What I did:

  1. Install Python 3.8.2 in 2 locations (e.g. C:\Python38 and C:\Python\Py38)
  2. Create a virtual environment from one C:\Python38\python.exe -m venv .venv
  3. Create an empty test.py file touch test.py
  4. activate .venv
  5. SET PYTHONHASHSEED=1
  6. invoke pyinstaller --clean -y test.py
  7. copy the output to another location
  8. repeat steps 2-6 with the other install
  9. compare base library of the 2 builds and observe the differences in the .pyc files

More:

The root cause seems to be that the mod.code is not having strip_paths_in_code invoked on it prior to writing it like is done in other areas of the code.

with io.BytesIO() as fc:
# Prepare all data in byte stream file-like object.
fc.write(BYTECODE_MAGIC)
if is_py37:
# Additional bitfield according to PEP 552
# 0b01 means hash based but don't check the hash
fc.write(struct.pack('<I', 0b01))
with open(mod.filename, 'rb') as fs:
source_bytes = fs.read()
source_hash = importlib_source_hash(source_bytes)
fc.write(source_hash)
else:
fc.write(struct.pack('<II', timestamp, size))
marshal.dump(mod.code, fc)

@rokm
Copy link
Member

rokm commented Feb 18, 2021

Well spotted, thanks! The base python location is even more invariant than the virtual environment location, which makes this even harder to notice.

I guess the real challenge here is getting strip_paths_in_code imported without causing import cycles between PyInstaller's modules...

@rokm rokm added the bug label Feb 18, 2021
rokm added a commit to rokm/pyinstaller that referenced this issue Feb 18, 2021
Strip paths from modules collected into base_library.zip, to enable
reproducible builds regardless of python's install location.

Fixes pyinstaller#5563.
Legorooj pushed a commit that referenced this issue Feb 19, 2021
#5564)

Strip paths from modules collected into base_library.zip, to enable
reproducible builds regardless of python's install location.

Fixes #5563.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants