loader: implement FrozenImporter source loading from .py files #5697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Provide basic FrozenImporter.get_source() implementation that allows loading of source from .py files that have been collected by hooks as data files.
rokm
force-pushed
the
importer_get_source
branch
from
b8168a2
to
e850685
Compare
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Provide basic
FrozenImporter.get_source()implementation that allows loading of source from.pyfiles that have been collected by hooks as data files.This is, for example, required by PyTorch's JIT compilation mechanism.
The issue of missing source code inside frozen application, which is required by some packages, is tracked under #4764. However, it seems the primary solution discussed there is collecting
.pyfiles into PYZ archive. This PR instead adds support only for loading source from on-filesystem.pyfiles that have been collected viacollect_data_files()withinclude_py_files=True, which was also suggested by #4764 (comment). But in the end, the two approaches are complementary, and even if we really wanted to have.pyfiles primarily collected into PYZ, we would still need an on-filesystem fallback (fornoarchiveoption).