Python snapshots: Only load dynamic libraries that are needed #1872
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Since this breaks the memory snapshot format, it's a good time to add a version header to the memory snapshots so we can more easily support multiple versions of them. |
|
I also wonder if we could use capnp for the memory snapshot in JS |
hoodmane
force-pushed
the
hoodmane/prealloc-dynlibs
branch
2 times, most recently
from
6d4f003
to
585386e
Compare
Nonfunctional change. Rename reportError.js into util.js and move simpleRunPython there. Also run formatter.
Rather than actually preloading all libraries, we just preallocate space for them. There is a function called `getMemory` that determines the location of the dynamic libraries and nothing else so we can patch this to ensure that the libraries get allocated in their dedicated location if they are loaded at all. This allows us to ensure that their metadata always lands in the right spot. We still have to make sure to load all the libraries in the correct order. There is also the possibility that someone could use ctypes and mess everything up. Though ctypes doesn't work with our snapshots before this PR. I guess it could be fixed by patching libffi to record the trampoline address and function table slot into the DSO_METADATA and then recreate them the same way when restoring the snapshot. We'd also need to record the function table base for each loaded library. Once we do all this, we should be safe again...
hoodmane
force-pushed
the
hoodmane/prealloc-dynlibs
branch
from
585386e
to
9b639a0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rather than actually preloading all libraries, we just preallocate space for them. There is a function called
getMemorythat determines the location of the dynamic libraries and nothing else so we can patch this to ensure that the libraries get allocated in their dedicated location if they are loaded at all. This allows us to ensure that their metadata always lands in the right spot. We also make sure to load all the libraries in the correct order so that they end up in the correct spot in the function pointer table.There is also the possibility that someone could use ctypes and mess everything up. ctypes also doesn't work with our snapshots before this PR. It could be fixed by patching libffi to record the trampoline address and function table slot into the DSO_METADATA and then recreate them the same way when restoring the snapshot. We'd also need to record the function table base for each loaded library. Once we do all this, we should be safe again...