feat: Store file contents statically and use binary search for lookup. #217
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For large directories, RustEmbed generates large amounts of LLVM IR. This is mostly caused by use of match expression with string literals. In Rust, literals from match set are compared one by one; it is kind of like a big if-else chain. Instead, we can store a static lookup table (sorted by file name) and run a binary search over that. This should make file lookup more efficient in runtime and improve compile time too. This improves LLVM IR even for small uses of rust-embed; for examples/basic.rs a size of generated IR goes down by 7% from 4397 to 4082 lines.
If 'compression' is enabled, the static data contains function pointers to get contents of embedded file. That is done so as to work around 'include_flate' using lazy_static.
|
Will bump this to v8 as the implementation kinda changes and want to keep all the packages aligned. |
|
Thanks! |
what happened to |
|
Skipped it to pin all the sub packages to the same version as its getting a bit confusing which dependency matches the main library. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For large directories, RustEmbed generates large amounts of LLVM IR. This is mostly caused by use of match expression with string literals.
In Rust, literals from match set are compared one by one; it is kind of like a big if-else chain. Instead, we can store a static lookup table (sorted by file name) and run a binary search over that. This should make file lookup more efficient in runtime and improve compile time too. This improves LLVM IR even for small uses of rust-embed; for examples/basic.rs a size of generated IR goes down by 7% from 4397 to 4082 lines. For a crate I'm working on this reduced LLVM IR size from 1.7M to 1.55M - RustEmbed is no longer the largest contributor to IR size.
I'd appreciate response to #216, as I had to mark
__rust_embed_newasconstto use it in array element initializer. This can probably be circumvented, though I think that this PR as a whole does not require a recent Rust version to work, so unless MSRV is very strict, we should be okay.