-
Notifications
You must be signed in to change notification settings - Fork 145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up ledger-from-archive #818
base: main
Are you sure you want to change the base?
Speed up ledger-from-archive #818
Conversation
86c1432
to
d19db7e
Compare
d19db7e
to
c98ff92
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, based on discussions. Not checking for Rust completeness.
loop { | ||
let end = start + config.block_chunk_size; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be min(start + config.block_chunk_size, config.num_blocks.unwrap_or(u64::maxint))
?
err | ||
); | ||
return; | ||
.unwrap(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.expect
/// | ||
/// Peers are queried concurrently. Currently, this method will run indefinitely | ||
/// until all transactions have been retrieved. | ||
fn get_block_contents( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason to copy this entire thing instead of making the one in ledger_sync_service pub
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, I see that it is less strict so using the one in ledger_sync_service would be difficult.
); | ||
match transactions_fetcher.block_from_url(&url) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replacing this call with transactions_fetcher.get_block_data_by_index(block_index, None)
should be sufficient for benefiting from the merged blocks and getting the speed increase you are looking for.
This still seems useful; do we want to merge it? |
Soundtrack of this PR: I love it
Motivation
Ledger-from-archive is painfully slow, especially on TestNet at 160K blocks currently
In this PR
Future Work