Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve scheduling in account-based migration #5901

Merged
merged 5 commits into from May 16, 2024

Conversation

turbolent
Copy link
Member

@turbolent turbolent commented May 13, 2024

Depends on #5897

Closes onflow/cadence#3308

Currently, the migrations of accounts are scheduled in address order. If there are large accounts at high addresses, the migration might end up waiting for a these accounts, with most available workers idling.

Schedule the migration of large accounts first to increase utilization of available workers.

@turbolent turbolent requested review from a team May 13, 2024 16:47
@turbolent turbolent self-assigned this May 13, 2024
Copy link
Member

@SupunS SupunS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea!

Copy link
Member

@fxamacker fxamacker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

I only left a suggestion about avoiding overhead of creating and sorting allAccountRegisters slice by only tracking the largest accounts above a threshold.

cmd/util/ledger/migrations/account_based_migration.go Outdated Show resolved Hide resolved
Base automatically changed from bastian/optimize-migrations to master May 14, 2024 17:24
@turbolent turbolent requested review from fxamacker, SupunS and a team May 15, 2024 18:44
@turbolent
Copy link
Member Author

@codecov-commenter
Copy link

codecov-commenter commented May 15, 2024

Codecov Report

Attention: Patch coverage is 80.00000% with 20 lines in your changes are missing coverage. Please review.

Project coverage is 55.61%. Comparing base (15d2e63) to head (db1b024).
Report is 181 commits behind head on master.

Files Patch % Lines
...ledger/migrations/account_size_filter_migration.go 0.00% 11 Missing ⚠️
.../util/ledger/migrations/account_based_migration.go 84.21% 7 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5901      +/-   ##
==========================================
+ Coverage   55.58%   55.61%   +0.02%     
==========================================
  Files        1127     1128       +1     
  Lines       88840    88926      +86     
==========================================
+ Hits        49380    49453      +73     
- Misses      34751    34761      +10     
- Partials     4709     4712       +3     
Flag Coverage Δ
unittests 55.61% <80.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@fxamacker fxamacker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for updating PR to replace very large sorted slice with a tiny queue!

I only left 1 comment about Less() for topDurations.

cmd/util/ledger/migrations/account_based_migration.go Outdated Show resolved Hide resolved
Co-authored-by: Faye Amacker <33205765+fxamacker@users.noreply.github.com>
@turbolent turbolent requested review from a team May 16, 2024 01:06
@turbolent turbolent enabled auto-merge May 16, 2024 01:29
@turbolent turbolent added this pull request to the merge queue May 16, 2024
Merged via the queue into master with commit 64d7049 May 16, 2024
55 checks passed
@turbolent turbolent deleted the bastian/improve-migration-scheduling branch May 16, 2024 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Schedule migration of accounts based on size, largest first
4 participants