walletdb/bdb: attempt to set initial memory map size when opening #697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Roasbeef
force-pushed
the
bbolt-mmap-options
branch
2 times, most recently
from
b443f89
to
d4264e1
Compare
In this commit, we modify the Create and Open methods for bbolt to accept the bbolt config struct directly. With this change, we don't need to update this file each time we want to expose a new config option for the caller to set.
In this commit, we start to set the initial memory map size when opening the database. This helps with bulk insertions or migrations of the database, as if we set this to a larger value, then bbolt needs to remap less often. Remapping can be memory intensive as it needs to copy over the entire existing database. For 32-bit systems, we take care to clamp this value to ensure we don't exceed the largest addressable memory on such systems.
Roasbeef
force-pushed
the
bbolt-mmap-options
branch
from
d4264e1
to
d0cfd37
Compare
Crypt-iQ
reviewed
May 5, 2020
| // If we're on a 32-bit system, then we'll need to limit this | ||
| // value to ensure we don't run into errors down the line. | ||
| if !is64Bit { | ||
| mmapSize = max32BitMapSize |
There was a problem hiding this comment.
If we always set this for 32-bit systems, this will fail bc the process may be using other virtual memory. Also the mmap is a contiguous block of vmem, so even if this value is lower, this problem can still occur. Part of the reason why I think 32-bit support should be dropped
| if !create { | ||
| // The other value that we want to set is the initial memory | ||
| // map size. When bolt expands the mmap, it actually copies | ||
| // over the entire database. By setting this to a large value |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this commit, we start to set the initial memory map size when opening
the database. This helps with bulk insertions or migrations of the
database, as if we set this to a larger value, then bbolt needs to remap
less often. Remapping can be memory intensive as it needs to copy over
the entire existing database.
For 32-bit systems, we take care to clamp this value to ensure we don't
exceed the largest addressable memory on such systems.
In PR commit, also we modify the Create and Open methods for bbolt to
accept the bbolt config struct directly. With this change, we don't need
to update this file each time we want to expose a new config option for
the caller to set.