-
Notifications
You must be signed in to change notification settings - Fork 555
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Backports stable/8.0] Preallocate segment files #9777
Conversation
Adds a method to preallocate a new file of the expected length, effectively reserving the disk space for later use. The current implementation is a "dumb" one, which simply writes 4KB blocks of 0s until the expected length is reached (meaning the size may be up to 4KB - 1 byte greater than the desired length, which is acceptable for our use case). While potentially slow, this is not in a hot path, and will be later optimized with a native syscall to fallocate.
Adds basic tests for `FileUtil#preallocate`. A new dependency was added, `jnr-posix`, which allows us to check the actual size on disk of the file in UNIX systems. This gives us the accurate number of blocks reserved for the file, on disk, even on file systems with compression or sparse files (i.e. most modern Linux systems), and also ensures that we don't only read the metadata (e.g. what `Files.size` returns) but really guarantee we reserved the disk space.
Allows preallocating segment files if configured to do so. On segment creation, we now preallocate the segment file by default before using it. If it already existed (but was unused by the log so far), we take the easy way out and simply delete/recreate it instead of attempting to grow/shrink it.
I had to merge the changes to The main thing to review is I did think we might want to backport those changes to make it easier to fix bugs in the future, but I figured maybe YAGNI, so let's wait until it happens a second time at least. |
I will backport this to 1.3 after since it's closer to 1.3 than SNAPSHOT is. |
Add a new dependency on `jnr-posix` to accurately get the size of the file on disk. This is necessary to avoid tests passing if we only change the file's metadata - for example, when mmap-ing a file with a mapping of length X, then the file will report it has a length of X, even though it has possibly only one block allocated on disk.
fa77975
to
61f1b4e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, looks good to me 👍
bors merge |
Description
Backports #9731 to 8.0.x.
Related issues
closes #6504
closes #8099
Definition of Done
Not all items need to be done depending on the issue and the pull request.
Code changes:
backport stable/1.3
) to the PR, in case that fails you need to create backports manually.Testing:
Documentation:
Please refer to our review guidelines.