Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store Gateway: Index header disk space management #7029

Open
yeya24 opened this issue Jan 3, 2024 · 0 comments · May be fixed by #7118
Open

Store Gateway: Index header disk space management #7029

yeya24 opened this issue Jan 3, 2024 · 0 comments · May be fixed by #7118

Comments

@yeya24
Copy link
Contributor

yeya24 commented Jan 3, 2024

Is your proposal related to a problem?

#6984 adds support for lazy downloaded index headers. It can be useful if only some blocks are accessed more often than other blocks. For example, data from recent one month are usually accessed more often than data older than one year. With lazy downloaded index header feature, we can speed up store gateway start up time and index header disk space usage because we download less index headers.

However, there is no way to clean up index headers on local disk if they are not accessed anymore. If an index header file is downloaded and accessed only once, it remain on the disk forever. We only do mmap unload but there is no local data retention.

Describe the solution you'd like

  1. Purge local index headers periodically if files are not used after some time.
  2. Implement size based strategy as well for ^

We can start with option 1 and add size based strategy later.

Describe alternatives you've considered

NA

@bodevone bodevone linked a pull request Feb 2, 2024 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant