Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tool: build an sstable scan cmd that can read the backing file contents through a virtual reader #3302

Open
msbutler opened this issue Feb 14, 2024 · 2 comments

Comments

@msbutler
Copy link
Contributor

msbutler commented Feb 14, 2024

Currently, there isn't a great way to inspect the backing file contents of a virtual file. We ought to build an sstable scan-like command that takes in just a file number, resolves the path to it (and opens it), and applies all transforms to a file and spits out all the keys within it.

Note: this issue was originally written as:
Currently, the reader.Layout() command serves a similar purpose: the sstable tool and the checksum codepath use the reader.Layout() function to inspect the block organization of a local sstable. If a Virtual Reader calls Layout() to inspect its backing file, Layout() is naive to certain transformations (e.g. virtual sst bounds, synthetic suffix replacement) that normally affect the virtual read path. While this doesn't seem to be problem currently, as these transformations do not affect checksum results, we should consider teaching the Layout() call about virtual sst transformations so that backing files are read in a consistent manner throughout the codebase.

@itsbilal
Copy link
Member

@msbutler we talked about this in the storage weekly and felt that Layout() is probably best suited at continuing to reflect the true physical form of the sstable, that is, without any interpretative vsst transformations from the FileMetadata applied on it. However, we'd probably like to see an sstable scan-like command that takes in just a file number, resolves the path to it (and opens it), and applies all transforms to a file and spits out all the keys within it. That will likely be useful. Are you open to changing up this issue to reflect that?

@msbutler msbutler changed the title sstable: consider teaching reader.Layout() about virtual SST transformations tool: build and sstable scan cmd that can read the backing file contents through a virtual reader Feb 20, 2024
@msbutler
Copy link
Contributor Author

Done.

@itsbilal itsbilal changed the title tool: build and sstable scan cmd that can read the backing file contents through a virtual reader tool: build an sstable scan cmd that can read the backing file contents through a virtual reader Feb 27, 2024
@itsbilal itsbilal moved this from Incoming to Disagg - Online restore in Storage Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Storage
  
Disagg - Online restore
Development

No branches or pull requests

2 participants