Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

db: elevate some table-level props to fileMetadata #3160

Open
jbowens opened this issue Dec 14, 2023 · 0 comments
Open

db: elevate some table-level props to fileMetadata #3160

jbowens opened this issue Dec 14, 2023 · 0 comments

Comments

@jbowens
Copy link
Collaborator

jbowens commented Dec 14, 2023

Block property collectors collect table-wide properties in addition to the more granular block-level properties. When an iterator configured with a corresponding block-property filter is constructed over a table, the table-level properties are read to determine if there's any intersection, and if not, the file is skipped without further reading it.

In order to read the table-wide properties, the file must first be opened (reading its footer, metaindex block and properties block). Typically in realistic Cockroach workloads, the table is already open and in the table cache, so all this work has already been done. However with online restore, it's more likely that the table has not yet been opened. Since reads to blob storage are higher latency, the repeated sequential reads during table opening are especially costly.

Alternatively, we could lift some table-level properties up into the file's manifest.FileMetadata, persisting them in the manifest when the file is linked in. This would allow a reader with a selective block-property filter to potentially exclude a sstable without ever opening it, on the basis of the table-wide property value.

We wouldn't want to encode the entire block-property key name with each NewFileEntry in a version edit, so we would need to persist some property to enum variant mapping within the manifest.

Somewhat related to #2002 which also seeks to lift MVCC properties in the FileMetadata (although not necessarily via the manifest).

Internal slack thread: https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1702567335723599

@blathers-crl blathers-crl bot added this to Incoming in Storage Dec 14, 2023
@nicktrav nicktrav moved this from Incoming to Backlog in Storage Jan 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Storage
  
Backlog
Development

No branches or pull requests

1 participant