Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replay: add support for capturing and replaying read operations #3560

Open
itsbilal opened this issue Apr 26, 2024 · 0 comments
Open

replay: add support for capturing and replaying read operations #3560

itsbilal opened this issue Apr 26, 2024 · 0 comments
Labels
Projects

Comments

@itsbilal
Copy link
Member

Currently, the replay package in Pebble works to capture and replay write batches and ingestions - all write operations to pebble. The purpose of this is to measure compaction performance, specifically how effectively Pebble can keep up with paced/unpaced write traffic - traffic that is actually CockroachDB-representative. This addressed a common issue with Pebble-specific ycsb benchmarks that didn't stress the LSM in the same way CRDB does.

However, the replay package is mostly only good for measuring changes to WAL/compaction write throughput. The read side of this equation is currently missing; we don't capture or replay read operations (eg. iterator creation and iteration on top of the db, batches, or snapshots). Capturing these operation(s) at the cockroach/pebble interface level, and measuring their latencies and/or ops/sec in addition to everything else already in replay would basically get us a Pebble-level, CockroachDB-representative benchmarking suite.

This is an umbrella issue that consists of many smaller pieces of work; as design progresses on this project, we will turn this into a meta issue and link subtasks here.

@itsbilal itsbilal added enhancement New feature or request A-storage labels Apr 26, 2024
@blathers-crl blathers-crl bot added this to Incoming in Storage Apr 26, 2024
@nicktrav nicktrav moved this from Incoming to Backlog in Storage Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Storage
  
Backlog
Development

No branches or pull requests

1 participant