@guswynn had penciled down potential optimizations to our spill-to-disk implementation. Now that we have additional disk space on replicas, a quick win to improve rehydration times of upsert sources would be to use RocksDB's native merge operator. This would make rehydration a pure write workload, rather than a read-then-write, and allow us to trade-off CPU for speed.

In the future, we might reconsider using RocksDB. See this Slack thread for initial benchmarks of rehydration performance for upsert sources using different state backends.

Action log

• 15 March 2024. Set release date as April 19 with medium confidence.