[Epic]: Storage/v2

[tac0turtle]

Summary

In Q1 of 2023 we worked on agreement of a trajectory for storage. We came to a consensus on the separation of storage and commitment. This epic is meant to be the follow up on this ADR.

Work Breakdown

Phase 1

Completion Estimate: 2-3 weeks (23rd August)

Tasks for SS

Beta Give feedback

1. State Storage (SS) API PoC #17223
   C:Store WG: Storage +2
   
2. Implement SS RocksDB Backend #17224
   C:Store WG: Storage +2
   
3. Implement SS PebbleDB Backend #17225
   C:Store WG: Storage +2
   
4. Implement SS BTree Backend #17278
   C:Store WG: Storage +2
   
5. Benchmark SS #17322
   C:Store WG: Storage +2
   

Tasks for SC

Beta Give feedback

1. State Commitment (SC) API PoC #17226
   0 of 5
   C:Store WG: Storage +2
   
2. Benchmarking for SC #17578
   1 of 3
   C:Store +1
   
3. Implement Default SC #17227
   C:Store WG: Storage +2
   

Phase 2

Completion Estimate: TBD

Tasks

Beta Give feedback

1. Update Store Tests CI Job #17341
   C:Store WG: Storage +2
   
2. Finish SS Implementation #17279
   C:Store WG: Storage +2
   
3. Implement RootStore (RS) #17314
   5 of 5
   C:Store WG: Storage +2
   
4. GasKV Store #18225
   C:Store WG: Storage +2
   
5. Implement (Multi) RootStore (RS) #18226
   C:Store +1
   
6. Implement PebbleDB Pruning and Fixes #18462
   C:Store +1
   
7. Snapshot Manager #18329
   4 of 4
   C:Store +1
   
8. Move Relevant Types to Core #17576
   C:Store +1
   
9. Pruning Manager #18180
   C:Store WG: Storage +2
   
10. Tune PebbleDB Options #18178
    C:Store WG: Storage +2
    
11. Store: Export Metrics #18463
    C:Store +1
    
12. Commit SS and SC Async #18464
    C:Store +1
13. Remove the internal module from store #18738
    C:Store +1
    
14. Framework Integration #18466
    C:Store +1
    
15. Store: Support Direct SS Writes #18467
    C:Store +1
    
16. Streaming Manager #18527
    C:streaming +1
    
17. Store v2 Migration #18603
    2 of 4
    C:Store +1
    

Phase 3

Completion Estimate: TBD

Tasks

Beta Give feedback

1. Test on existing mainnets (if possible)
2. Write READMEs for SS and SC
3. Reevaluate Storage v2 ADR #18465
   C:Store +1
   
4. Release Store & Core Package
5. Write Blog post
6. Update Documentation

[tac0turtle]

posting a link to some issues that are relevant to the new store pacakge

#13058
#10212
#14377