Commits on Jul 19, 2023

1. opt(stream): add option to directly copy over tables from lower levels ( …

   #1700)
   
   Also takes a bug fix from PR #1712, commit 58d0674
   
   This PR adds FullCopy option in Stream. This allows sending the
   table entirely to the writer. If this option is set to true we
   directly copy over the tables from the last 2 levels. This option
   increases the stream speed while also lowering the memory
   consumption on the DB that is streaming the KVs.
   
   For 71GB, compressed and encrypted DB we observed 3x improvement
   in speed. The DB contained ~65GB in the last 2 levels while
   remaining in the above levels.
   
   To use this option, the following options should be set in Stream.
   
   stream.KeyToList = nil
   stream.ChooseKey = nil
   stream.SinceTs = 0
   db.managedTxns = true
   
   If we use stream writer for receiving the KVs, the encryption mode
   has to be the same in sender and receiver. This will restrict
   db.StreamDB() to use the same encryption mode in both input and
   output DB. Added TODO for allowing different encryption modes.

   

   mangalaman93 committed Jul 19, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 1218725
   • Browse repository at this point

   Copy the full SHA

   1218725 View commit details

   Browse the repository at this point in the history