New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make copyTransactionsFrom much faster in history-free storages when possible #431
Commits on Oct 8, 2020
-
Initial implementation of specialized copyTransactionsFrom for HF sto…
…rage. Increases the copy rate from 1.01MB/s to 5.58MB/s on a sample of transactions, or 180000 objects to 417000 objects. Outstanding issues: zodbconvert's incremental mode doesnt work. zc.zlibstorage doesn't work on the destination during the copy. Maybe that's ok. Because you could just not use a decompressing wrapper on the source to get identical copies. Logging is messed up.
Configuration menu - View commit details
-
Copy full SHA for 0f54069 - Browse repository at this point
Copy the full SHA 0f54069View commit details -
Fix the record_iternext tests and explicitly disable the copy optimiz…
…ations when using incremental zodbconvert. This is still not fully safe, though: If you start copying, stop, start again in incremental mode, because one copies by OID and the other by TID you could wind up missing data. Needs more thought.
Configuration menu - View commit details
-
Copy full SHA for 69d9af9 - Browse repository at this point
Copy the full SHA 69d9af9View commit details
Commits on Oct 9, 2020
-
Add tests with blobs, more incremental testing, and tests with history-free destinations. This found, and added a workaround for, zopefoundation/ZODB#330
Configuration menu - View commit details
-
Copy full SHA for 57fa895 - Browse repository at this point
Copy the full SHA 57fa895View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2917373 - Browse repository at this point
Copy the full SHA 2917373View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5aa7410 - Browse repository at this point
Copy the full SHA 5aa7410View commit details
Commits on Oct 12, 2020
-
Handle storage wrappers like zc.zlibstorage in copyTransactionsFrom.
We'll manually transform the data there, allowing us to replace the wrappers copyTransactionsFrom method with our own optimized implementation in registerDB.
Configuration menu - View commit details
-
Copy full SHA for e5fe15d - Browse repository at this point
Copy the full SHA e5fe15dView commit details -
Update zodbconvert logging for fast copies. Also document the (unfort…
…unate) interaction betwenn incremental copies and fast copies.
Configuration menu - View commit details
-
Copy full SHA for c16c3bc - Browse repository at this point
Copy the full SHA c16c3bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7ee766b - Browse repository at this point
Copy the full SHA 7ee766bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 798d34c - Browse repository at this point
Copy the full SHA 798d34cView commit details