Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make copyTransactionsFrom much faster in history-free storages when possible #431

Merged
merged 9 commits into from Oct 13, 2020

Commits on Oct 8, 2020

  1. Initial implementation of specialized copyTransactionsFrom for HF sto…

    …rage.
    
    Increases the copy rate from 1.01MB/s to 5.58MB/s on a sample of transactions, or 180000 objects to 417000 objects.
    
    Outstanding issues:
    
    zodbconvert's incremental mode doesnt work.
    zc.zlibstorage doesn't work on the destination during the copy. Maybe that's ok.
    Because you could just not use a decompressing wrapper on the source to get identical copies.
    Logging is messed up.
    jamadden committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    0f54069 View commit details
    Browse the repository at this point in the history
  2. Fix the record_iternext tests and explicitly disable the copy optimiz…

    …ations when using incremental zodbconvert.
    
    This is still not fully safe, though: If you start copying, stop, start again in incremental mode, because one copies by OID and the other by TID you could wind up missing data.
    
    Needs more thought.
    jamadden committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    69d9af9 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2020

  1. zodbconvert: more testing.

    Add tests with blobs, more incremental testing, and tests with history-free destinations.
    
    This found, and added a workaround for, zopefoundation/ZODB#330
    jamadden committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    57fa895 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2917373 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5aa7410 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2020

  1. Handle storage wrappers like zc.zlibstorage in copyTransactionsFrom.

    We'll manually transform the data there, allowing us to replace the wrappers copyTransactionsFrom method with our own optimized implementation in registerDB.
    jamadden committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    e5fe15d View commit details
    Browse the repository at this point in the history
  2. Update zodbconvert logging for fast copies. Also document the (unfort…

    …unate) interaction betwenn incremental copies and fast copies.
    jamadden committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    c16c3bc View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7ee766b View commit details
    Browse the repository at this point in the history
  4. coverage

    jamadden committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    798d34c View commit details
    Browse the repository at this point in the history