Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration 11-12: migrate CIDsv1 to raw multihashes #95

Merged
merged 13 commits into from
Nov 17, 2021

Commits on Apr 20, 2021

  1. Configuration menu
    Copy the full SHA
    5da50d9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    89c5b50 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    848595b View commit details
    Browse the repository at this point in the history
  4. 11-12: Track if the raw mhash already existed before migrating

    In this case, we need to keep them around during the revert.
    
    We do this by pre-pending + to the list.
    
    When doing reverts, we do not delete raw multihashes that already existed.
    
    This happens only if the user had both the CidV1 and  the CidV0 for something.
    hsanjuan committed Apr 20, 2021
    Configuration menu
    Copy the full SHA
    cf67de4 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2021

  1. 11-12: Ensure the pinset/mfs trees are migrated back

    This commit ensure that we traverse all the CIDs in the pinset and MFS trees
    and revert them to cid-based addressing in the datastore when the migration is
    reversed. This addresses the problem when a user has added new CIDv1 pins or
    modified MFS with CIDv1 content before reverting the migration. In that case,
    those new multihash-addressed blocks would stay as such and while they could
    potentially still be read, they would be swiped out by GC in the older version
    of the code.
    
    Additionally, there are corner cases when content is added with both CIDv1 and
    CIDv0 (but the same multihash). These were partially addressed already by
    keeping track of CIDv0 blocks that existed already for CIDv1 equivalents, and
    by some logic during reverts, to handle blocks referenced from several
    CIDv1s. However, things like adding CIDv1 and CIDv0 equivalent blocks to the
    pinset after migration would not be handled properly.
    
    To address all these issues, and simplify, we have opted to not delete any
    raw-multihash-addressed block during revert. We convert as much as we can to
    CIDv1 (using the backup file and the pinset/mfs trees), but otherwise leave
    all the blocks. This simplifies the code and makes reverts more
    memory-friendly at the expense of extra disk space.
    
    As a final reminder, CIDv1s are not the default. We can expect that in the
    general case, people have content that is CIDv0 addressed and requires no
    action whatsoever in both migrating forward and backwards.
    hsanjuan committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    fc971af View commit details
    Browse the repository at this point in the history
  2. Do not unswap Cidv0s in MFS/pinset tree during reversions.

    They are fine as they are.
    hsanjuan committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    a1fe736 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2021

  1. Migration 11-12: Add a test

    hsanjuan committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    e3dd7b8 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2021

  1. Configuration menu
    Copy the full SHA
    6aadb8c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e82e5ce View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2ea97fd View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b059b50 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2021

  1. Configuration menu
    Copy the full SHA
    bc7f546 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2021

  1. Merge pull request #131 from ipfs/feat/repo-migration-raw-multihashes…

    …-vendored
    
    Migration 11-12: vendor dependencies
    hsanjuan committed Nov 17, 2021
    Configuration menu
    Copy the full SHA
    f91ca2d View commit details
    Browse the repository at this point in the history