* dag store lotus mount

* storage client code complete-tests remain

* storage miner first draft

* second draft

* provider state tests are now working

* most unit tests working and more unit tests

* refactor: car store trackers (#559)

* Apply suggestions from code review

Co-authored-by: dirkmc <dirkmdev@gmail.com>

* nits and review

* integration testing changes

* storage test harness changes

* added more TODOs

* fix itests for storage

* get offline deal tests working

* tests work

* more tests

* integration tests

* fix blockstore finalize

* Integrate CARv2 blockstore in the retrieval market (#560)

* refactor: integrate dag store into retrieval market

* fix all tests and the padding issue

* refactor: move mount from shared testutil to dagstore dir

* refactor: add tests for lazy blockstore

* refactor: code cleanup

* feat: update go-car to latest

* Dagstore lotus mount Implementation with tests (#564)

* dagstore lotus mount impl

* refactor: nicer error messages

* mount api tests

* refactor: integrate dag store (#565)

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* some storage market fixes by aarsh

* fix: better error messages in DAG store wrapper

* refactor: simplify mock dag store wrapper

* fix: TestBounceConnectionDealTransferOngoing

* refactor: remove some commented out code

* refactor: closable blockstore interface to use full blockstore

* fix: TestBounceConnectionDealTransferUnsealing

* refactor: add comment explaining lotus mount template

* test: verify that the lazy blockstore is only initialized once

* fix: comment

* fix: always finalize blockstore before reaching complete state (#567)

Co-authored-by: aarshkshah1992 <aarshkshah1992@gmail.com>

Co-authored-by: dirkmc <dirkmdev@gmail.com>

…irst attempt fails with `ErrShardUnknown` (#571)

* register a shard during loading and attempt to load again

* use dagstore 0.1

* address nit

* integrate CARv2 blockstore  resumption

* fix: enable finalize blockstore after restart

* fix: use latest go-car

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

* feat: migration to register all active deals as shards

* feat: check if deal unsealed during register shard migration

* fix: register shards in parallel

* test: shard registration test

* refactor: dag store wrapper register shard

* fix: address review

* refactor: use map to check if deal in sealing state

* refactor: rename shard registration methods

* migration: start consuming results ASAP.

* simplify indirection.

* fix a bug; thanks @dirkmc

* fix: lint imports

Co-authored-by: Raúl Kripalani <raul@protocol.ai>

* remove badger from storage client

* fix test compilation

* update go.mod

* add docs

* fix go mod

* update carv2 and tests for the filestore adapter

* fix tests and update carv2

#575)

* fix isUnsealed check in shard migration

* changes as per review

… complete (#579)

* refactor: move lotus mount and dag store wrapper to lotus

* rebase

Co-authored-by: aarshkshah1992 <aarshkshah1992@gmail.com>

* remove wrong peer check in push deal validation (#585)

* remove wrong peer check in push deal validation

* kill the pull side of peer id checking as well

* fixup comments

* Do not hex-encode CIDs in logs (#561)

* On overloaded CI 10 seconds just isn't enough (#587)

* support padding out smaller files (#536)

* support padding out smaller files

* Actually write out the padding - AP depends on it

Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>

Co-authored-by: Whyrusleeping <why@ipfs.io>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>

…gstore

This commit moves the migration logic out of this module
and into Lotus, as it is implementation specific.

The migration now lives in the DAGStoreWrapper, which
also makes for a simpler design.

redesign dagstore migration.