integrate dagstore and carv2, replacing monolithic blockstore.

This commit introduces changes in state machines, abstractions,
and provider/client environments to remove the reliance on a
monolithic blockstore, and instead migrate to a solution backed
by the dagstore (https://github.com/filecoin-project/dagstore).

Blockstores are not managed by this module, and are managed by
the caller. When this module needs a blockstore in the context of
a storage deal or a retrieval, it calls the relevant
BlockstoreAccessor.

We believe the test coverage to be pretty satisfactory, with a
substantial portion of the LoC diff corresponding to tests.

This commit is a squashed version of 89 commits developed by
@aarshkshah1992, @dirkmc, and @raulk over the course of many months.

This contribution was thoroughly tested in the M1 milestone with
minerX.2: filecoin-project/lotus#6852.

Read more about the motivation here:
- protocol/web3-dev-team#116
- https://github.com/filecoin-project/dagstore/blob/master/docs/design.md

Co-authored-by: aarshkshah1992 <aarshkshah1992@gmail.com>
Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>

.circleci/config.yml

@@ -7,7 +7,7 @@ orbs:
 executors:
   golang:
     docker:
-      - image: circleci/golang:1.14-node
+      - image: circleci/golang:1.16.4
     resource_class: large
 
 commands:

docs/retrievalclient.mmd

@@ -26,6 +26,10 @@ stateDiagram-v2
 	state "DealStatusWaitForAcceptanceLegacy" as DealStatusWaitForAcceptanceLegacy
 	state "DealStatusWaitingForLastBlocks" as DealStatusWaitingForLastBlocks
 	state "DealStatusPaymentChannelAddingInitialFunds" as DealStatusPaymentChannelAddingInitialFunds
+	state "DealStatusErroring" as DealStatusErroring
+	state "DealStatusRejecting" as DealStatusRejecting
+	state "DealStatusDealNotFoundCleanup" as DealStatusDealNotFoundCleanup
+	state "DealStatusFinalizingBlockstore" as DealStatusFinalizingBlockstore
 	DealStatusNew : On entry runs ProposeDeal
 	DealStatusPaymentChannelCreating : On entry runs WaitPaymentChannelReady
 	DealStatusPaymentChannelAddingFunds : On entry runs WaitPaymentChannelReady
@@ -42,24 +46,28 @@ stateDiagram-v2
 	DealStatusCancelling : On entry runs CancelDeal
 	DealStatusRetryLegacy : On entry runs ProposeDeal
 	DealStatusPaymentChannelAddingInitialFunds : On entry runs WaitPaymentChannelReady
+	DealStatusErroring : On entry runs FailsafeFinalizeBlockstore
+	DealStatusRejecting : On entry runs FailsafeFinalizeBlockstore
+	DealStatusDealNotFoundCleanup : On entry runs FailsafeFinalizeBlockstore
+	DealStatusFinalizingBlockstore : On entry runs FinalizeBlockstore
 	[*] --> DealStatusNew
 	note right of DealStatusNew
 		The following events are not shown cause they can trigger from any state.
 
-		ClientEventWriteDealProposalErrored - transitions state to DealStatusErrored
+		ClientEventWriteDealProposalErrored - transitions state to DealStatusErroring
 		ClientEventUnknownResponseReceived - transitions state to DealStatusFailing
-		ClientEventDataTransferError - transitions state to DealStatusErrored
-		ClientEventWriteDealPaymentErrored - transitions state to DealStatusErrored
+		ClientEventDataTransferError - transitions state to DealStatusErroring
+		ClientEventWriteDealPaymentErrored - transitions state to DealStatusErroring
 		ClientEventProviderCancelled - transitions state to DealStatusCancelling
 		ClientEventCancel - transitions state to DealStatusCancelling
 	end note
 	DealStatusNew --> DealStatusNew : ClientEventOpen
 	DealStatusNew --> DealStatusWaitForAcceptance : ClientEventDealProposed
 	DealStatusRetryLegacy --> DealStatusWaitForAcceptanceLegacy : ClientEventDealProposed
 	DealStatusWaitForAcceptance --> DealStatusRetryLegacy : ClientEventDealRejected
-	DealStatusWaitForAcceptanceLegacy --> DealStatusRejected : ClientEventDealRejected
-	DealStatusWaitForAcceptance --> DealStatusDealNotFound : ClientEventDealNotFound
-	DealStatusWaitForAcceptanceLegacy --> DealStatusDealNotFound : ClientEventDealNotFound
+	DealStatusWaitForAcceptanceLegacy --> DealStatusRejecting : ClientEventDealRejected
+	DealStatusWaitForAcceptance --> DealStatusDealNotFoundCleanup : ClientEventDealNotFound
+	DealStatusWaitForAcceptanceLegacy --> DealStatusDealNotFoundCleanup : ClientEventDealNotFound
 	DealStatusWaitForAcceptance --> DealStatusAccepted : ClientEventDealAccepted
 	DealStatusWaitForAcceptanceLegacy --> DealStatusAccepted : ClientEventDealAccepted
 	DealStatusPaymentChannelCreating --> DealStatusFailing : ClientEventPaymentChannelErrored
@@ -95,8 +103,8 @@ stateDiagram-v2
 	DealStatusOngoing --> DealStatusBlocksComplete : ClientEventAllBlocksReceived
 	DealStatusFundsNeededLastPayment --> DealStatusSendFundsLastPayment : ClientEventAllBlocksReceived
 	DealStatusBlocksComplete --> DealStatusBlocksComplete : ClientEventAllBlocksReceived
-	DealStatusCheckComplete --> DealStatusCompleted : ClientEventAllBlocksReceived
-	DealStatusWaitingForLastBlocks --> DealStatusCompleted : ClientEventAllBlocksReceived
+	DealStatusCheckComplete --> DealStatusFinalizingBlockstore : ClientEventAllBlocksReceived
+	DealStatusWaitingForLastBlocks --> DealStatusFinalizingBlockstore : ClientEventAllBlocksReceived
 	DealStatusFundsNeeded --> DealStatusFundsNeeded : ClientEventBlocksReceived
 	DealStatusSendFunds --> DealStatusOngoing : ClientEventBlocksReceived
 	DealStatusSendFundsLastPayment --> DealStatusOngoing : ClientEventBlocksReceived
@@ -129,10 +137,15 @@ stateDiagram-v2
 	DealStatusOngoing --> DealStatusCheckComplete : ClientEventComplete
 	DealStatusFundsNeededLastPayment --> DealStatusCheckComplete : ClientEventComplete
 	DealStatusBlocksComplete --> DealStatusCheckComplete : ClientEventComplete
-	DealStatusFinalizing --> DealStatusCompleted : ClientEventComplete
-	DealStatusCheckComplete --> DealStatusCompleted : ClientEventCompleteVerified
-	DealStatusCheckComplete --> DealStatusErrored : ClientEventEarlyTermination
+	DealStatusFinalizing --> DealStatusFinalizingBlockstore : ClientEventComplete
+	DealStatusCheckComplete --> DealStatusFinalizingBlockstore : ClientEventCompleteVerified
+	DealStatusCheckComplete --> DealStatusErroring : ClientEventEarlyTermination
 	DealStatusCheckComplete --> DealStatusWaitingForLastBlocks : ClientEventWaitForLastBlocks
+	DealStatusErroring --> DealStatusErrored : ClientEventBlockstoreFinalized
+	DealStatusRejecting --> DealStatusRejected : ClientEventBlockstoreFinalized
+	DealStatusDealNotFoundCleanup --> DealStatusDealNotFound : ClientEventBlockstoreFinalized
+	DealStatusFinalizingBlockstore --> DealStatusCompleted : ClientEventBlockstoreFinalized
+	DealStatusFinalizingBlockstore --> DealStatusErrored : ClientEventFinalizeBlockstoreErrored
 	DealStatusFailing --> DealStatusErrored : ClientEventCancelComplete
 	DealStatusCancelling --> DealStatusCancelled : ClientEventCancelComplete
 	DealStatusInsufficientFunds --> DealStatusCheckFunds : ClientEventRecheckFunds
@@ -166,3 +179,6 @@ stateDiagram-v2
 
 	note left of DealStatusPaymentChannelAddingInitialFunds : The following events only record in this state.<br><br>ClientEventLastPaymentRequested<br>ClientEventPaymentRequested<br>ClientEventAllBlocksReceived<br>ClientEventBlocksReceived
 
+
+	note left of DealStatusFinalizingBlockstore : The following events only record in this state.<br><br>ClientEventWaitForLastBlocks
+

docs/storageprovider.mmd

@@ -77,8 +77,8 @@ stateDiagram-v2
 	4 --> 11 : ProviderEventMultistoreErrored
 	4 --> 11 : ProviderEventDealHandoffFailed
 	4 --> 29 : ProviderEventDealHandedOff
-	29 --> 11 : <invalid Value>
-	29 --> 5 : <invalid Value>
+	29 --> 11 : ProviderEventDealPrecommitFailed
+	29 --> 5 : ProviderEventDealPrecommitted
 	5 --> 11 : ProviderEventDealActivationFailed
 	5 --> 6 : ProviderEventDealActivated
 	29 --> 6 : ProviderEventDealActivated

filestore/filestore.go

@@ -14,16 +14,12 @@ type fileStore struct {
 }
 
 // NewLocalFileStore creates a filestore mounted on a given local directory path
-func NewLocalFileStore(basedirectory OsPath) (FileStore, error) {
-	base := filepath.Clean(string(basedirectory))
-	info, err := os.Stat(string(base))
+func NewLocalFileStore(baseDir OsPath) (FileStore, error) {
+	base, err := checkIsDir(string(baseDir))
 	if err != nil {
-		return nil, fmt.Errorf("error getting %s info: %s", base, err.Error())
-	}
-	if !info.IsDir() {
-		return nil, fmt.Errorf("%s is not a directory", base)
+		return nil, err
 	}
-	return &fileStore{string(base)}, nil
+	return &fileStore{base}, nil
 }
 
 func (fs fileStore) filename(p Path) string {
@@ -73,3 +69,15 @@ func (fs fileStore) CreateTemp() (File, error) {
 	filename := filepath.Base(f.Name())
 	return &fd{File: f, basepath: fs.base, filename: filename}, nil
 }
+
+func checkIsDir(baseDir string) (string, error) {
+	base := filepath.Clean(string(baseDir))
+	info, err := os.Stat(base)
+	if err != nil {
+		return "", fmt.Errorf("error getting %s info: %s", base, err.Error())
+	}
+	if !info.IsDir() {
+		return "", fmt.Errorf("%s is not a directory", base)
+	}
+	return base, nil
+}

filestore/types.go

@@ -1,6 +1,8 @@
 package filestore
 
-import "io"
+import (
+	"io"
+)
 
 // Path represents an abstract path to a file
 type Path string

go.mod

@@ -3,7 +3,8 @@ module github.com/filecoin-project/go-fil-markets
 go 1.13
 
 require (
-	github.com/filecoin-project/go-address v0.0.3
+	github.com/filecoin-project/dagstore v0.4.2
+	github.com/filecoin-project/go-address v0.0.5
 	github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
 	github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7
 	github.com/filecoin-project/go-data-transfer v1.7.0
@@ -12,40 +13,44 @@ require (
 	github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
 	github.com/filecoin-project/go-multistore v0.0.3
 	github.com/filecoin-project/go-padreader v0.0.0-20210723183308-812a16dc01b1
-	github.com/filecoin-project/go-state-types v0.0.0-20201102161440-c8033295a1fc
+	github.com/filecoin-project/go-state-types v0.1.1-0.20210506134452-99b279731c48
 	github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe
-	github.com/filecoin-project/go-statestore v0.1.0
+	github.com/filecoin-project/go-statestore v0.1.1
 	github.com/filecoin-project/specs-actors v0.9.13
-	github.com/filecoin-project/specs-actors/v2 v2.3.2
+	github.com/filecoin-project/specs-actors/v2 v2.3.5-0.20210114162132-5b58b773f4fb
 	github.com/hannahhoward/cbor-gen-for v0.0.0-20200817222906-ea96cece81f1
 	github.com/hannahhoward/go-pubsub v0.0.0-20200423002714-8d62886cc36e
-	github.com/ipfs/go-block-format v0.0.2
-	github.com/ipfs/go-blockservice v0.1.4-0.20200624145336-a978cec6e834
-	github.com/ipfs/go-cid v0.0.7
+	github.com/ipfs/go-block-format v0.0.3
+	github.com/ipfs/go-blockservice v0.1.5
+	github.com/ipfs/go-cid v0.0.8-0.20210716091050-de6c03deae1c
+	github.com/ipfs/go-cidutil v0.0.2
 	github.com/ipfs/go-datastore v0.4.5
+	github.com/ipfs/go-filestore v1.0.0
 	github.com/ipfs/go-graphsync v0.6.4
-	github.com/ipfs/go-ipfs-blockstore v1.0.3
+	github.com/ipfs/go-ipfs-blockstore v1.0.4
 	github.com/ipfs/go-ipfs-blocksutil v0.0.1
 	github.com/ipfs/go-ipfs-chunker v0.0.5
 	github.com/ipfs/go-ipfs-ds-help v1.0.0
 	github.com/ipfs/go-ipfs-exchange-offline v0.0.1
 	github.com/ipfs/go-ipfs-files v0.0.8
 	github.com/ipfs/go-ipld-format v0.2.0
-	github.com/ipfs/go-log/v2 v2.1.2-0.20200626104915-0016c0b4b3e4
+	github.com/ipfs/go-log/v2 v2.3.0
 	github.com/ipfs/go-merkledag v0.3.2
-	github.com/ipfs/go-unixfs v0.2.4
+	github.com/ipfs/go-unixfs v0.2.6
 	github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d
+	github.com/ipld/go-car/v2 v2.0.2
 	github.com/ipld/go-ipld-prime v0.5.1-0.20201021195245-109253e8a018
 	github.com/jbenet/go-random v0.0.0-20190219211222-123a90aedc0c
 	github.com/jpillora/backoff v1.0.0
-	github.com/libp2p/go-libp2p v0.12.0
-	github.com/libp2p/go-libp2p-core v0.7.0
+	github.com/libp2p/go-libp2p v0.14.0
+	github.com/libp2p/go-libp2p-core v0.8.5
 	github.com/multiformats/go-multiaddr v0.3.1
 	github.com/multiformats/go-multibase v0.0.3
-	github.com/stretchr/testify v1.6.1
-	github.com/whyrusleeping/cbor-gen v0.0.0-20210219115102-f37d292932f2
-	golang.org/x/exp v0.0.0-20200207192155-f17229e696bd
-	golang.org/x/net v0.0.0-20201021035429-f5854403a974
+	github.com/multiformats/go-multihash v0.0.15
+	github.com/stretchr/testify v1.7.0
+	github.com/whyrusleeping/cbor-gen v0.0.0-20210713220151-be142a5ae1a8
+	golang.org/x/exp v0.0.0-20210715201039-d37aa40e8013
+	golang.org/x/net v0.0.0-20210423184538-5f58ad60dda6
 	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
 )