The retrievalmarket module is intended for Filecoin node implementations written in Go. It implements functionality to allow execution of retrieval market deals on the Filecoin network. The node implementation must provide access to chain operations, and persistent data storage.
Please see the Filecoin Retrieval Market Specification.
The build process for retrievalmarket requires Go >= v1.13.
To install:
go get github.com/filecoin-project/go-fil-markets/retrievalmarketThe retrievalmarket package provides high level APIs to execute data retrieval deals between a
retrieval client and a retrieval
provider (a.k.a. retrieval miner) on the Filecoin netwwork.
The node must implement the PeerResolver, RetrievalProviderNode, and
RetrievalClientNode interfaces in order to construct and use the module.
Deals are expected to survive a node restart; deals and related information are expected to be stored on disk.
retrievalmarket communicates its deal operations and requested data via
go-data-transfer using
go-graphsync.
Once required Node APIs are implemented and the retrievalmarket APIs are exposed to your desired consumers (such as a command-line or web interface), a retrieval from the client side could proceed roughly like so:
- Your node has a record of data with payloadCIDs and their respective pieceCIDs. Someone,
possibly you, wants to retrieve data referenced by
paylaodCID. - It calls
PeerResolver.GetPeersto obtain a list of retrieval providers storing data referenced bypayloadCID. - It obtains retrieval deal terms by calling each retrieval miners'
Queryfunction. - The node selects the best terms for a retrieval deal and initiates a deal by calling
the retrieval client's
Retrievefunction with the selected retrieval miner and piece info. - The deal then proceeds automatically until all the data is returned and full payment in the form of vouchers is made to the retrieval provider, or the deal errors.
- Once the deal is complete and the final payment voucher is posted to chain, your client account balance will be adjusted according to the terms of the deal.
A retrieval from the provider side is more automated; the RetrievalProvider would be listening for retrieval Query and Retrieve requests, and respond accordingly.
- Your node stores a record of what it has stored locally, or possibly a record of peers with data.
- Your node receives a Query for
payloadCIDand responds automatically with the terms you the node operator have set for retrieval deals. - Your node receives a DealProposal for retrieval, and automatically validates and accepts or rejects it. If accepted, the deal proceeds and your node begins sending data in pieces, stopping every so often to request another voucher for a greater value.
- Once the deal is complete and your node has received a voucher sufficient to cover the entire data transfer, you the node operator may then redeem the voucher and collect FIL.
To collect your FIL, your node must send on-chain
messages directly to the payment channel actor to send all the vouchers,
Settle, and Collect on the deal. This will finalize the client and provider balances for the
retrieval deal on the Filecoin blockchain. Implementation and timing of these calls is the node's
responsibility and is not a part of retrievalmarket. For more information about how
to interact with the
payment channel actor, see the
github.com/filecoin-project/specs-actors repo.
- Decide if your node can be configured as a Retrieval Provider, a Retrieval Client or both.
- Determine how and where your retrieval calls to RetrievalProvider and RetrievalClient functions will be made.
- Implement the required interfaces as described in this section.
- Construct a RetrievalClient in your node's startup, if your node will be a client.
- Construct a RetrievalProvider in your node's startup, if your
node will be a provider.
If setting up a RetrievalProvider, call its
Startfunction it in the appropriate place, and itsStopfunction in the appropriate place. - Expose desired
retrievalmarketfunctionality to whatever internal modules desired, such as command line interface, JSON RPC, or HTTP API.
Implement the PeerResolver, RetrievalProviderNode,
and RetrievalClientNode
interfaces in retrievalmarket/types.go, described below:
PeerResolver is an interface for looking up providers that may have a piece of identifiable data. Its functions are:
func GetPeers(payloadCID cid.Cid) ([]RetrievalPeer, error)Return a slice of RetrievalPeers that store the data referenced by payloadCID.
RetrievalClientNode contains the node dependencies for a RetrievalClient. Its functions are:
AllocateLaneGetChainHeadGetOrCreatePaymentChannelCreatePaymentVoucherWaitForPaymentChannelAddFundsWaitForPaymentChannelCreation
func AllocateLane(paymentChannel address.Address) (uint64, error)Create a lane within paymentChannel so that calls to CreatePaymentVoucher will
automatically make vouchers only for the difference in total. Note that payment channel
Actors have a
lane limit.
func CreatePaymentVoucher(ctx context.Context, paymentChannel address.Address,
amount abi.TokenAmount, lane uint64, tok shared.TipSetToken
) (*paych.SignedVoucher, error)Create a new payment voucher for paymentChannel with amount, for lane lane, given chain
state at tok.
func GetChainHead(ctx context.Context) (shared.TipSetToken, abi.ChainEpoch, error)Get the current chain head. Return its TipSetToken and its abi.ChainEpoch.
func GetOrCreatePaymentChannel(ctx context.Context, clientAddress, minerAddress address.Address,
amount abi.TokenAmount, tok shared.TipSetToken
) (address.Address, cid.Cid, error)If there is a current payment channel for deals between clientAddress and minerAddress,
add amount to the channel, then return the payment channel address and cid.Undef.
If there isn't, construct a new payment channel actor with amount funds by posting
the corresponding message on chain, then return address.Undef and the posted message cid.Cid.
For more information about how to construct a payment channel actor, see
github.com/filecoin-project/specs-actors
func WaitForPaymentChannelAddFunds(messageCID cid.Cid) errorWait for message with CID messageCID on chain that funds have been sent to a payment channel.
func WaitForPaymentChannelCreation(messageCID cid.Cid) (address.Address, error)Wait for a message on chain with CID messageCID that a payment channel has been created.
RetrievalProviderNode contains the node dependencies for a RetrievalProvider.
Its functions are:
func GetChainHead(ctx context.Context) (shared.TipSetToken, abi.ChainEpoch, error)Get the current chain head. Return its TipSetToken and its abi.ChainEpoch.
func GetMinerWorkerAddress(ctx context.Context, addr address.Address, tok shared.TipSetToken,
) (address.Address, error)Get the miner worker address for the given miner owner, as of tok.
func UnsealSector(ctx context.Context, sectorID uint64, offset uint64, length uint64,
) (io.ReadCloser, error)Unseal length data contained in sectorID, starting at offset. Return an io.ReadCloser for accessing the data.
func SavePaymentVoucher(ctx context.Context, paymentChannel address.Address,
voucher *paych.SignedVoucher, proof []byte, expectedAmount abi.TokenAmount,
tok shared.TipSetToken) (abi.TokenAmount, error)Save the provided paych.SignedVoucher for paymentChannel. The RetrievalProviderNode
implementation should validate the SignedVoucher using the provided proof, expectedAmount, based on the chain state referenced by tok. The value of the
voucher should be equal or greater than the largest previous voucher by
expectedAmount. It returns the actual difference.
package retrievalimpl
func NewClient(
netwk network.RetrievalMarketNetwork,
bs blockstore.Blockstore,
node retrievalmarket.RetrievalClientNode,
resolver retrievalmarket.PeerResolver,
ds datastore.Batching,
storedCounter *storedcounter.StoredCounter,
) (retrievalmarket.RetrievalClient, error)-
netwk rmnet.RetrievalMarketNetworkRetrievalMarketNetworkis an interface for creating and handling deal streams. To create it:package network func NewFromLibp2pHost(h host.Host) RetrievalMarketNetwork
where
h host.Hostis your node's libp2p Host. See github.com/libp2p/go-libp2p-core/host. -
bs blockstore.Blockstoreis an IPFS blockstore for storing and retrieving data for deals. See github.com/ipfs/go-ipfs-blockstore. -
node retrievalmarket.RetrievalClientNodeis theRetrievalClientNodeinterface you have implemented. -
resolver retrievalmarket.PeerResolveris thePeerResolverinterface you have implemented. -
ds datastore.Batchingis a datastore for the deal's state machine. It is typically the node's own datastore that implements the IPFS datastore.Batching interface. See github.com/ipfs/go-datastore. -
storedCounter *storedcounter.StoredCounteris a file-based stored counter used to generate new dealIDs. See github.com/filecoin-project/go-storedcounter.
package retrievalimpl
func NewProvider(minerAddress address.Address,
node retrievalmarket.RetrievalProviderNode,
netwk network.RetrievalMarketNetwork,
pieceStore piecestore.PieceStore,
bs blockstore.Blockstore,
ds datastore.Batching,
) (retrievalmarket.RetrievalProvider, error)minerAddress address.Addressis the address of the retrieval miner owner.node retrievalmarket.RetrievalProviderNodeis theRetrievalProviderNodeAPI you have implemented.netwk rmnet.RetrievalMarketNetworkis the same interface for creating and handling deal streams as for constructing a RetrievalClient.pieceStore piecestore.PieceStoreis the database of deals and pieces associated with them. See this repo's piecestore module.bs blockstore.Blockstoreis the same interface as for constructing a RetrievalClient.ds datastore.Batchingis the same batching datastore interface as for constructing a RetrievalClient.
-
GoDoc contains an architectural overview and robust API documentation
-
Retrieval Client FSM diagram:
- Retrieval Provider FSM diagram:
