IPFS retrieval from Boost

[aarshkshah1992]

This issue aims to capture the current state of the work to support IPFS retrievals from Boost.

We have a working POC for IPFS retrievals from Filecoin and it is currently accomplished by instantiating a Bitswap server in Boost with a Read-Only blockstore that can return blocks for all cids across all free and unsealed deals.

The following PRs were raised to implement the POC:

• DAGStore PR to implement a blockstore that can return blocks for all cids across all (Boost + legacy market deals) that are free and unsealed. This PR is implemented in the dagstore and constitutes the backbone of this work. Here's how it works at a high level:

  • On being asked for a cid, the blockstore refers to the top level index in the dagstore to lookup all deals(henceforth referred to as a "shard") containing the given cid.
  • It runs the shards through a "shard selector function" that selects a shard to serve the cid from. This shard selector function is injected into the blockstore during instantiation. For the purpose of the POC, we configure the shard selector function to select a free and unsealed deal. We use the dynamic retrieval pricing feature implemented in go-fil-markets to determine if a deal is free or paid.
  • The blockstore than opens the CARv2 index for the selected shard to identify the offset of the requested cid in the CARv1 payload of the shard. (Note: The CARv2 indices for all deals are persisted in the configured dagstore repository and the dagstore runs as a library in the Boost process).
  • The blockstore then makes a http range request to the lotus sector storage subsystem using the offset it gets from the CARv2 index to fetch the block from the CARv1 payload for the deal and returns it to the caller. (Note that no unsealing happens for this as we only make these requests for deals that we've already identified as being unsealed). This http range request is accomplished by using the Lotus Mount configured for the shard at the time of registering a shard with the dagstore.

• Boost PR to instantiate a Bitswap server

  • We instantiate a Bitswap server in Boost that uses the Blockstore described above. The blockstore is instantiated with a "shard selector function" which limits the blockstore to only returning blocks that exist in free and unsealed deals.
  • Boost announces cid manifests for all deals as being available for retrieval on both Bitswap & Graphsync to the network Indexer.

• Markets PR to expose an API to determine if a deal is free and unsealed.

Future work based on feedback

• Run Bitswap retrievals in a separate process.
• Implement Graphsync-esque hooks, controls and rate limiting in Bitswap so SPs have more fine grain control over what cids to serve and which clients to serve to.

[aarshkshah1992]

Closed as duplicate of #348.