IPIP-402: Partial CAR Support on Trustless Gateways #402
Conversation
Add specification for queries for verifiable CAR requests
hannahhoward
changed the title
hannahhoward
force-pushed
the
feat/improving-trustless-gateways
branch
from
b54635b
to
6f7f7b3
Compare
There was a problem hiding this comment.
Thank you for opening this @hannahhoward!
I don't think this is a breaking change, since by default, the change is to return MORE blocks, than requested.
There may be perf. penalty due to unexpected blocks, but there should not be any breakage.
Initial thoughts in comments inline.
Once we are happy with the API, the next steps will be:
- dogfood this in Rhea for few weeks
- implement it in
boxo/gatewaysobifrost-gateway,kubo, and probablyboosttoo, all benefit and expose the same API - add IPIP document to this PR and include description of text fixtures we need
- write conformance tests that use the same test fixtures, allowing others (e.g. Lassie compatible http api for fetching CARs web3-storage/freeway#34) to test compatibility
lidel
changed the title
|
Related tracking items that would be good to add here:
|
Co-authored-by: Henrique Dias <hacdias@gmail.com> Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
are you specifically interested in feature detection of what codecs are supported by the gateway, or rather if the gateway is able to support e.g. the car scope's being introduced here? |
Both. Codecs, multibases, hash algos, any parameter one might add to a gateway request, etc.. Really any feature that a conceivable MVP gateway might not support and still be considered roughly compliant and still a gateway some client might want to use for other requests. I brought up "?format=raw" because I use that a lot currently (one could argue whether that's a good idea, but... if I reached for it someone else will at some point too) and a fair number of gateways currently don't support it. Perhaps the bigger pain point has been the IPNS records, which are obviously a new feature and thus haven't been widely deployed for that reason. But one could also imagine an upstart gateway implementation not wanting to support it on day 1 either. I don't think we need to be too careful about putting too many things in OPTIONS - it's not something a client would likely ask very frequently, each included feature is a handful of bytes so it's not much of a network traffic problem, and the gateway likely already has the full response in memory anyhow (since it's not dependent on who is asking, nor is it changing) so it's not exactly a DOS problem. |
add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string supports carScope to specify what blocks to return for the resolved dag - `'all'`: return the entire dag starting at path. (default) - `'block'`: return the block identified by the path. - `'file'`: Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents. see: web3-storage/freeway#33 see: web3-storage/freeway#34 see: ipfs/specs#402 TODO: - [x] find out how to identify the boundaries of a unixfs hamt ...unixfs-exporter seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems loose... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42 License: MIT --------- Signed-off-by: Oli Evans <oli@protocol.ai> Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
lidel
force-pushed
the
feat/improving-trustless-gateways
branch
from
608e7b7
to
e1af6e4
Compare
This backports CIDs and CARs from gateway conformance to the IPIP-402, and provides some basic hints on how each fixture should be used.
| @@ -235,9 +230,14 @@ single request. | |||
|
|
|||
| ## Test fixtures | |||
There was a problem hiding this comment.
ℹ️ A detailed list of compliance checks for dag-scope and entity-bytes can be found in gateway-conformance v0.2.0 → trustless_gateway_car_test.go. I've made another pass and double-checked them + pushed 4ebbb96 which adds CIDs, CARs, and short summary of each fixture to this IPIP.
This should be enough as a go-to resource for people who want to test with some reference data, or understand what the compliance tests do and where to find them.
|
2023-07-06 Kubo maintainer conversation: we need to ensure a conformance test involving inline CIDs. |
This change incorporates feedback from Adin, Rod and Juan: - bytes: #402 (review) - car-scope: #402 (comment) I really hope these names will be good enough, but I am running on artisan, recycled electrons so can do this all day :-)
We have no spec for signaling this, we may add it as opt-in later #402 (comment) #402 (comment)
Discussions in #402 illustrated deeper problem with CAR determinism, and we made a decision to remove its aspects from IPIP-402. Ref. #402 (comment)
There was a problem hiding this comment.
We have conformance tests, reference implementation is in boxo/gateway and shipped with Kubo 0.21 and the same API been used in Rhea/Saturn for a while.
Thank you to everyone involved, especially asking questions and pointing gaps – it all made the specification better, and also produced IPIP-412: ordered cars and IPIP-425: signaling features where we continue some of the work that started here.
I believe this is ready for being rattified.
Will flag this during IPFS-Implementers-Sync-2023-07-20.
|
There were no ratification concerns during IPFS Implementers sync, nor during the week after it. For future reference:
Thank you to everyone involved in this one ❤️ |
Goals
Improve the capabilities of a trustless HTTP gateway by adding additional capabilities to requests for CAR files.
The goal is to enable a client capable of translating/decoding CAR files to make a single request to a trustless gateway that in most case allows them to render the same output generated via a request to a trusted gateway. (or if not in a single request, as few requests as possible)
Additions That Are Non-Breaking
Adds two query parameters,
car-scopedag-scope=block|entity|allandbytesentity-bytes=from:to, specific to CAR file requests only, that allow a more precise specification of the type of DAG to download at the terminus of a content path (CID+path). The default values for these parameters match current behavior, such that if they are not specified, the return CAR file matches currently specified behavior.Breaking changes
In order to serve requests for content paths other than just a CID root in a trustless manner, we are requiring gateways to return intermediate blocks from the CID root to the path terminus as part of the returned CAR file. This was not previously well specified, but most major HTTP Gateway implementations are currently only returning blocks starting at the end of the content path and will need to add intermediate blocks.
closes #348 cc ipfs/in-web-browsers#128