feat: getPath with carScope #8

olizilla · 2023-04-17T16:11:26Z

add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string

supports carScope to specify what blocks to return for the resolved dag

'all': return the entire dag starting at path. (default)
'block': return the block identified by the path.
'file': Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents.

see: web3-storage/freeway#33
see: web3-storage/freeway#34
see: ipfs/specs#402

TODO:

find out how to identify the boundaries of a unixfs hamt

...unixfs-exporter seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems loose... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42

License: MIT

add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string supports carScope to specify what blocks to return for the resolved dag - 'all': return the entire dag starting at path. (default) - 'block': return the block identified by the path. - 'file': Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents. see: web3-storage/freeway#33 see: web3-storage/freeway#34 TODO: - [] find out how to identify the boundaries of a unixfs hamt (unixfs-exported seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems risky... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42 License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla · 2023-04-17T16:22:00Z

@alanshaw plz to sanity check. also if you can point me at the right way to detect the boundary of the current hamt then that would save me some staring at things.

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla · 2023-04-18T11:07:07Z

package.json

@@ -49,8 +49,7 @@
    "blockstore-core": "^1.0.5",
    "ipfs-unixfs": "^11.0.0",
    "miniswap": "^2.0.0",
-    "standard": "^17.0.0",
-    "uint8arrays": "^3.0.0"


turns out the swiss-army-knife has toString and fromString under multiformats/bytes! who knew!

alanshaw

LGTM

index.js

alanshaw · 2023-04-19T12:41:09Z

index.d.ts

@@ -61,7 +75,7 @@ export declare class Dagula implements IDagula {
  /**
   * Emit nodes for all path segements and get UnixFS files and directories
   */
-  walkUnixfsPath (path: CID|string, options?: AbortOptions): Promise<UnixFSEntry>
+  walkUnixfsPath (path: CID|string, options?: AbortOptions): AsyncIterableIterator<UnixFSEntry & Block>


Is UnixFSEntry not already a Block?

No, it lacks the all important bytes

alanshaw · 2023-04-19T12:44:25Z

...unixfs-exporter seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems loose... what happens if the actual dir listing has 2 char long link names?

You just strip the first two characters, and if the result is empty then it's a HAMT node, and if it is not empty it is not.

olizilla · 2023-04-19T14:18:52Z

is it the case that there is no edge case here because the user cannot create a dir with an um-named link.

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla · 2023-04-26T10:02:13Z

index.js

+    for await (const item of this.walkUnixfsPath(cidPath, { signal: options.signal })) {
+      base = item
+      yield item
+    }


This should emit blocks not unixfs entries... where an entry is for a hamt we will only emit one thing, but we want to emit all the blocks that we traverse through the hamt. Also we don't want to emit a mix of unixfs entries and blocks.

Apologies I should have caught that in the review!

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla · 2023-04-28T14:51:20Z

@alanshaw ok the bits i missed first time around are now fixed, plz to eyeball

as you pointed out walkPath emits a single entry even for multi-block structures: fixed by passing in a custom blockstore that caches the traversed blocks so we can emit them all.
i was using the multiformats flavour block.links() function in my linkFilter, but that only understands ipld flavour link parsing, so dag-pb nodes link names are Links/<index>/Hash regardless of a Name field being present, so my hamtSearch wasn't working: fixed by special casing dagPB, which it would need to be anyway.

I've added tests for pathing through a sharded dir, as well as targeting a sharded dir as the end of the path as it's the most interesting case for carScope.

I've gone this root to ride on the work in unixfs-exporter for today, and get it working, hopefully without introducing any extra edge cases. I plan to remove this logic from here next week and put it in to https://github.com/web3-storage/ipfs-path which will become a block-centric rather than unixfsEntry-centric lib just for resolving ipfs paths... but first i want to land and ship verifiable paths and carScope to our gateway.

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

alanshaw

LGTM

index.js

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

🤖 I have created a release *beep* *boop* --- ## [6.0.0](v5.0.0...v6.0.0) (2023-05-01) ### ⚠ BREAKING CHANGES * support for multiple hash types ([#9](#9)) ### Features * add walkUnixfsPath to emit nodes for each path segment ([#7](#7)) ([6c0eed1](6c0eed1)) * getPath with carScope ([#8](#8)) ([a613b45](a613b45)) * support for multiple hash types ([#9](#9)) ([c147fdb](c147fdb)) ### Other Changes * publish from ci ([#13](#13)) ([5df4116](5df4116)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

olizilla added 2 commits April 17, 2023 17:01

chore: fix doc

b7898ee

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla requested a review from alanshaw April 17, 2023 16:20

olizilla added 2 commits April 18, 2023 11:52

chore: simplify tests

3b75841

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

chore: always use encode. remove uint8arrays dep

2efdcf9

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla commented Apr 18, 2023

View reviewed changes

alanshaw approved these changes Apr 19, 2023

View reviewed changes

olizilla added 2 commits April 19, 2023 20:28

feat: car-scope for hamt-sharded-directory

59c666b

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

chore: add test for hamt-sharded-dir

3acbe11

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla commented Apr 26, 2023

View reviewed changes

olizilla mentioned this pull request Apr 26, 2023

feat!: support for multiple hash types #9

Merged

olizilla added 2 commits April 28, 2023 10:05

Merge remote-tracking branch 'origin/main' into get-path

cdd530d

chore: fix hamt traversal

2769c69

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

chore: lint

e942e82

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

olizilla requested a review from alanshaw April 28, 2023 20:31

chore: typos

4aed254

License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>

alanshaw approved these changes May 1, 2023

View reviewed changes

index.js Outdated Show resolved Hide resolved

index.js Outdated Show resolved Hide resolved

Apply suggestions from code review

26f019a

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

olizilla merged commit a613b45 into main May 1, 2023
3 checks passed

olizilla deleted the get-path branch May 1, 2023 13:38

github-actions bot mentioned this pull request May 1, 2023

chore(main): release 6.0.0 #14

Merged

This was referenced May 2, 2023

Support car-scope param web3-storage/freeway#33

Closed

Support verifiable CAR responses web3-storage/freeway#32

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: getPath with carScope #8

feat: getPath with carScope #8

olizilla commented Apr 17, 2023 •

edited

olizilla commented Apr 17, 2023

olizilla Apr 18, 2023

alanshaw left a comment

alanshaw Apr 19, 2023

olizilla Apr 19, 2023

alanshaw commented Apr 19, 2023

olizilla commented Apr 19, 2023

olizilla Apr 26, 2023

alanshaw Apr 26, 2023

olizilla commented Apr 28, 2023 •

edited

alanshaw left a comment

feat: getPath with carScope #8

feat: getPath with carScope #8

Conversation

olizilla commented Apr 17, 2023 • edited

olizilla commented Apr 17, 2023

olizilla Apr 18, 2023

Choose a reason for hiding this comment

alanshaw left a comment

Choose a reason for hiding this comment

alanshaw Apr 19, 2023

Choose a reason for hiding this comment

olizilla Apr 19, 2023

Choose a reason for hiding this comment

alanshaw commented Apr 19, 2023

olizilla commented Apr 19, 2023

olizilla Apr 26, 2023

Choose a reason for hiding this comment

alanshaw Apr 26, 2023

Choose a reason for hiding this comment

olizilla commented Apr 28, 2023 • edited

alanshaw left a comment

Choose a reason for hiding this comment

olizilla commented Apr 17, 2023 •

edited

olizilla commented Apr 28, 2023 •

edited