Gateway: client opt-in JSON responses for directory listing

[lucidNTR]

Followup to #7431, that was closed but had the valuable feature of accessing json manifest of folder

requests to a "folder", especially the root of a folder published to ipns such as http://bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy.ipns.localhost/
should be served the same response as
https://ipfs.io/api/v0/ls?arg=bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy

IF the accept content type header is set to "application/json" that way there is no interference with existing gateway behaviour and with serving the index.html file at the root

[lidel]

Thank you for filling this @lucidNTR – below are my thoughts why we should consider this:

On mutability of directory listings

Directory listing is a special type of response returned by the gateway in that it is generated by the gateway itself. Even for immutable /ipfs/, the response can change, because we may update the HTML/CSS.

That is why for directories we don't return cache-control: (..) immutable header

If we add support for JSON response, then we could truly return an immutable response, instead of less permanent Etag-based cache control hint.

Needs: mandatory versioning

The only concern is that doing plain application/json would leak the current schema of /api/v0/ls into Gateway outside /api/v0. What happens when we change the API and introduce /api/v1/ls that produces different JSON?

I think we need to include API version in the request somehow:

• Accept: application/json;version=0
• Accept: application/vnd.ipfs.v0+json

so we are able to return clear error when client asks for API response in a version that is no longer supported. Having API version as a part of request would also let us return proper cache control headers that mark response as immutable under /ipfs/*

TLDR: this would be useful feature

Personally, I would be open to a PR that adds this feature, as long versioning is mandatory.
Thoughts @aschmahmann?

[aschmahmann]

@lidel I'm generally fine if we want to basically alias something we already do (e.g. /api/v0/ls) using accept headers.

A couple thoughts + questions though:

1. It doesn't seem like it's worth thinking too hard about caching in browsers since if you really wanted caching you could just use a local ipfs daemon.
2. I'm not very familiar with accept headers, if we were to remove this functionality in the future (or a particular gateway wanted to disable it) what would the result look like?

[lidel]

1. Avoiding unnecessary localhost requests still saves some resources, but I agree its diminishing returns in practice :)
2. If there is no support for Accept header, a regular HTML directory listing would be returned (current behavior)

[aschmahmann]

SGTM then.

If there is no support for Accept header, a regular HTML directory listing would be returned (current behavior)

I assume this is true even if we keep utilizing accept headers but that particular combination is no longer supported, right?

[lidel]

Yes, the value of Accept from request would need to be a direct match to produce non-HTML response.

[lidel]

Note to self: if we enable this via Accept header, make sure "version" format follows same convention as CAR format type from ipfs/in-web-browsers#170 (that is, if CAR itself ends up versioned - TBD).

[lidel]

#8234 could be relevant here:

if we find a way to represent dag-pb directory as dag-cbor document, then ?format=dag-json could be an elegant way for supporting JSON responses for directory listing

Going with dag-cbor representation not only solves versioning, but makes it possible to programmatically traverse unixfs