IPIP-425: Signaling Features on HTTP Gateways #425

lidel · 2023-07-06T18:20:45Z

merge IPIP-402 and IPIP-412 first (in order), this PR is based on top of them
gateway conformance test

Initial version to kick-off discussion

lidel · 2023-07-06T20:21:03Z

src/http-gateways/path-gateway.md

+- `trustless-gateway` for :cite[trustless-gateway]
+  - `path-proof` indicates support for returning parent blocks up to the terminus element
+  - `car-version=1|2` indicates CAR support
+  - `dag-scope=block|entity|all` from :cite[ipip-0402]
+  - `entity-bytes` from :cite[ipip-0402], implies support for `dag-scope=entity` as well


@hannahhoward @alanshaw @olizilla I know you want a way for signaling support for partial cars and range requests between Lassie and .storage to avoid expensive feature-sniffing.

With this IPIP, signaling support for partial cars WITHOUT range requests woudl look like this:

Ipfs-Gateway-Features: dag-scope=block|entity|all

and if you add support for entity-bytes at some point:

Ipfs-Gateway-Features: dag-scope=block|entity|all, entity-bytes

lidel · 2023-07-06T20:24:07Z

src/http-gateways/path-gateway.md

+<!-- TODO do we want these too?
+- `multibase` list of prefixes, indicates which multibase encoding are supported in CIDs
+- `multihash` indicates which hash functions are supported in CIDs
+- `multicodec` indicates which codecs are supported in CIDs
+-->


❓

Thoughts on adding these or even more?

The need for list of supported IPLD codecs was raised by ipfs-chromium and Capyloon in #402 (comment), having the other two would not hurt too.

My only constraint would be to use numeric (decimal or hex) codes, to avoid problems if we ever do this again.

cc @aschmahmann as you noted in 402:

Not specifically related to the trustless gateway, but if we're adding OPTIONS support we might want to be able to discover things like supported hash functions and IPLD codecs.

Would below work? is there a better way of representing this?

multihash=0x12|0x13|etc # identity + goodset from https://github.com/ipfs/boxo/blob/cfad09d7156efa2f09822d620cacb2423d884067/verifcid/validate.go#L17 multicodec=0x51|0x55|0x70|0x71|0x72|0x0129|0x0200 # based on boxo/gateway

too much information, let's start simple for now

MarcoPolo · 2023-07-11T18:37:39Z

src/ipips/ipip-0425.md

+
+### Alternatives
+
+- Exposing the list of suported features via `GET


I agree that using OPTIONS here makes more sense for this use case than using .well-known. My argument is that this the HTTP Gateway API is an application-level protocol, and these are features for this application. .well-known on the other hand is for metadata about an origin, not a specific application.

This is also why libp2p/specs#508 uses .well-known for metadata on where application protocols are mounted. This is metadata about the origin.

lidel · 2023-07-18T11:39:48Z

src/http-gateways/path-gateway.md

+- `dnslink-gateway` for :cite[dnslink-gateway] support based on `Host` header
+- `ipns` indicating :cite[ipns-record] support on `/ipns/` content paths
+- `dnslink` indicating [DNSLink](https://dnslink.dev) support on `/ipns/` content paths
+


Another thing that could be useful is max block size:

Suggested change

- `max-block-size` to indicate what is the biggest block that can be retrieved by this gateway (default: 1MiB? 2MiB? -- whatever we currently have in boxo/gateway)

BigLep

Thanks for writing this up. Makes sense and seems like a great step forward for letting nodes announce the functionality they support.

Rather than embedding all of this in the header, is there any precedent for putting it into a body with JSON? (Just was wondering in case easier to work with and avoids any max header length situations?)

BigLep · 2023-07-25T17:54:31Z

src/ipips/ipip-0425.md

+that checks if the CORS protocol is understood and a server is aware using
+specific methods and headers.
+
+The `OPTIONS` request if often sent by web browser anyway, so we would not be


I assume this oten a heavily cachable response so we'd be hitting the browser cache.

BigLep · 2023-07-25T17:54:44Z

src/ipips/ipip-0425.md

+  - requires additional HTTP GET, while in some cases HTTP OPTIONS is already
+    sent (e.g., [browser's preflight
+    requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#preflighted_requests)),
+    and would not  introduce no overhead


(suggestion not working for me for some reason)

"and would not introduce overhead"

BigLep · 2023-07-25T17:55:22Z

src/ipips/ipip-0425.md

+      `.well-known` we need to make multiple GET requests for different
+      `.well-known/..` paths, vs sending a single HTTP OPTIONS and getting all
+      headers related to pre-existing path
+  - HTTP headers can be added as the rrqueest passes via reverse proxies, CDNs


lidel · 2023-08-18T17:37:35Z

src/http-gateways/path-gateway.md

+- `dnslink-gateway` for :cite[dnslink-gateway] support based on `Host` header
+- `ipns` indicating :cite[ipns-record] support on `/ipns/` content paths
+- `dnslink` indicating [DNSLink](https://dnslink.dev) support on `/ipns/` content paths
+


Another thing we could signal: if gateway is recursive (fetching content if not available locally) or not (cc @aschmahmann @Jorropo)

good idea but hold off until we have a real use-case for it reckon - unless you can conjure one now?

rvagg · 2023-10-11T05:22:48Z

src/http-gateways/path-gateway.md

@@ -542,6 +542,46 @@ Optional, present in certain response types:
  non-executable binary response types are not used in `<script>` and `<style>`
  HTML tags.

+### `Ipfs-Gateway-Features` (response header)
+
+Optional, this header SHOULD be only returned in response to HTTP `OPTIONS` request.


(just tuning into this PR now that I care about signalling, sorry!)

Whatever this header is, I think it should also be returned when a server doesn't implement a feature requested, either via Accept header or other means and in that case the server should return a 415. That way I can parse the list of Accept content types, and be strict about the parameters, and if I don't find one that I fully support, return a 415 with this signal that says what I do support.

That way, a client can optimistically make a request, but be prepared to downgrade if it doesn't get what it wants. Rather than needing to OPTIONS first, thereby saving a round-trip.

rvagg · 2023-10-11T05:23:39Z

src/http-gateways/path-gateway.md

+
+- `trustless-gateway` for :cite[trustless-gateway]
+  - `path-proof` indicates support for returning parent blocks up to the terminus element
+  - `car-version=1|2` indicates CAR support


Suggested change

- `car-version=1|2` indicates CAR support

- `car-version=1` indicates CAR support

we should strictly do v1 for now, v2 doesn't make sense for this (yet), but it might be worth signalling this so we can also add 3 etc. when we get to it

rvagg · 2023-10-11T05:24:08Z

src/http-gateways/path-gateway.md

+  - `dag-scope=block|entity|all` from :cite[ipip-0402]
+  - `entity-bytes` from :cite[ipip-0402], implies support for `dag-scope=entity` as well
+  - `car-block-order=dfs` from :cite[ipip-0412]
+  - `car-block-dupes=y|n`  from :cite[ipip-0412]


Suggested change

- `car-block-dupes=y|n` from :cite[ipip-0412]

- `dups=y|n` from :cite[ipip-0412]

rvagg · 2023-10-11T05:24:17Z

src/http-gateways/path-gateway.md

+  - `car-version=1|2` indicates CAR support
+  - `dag-scope=block|entity|all` from :cite[ipip-0402]
+  - `entity-bytes` from :cite[ipip-0402], implies support for `dag-scope=entity` as well
+  - `car-block-order=dfs` from :cite[ipip-0412]


Suggested change

- `car-block-order=dfs` from :cite[ipip-0412]

- `order=dfs` from :cite[ipip-0412]

rvagg · 2023-10-11T05:28:05Z

I ~like this, but would rather its feature set be somewhat limited within the bounds of 412 plus anything else we can come up with a legitimate use-case for signalling. I don't really buy the multicodec thing yet—I know why it's a problem, but how does signalling it help at this stage? The tighter we scope it the easier it is to agree and ship, and I'd really like to turn this "ignore" into an error so evolution is easier: https://github.com/ipld/go-trustless-utils/blob/27d979b92be45fa189c9bef4a351e5cb02d95285/http/parse.go#L187-L189

rvagg · 2023-10-11T05:39:06Z

My original thought for this was to just X-Supported-Content-Types and enumerate all the permutations of Accept that would be acceptable. That gets kind of combinatorially explosive as we add more items; although we could parameterize or regex it:

X-Supported-Content-Types: application/vnd.ipld.(raw|car);dups=(y|n);order=dfs;version=1

But I also suppose there's things in here that are not content-type related.

rvagg · 2023-10-11T05:40:24Z

src/http-gateways/path-gateway.md

+- `trustless-gateway` for :cite[trustless-gateway]
+  - `path-proof` indicates support for returning parent blocks up to the terminus element


Suggested change

- `trustless-gateway` for :cite[trustless-gateway]

- `path-proof` indicates support for returning parent blocks up to the terminus element

- `trustless-gateway` for :cite[trustless-gateway]

- `formats=car|raw` indicates whether CAR and/or raw block responses are supported

- `path-proof` indicates support for returning parent blocks up to the terminus element

ipip-425: Signaling Features on HTTP Gateways

0eacf6a

Initial version to kick-off discussion

lidel force-pushed the ipip/http-options-on-gateways branch from 3a0785a to 0eacf6a Compare July 6, 2023 20:09

lidel mentioned this pull request Jul 6, 2023

IPIP-412: Signaling Block Order in CARs on HTTP Gateways #412

Merged

3 tasks

lidel commented Jul 6, 2023

View reviewed changes

lidel mentioned this pull request Jul 6, 2023

IPIP-402: Partial CAR Support on Trustless Gateways #402

Merged

lidel commented Jul 6, 2023

View reviewed changes

lidel changed the title ~~IPIP-425: HTTP OPTIONS for Signaling Features on Gateways~~ IPIP-425: Signaling Features on HTTP Gateways Jul 6, 2023

BigLep assigned lidel Jul 6, 2023

ipip-425: explain why not .well-known path

9cfc305

MarcoPolo reviewed Jul 11, 2023

View reviewed changes

lidel mentioned this pull request Jul 17, 2023

Research ways we could leverage /webtransport peers little-bear-labs/ipfs-chromium#42

Open

lidel commented Jul 18, 2023

View reviewed changes

BigLep reviewed Jul 25, 2023

View reviewed changes

BigLep mentioned this pull request Aug 3, 2023

Release 0.22 ipfs/kubo#9911

Closed

Base automatically changed from ipip-car-order-signaling to main August 4, 2023 13:59

MarcoPolo mentioned this pull request Aug 9, 2023

libp2p + HTTP an exploratory refactor ipni/go-libipni#102

Closed

lidel mentioned this pull request Aug 17, 2023

Gateway: define a protocol.ID for the gateway protocol. #433

Closed

lidel commented Aug 18, 2023

View reviewed changes

BigLep mentioned this pull request Sep 7, 2023

[🏆 Golden path scenario] Browsers can reliably retrieve content from any modern Kubo node providing content ipfs/helia#255

Open

aschmahmann mentioned this pull request Sep 7, 2023

Serve Trustless IPFS HTTP Gateway API via libp2p (non-experimental) ipfs/kubo#10118

Open

6 tasks

Jorropo mentioned this pull request Oct 11, 2023

IPIP-0445: Option to Skip Raw Blocks in Gateway Responses #445

Draft

rvagg reviewed Oct 11, 2023

View reviewed changes

2color mentioned this pull request Feb 29, 2024

feat: add explicit support for subdomain gateways ipfs/helia#439

Open

3 tasks

lidel mentioned this pull request Mar 7, 2024

feat: add sessions to trustless gateways ipfs/helia#459

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IPIP-425: Signaling Features on HTTP Gateways #425

IPIP-425: Signaling Features on HTTP Gateways #425

lidel commented Jul 6, 2023 •

edited by hacdias

lidel Jul 6, 2023

lidel Jul 6, 2023 •

edited

lidel Jul 12, 2023

rvagg Oct 11, 2023

MarcoPolo Jul 11, 2023

lidel Jul 18, 2023

BigLep left a comment

BigLep Jul 25, 2023

BigLep Jul 25, 2023

BigLep Jul 25, 2023

lidel Aug 18, 2023 •

edited

rvagg Oct 11, 2023

rvagg Oct 11, 2023

rvagg Oct 11, 2023

rvagg Oct 11, 2023

rvagg Oct 11, 2023

rvagg commented Oct 11, 2023

rvagg commented Oct 11, 2023

rvagg Oct 11, 2023


		### Alternatives

		- Exposing the list of suported features via `GET


	- `max-block-size` to indicate what is the biggest block that can be retrieved by this gateway (default: 1MiB? 2MiB? -- whatever we currently have in boxo/gateway)

	- `car-version=1\|2` indicates CAR support
	- `car-version=1` indicates CAR support

	- `car-block-dupes=y\|n` from :cite[ipip-0412]
	- `dups=y\|n` from :cite[ipip-0412]

	- `car-block-order=dfs` from :cite[ipip-0412]
	- `order=dfs` from :cite[ipip-0412]

		- `trustless-gateway` for :cite[trustless-gateway]
		- `path-proof` indicates support for returning parent blocks up to the terminus element

IPIP-425: Signaling Features on HTTP Gateways #425

Are you sure you want to change the base?

IPIP-425: Signaling Features on HTTP Gateways #425

Conversation

lidel commented Jul 6, 2023 • edited by hacdias

Choose a reason for hiding this comment

lidel Jul 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BigLep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lidel Aug 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rvagg commented Oct 11, 2023

rvagg commented Oct 11, 2023

Choose a reason for hiding this comment

lidel commented Jul 6, 2023 •

edited by hacdias

lidel Jul 6, 2023 •

edited

lidel Aug 18, 2023 •

edited