meta: GRAPH_BACKEND fixes and latency improvements

[lidel]

This is a meta issue for tracking things we can do to improve correctness/latency.

Spec conformance

Tasks

Beta Give feedback

1. IPIP-402 conformance tests for dag-scope=block|entity|all → gateway-conformance#75 added tests
2. IPIP-402 conformance tests for entity-bytes=from:to → gateway-conformance tests
3. IPIP-402 conformance tests for entity-bytes with missing block scenario to confirm /ipfs/cid?format=car&bytes=from:to works fast for big UnixFS file, and does not block until the entire thing is prefetched to some L1 → gateway-conformance#85 added tests
4. Run gateway-conformance on Saturn CI: Run conformance test in CI filecoin-saturn/L1-node#447
5. Saturn L1 pass dag-scope tests PR
6. (TODO) Saturn L1 DOES NOT pass entity-bytes range request tests CI fails here

Perf. Improvements

Tasks

Beta Give feedback

1. IPIP-402: Partial CAR Support on Trustless Gateways specs#402
   +0
   
2. IPIP-412: Signaling Block Order in CARs on HTTP Gateways specs#412
   +0
   
3. (PRECONDITION) @aschmahmann IPIP-402 implementation in Boxo (Boxo gateway should support IPIP-402 parameters boxo#283) so can verify tests and performance improvement ideas without doing a Rhea bifrost-gateway deployment.
4. (OPTIMIZATION) @aschmahmann feat: IPIP-402 based backpressure #160
5. (OPTIMIZATION) @aschmahmann implement CAR based Gateway implementation #62
6. (FIX) ensure we use HTTP/2 when possible (fix: ForceAttemptHTTP2 despite custom TLSClientConfig #98 or Re-enable http/2 #95)
7. (FIX) @lidel correctness degrading over time (Investigate why success rate gets worse over time #92)
8. (OPTIONAL LATENCY FIX) @lidel / @hacdias / @aschmahmann 30m timeout for CARs is not enough, implement timeout-since-last-bytes-received that drops request/response processing if no new bytes were sent by L1 for [tbd] amount of time – stub in Research / improve timeouts on stalled L1 responses #91
9. (OPTIONAL CORRECTNESS FIX) @lidel / @aarshkshah1992 (content routing optimization, may improve correctness) when fetching path remainders or single blocks, pass content path affinity to L1 and forward to Lassie in Ipfs-Path-Affinity HTTP header as a hint to augument content routing in situations where internal blocks are not announced on DHT/IPNI
10. (OPTIONAL LATENCY FIX) (perf optimization) make caboose do more RAPIDE-like retrieval and split DAG into parallel fetches?

Note: these things aren't blocking for enabling GRAPH_BACKEND=true. The blocker is #92

[BigLep]

2023-05-09 conversation: @BigLep to create the issue for implementing IPIP in Boxo. Needed for:

• Conformance Tests
• CAR gateway implementation vetting
• Boost (since use Boxo gateway)

[lidel]

• added basic test script scenarios and information about needed fixtures to IPIP-402: Partial CAR Support on Trustless Gateways
• there is also work on deterministic CAR responses in IPIP-412: Signaling Block Order in CARs on HTTP Gateways (something Saturn wants to do internally, useful to standardize for light clients as well)
• Adin is focusing on implementing the above IPIPs in boxo, to use it in Kubo and Boost + use it as reference implementation to write gateway-conformance tests.

[BigLep]

2023-06-22 maintainer conversation: @aschmahmann is going to update the content of this issue to describe what's remaining that we're going to do.

[BigLep]

2023-10-03 conversation: we did all the mandatory things.