Send HTTP probes with both Accept (header) and ?format (query param)

[lidel]

iiuc right now, the probes do not set either Accept header not ?format= query parameter, and blindly assume that the response will be a CAR.

Problem

Probing as it is now, makes implicit assumptions about content types. This is very brittle and will produce errors on SPs that correctly implement specification, and/or default to block instead of CAR.

This will get worse over time, as more and more clients shifts towards Block (application/vnd.ipld.raw) being the lowest common denominator, and a CAR (application/vnd.ipld.car) being an optional performance optimization.

Example 1: misbehaving SP

There are SP that are not implementing specs correctly, and ignore ?format URL query parameter, returning a CAR when a single block is requested.

For example, below request for a raw block (?format=raw) returns a CAR instead:

$ curl "https://f010479.twinquasar.io/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi?format=raw" -I -X GET -s | grep -i -e http -e vnd.ipld
HTTP/2 200
content-type: application/vnd.ipld.car;version=1;order=dfs;dups=y

Passing both Accept and ?format is a reliable way of dealing with all permutations:

$ curl "https://f010479.twinquasar.io/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi?format=raw" -I -X GET -s -H "Accept: application/vnd.ipld.raw" | grep -i -e http -e vnd.ipld
HTTP/2 200
content-type: application/vnd.ipld.raw

This class of errors may go in other direction: the default response may be something other than block or a CAR.

Example 2: misbehaving probe code

getRetrievalUrl produces URL without explicit ?format=car:

spark-checker/lib/spark.js

Line 344 in 799cc5e

return `${baseUrl}/ipfs/${cid}?dag-scope=block`

This depends on some old Boost code? Will break if default response type is something other than CAR.

Suggested fix

Probes should be made with explicit content type ask (both Accept header and ?format),
mirroring most popular client implementations (boxo in GO / helia in JS) and avoiding false negatives and false positives.

References

• https://specs.ipfs.tech/http-gateways/trustless-gateway/#dedicated-probe-paths
• HTTP retrieval proposal ipfs/boxo#747
• https://www.npmjs.com/package/@helia/http
• https://github.com/ipfs/kubo/blob/master/docs/config.md#httpretrieval