From 9bc06e36228dbb6fd4793c3f0207010e86e4cbf8 Mon Sep 17 00:00:00 2001 From: achingbrain Date: Thu, 27 Nov 2025 13:02:12 +0200 Subject: [PATCH 1/5] IPIP-0524: Remove IPLD translation from IPFS HTTP Gateway The Accept-header driven translation between various IPLD formats is lossy, complicated, and often has us jumping through weird hoops to massage data into an acceptable format. Maybe we can do without it? Simplifying the spec will make creating server implementations easier and the translation can still be done by clients if they require it. Refs: https://github.com/ipfs/gateway-conformance/issues/200 --- src/http-gateways/path-gateway.md | 11 ++-- src/ipips/ipip-0524.md | 91 +++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 5 deletions(-) create mode 100644 src/ipips/ipip-0524.md diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index c207ae6d..8d807ec8 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -158,12 +158,13 @@ For example: - [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable raw [block](https://docs.ipfs.io/concepts/glossary/#block) to be returned - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned with implicit or explicit [`dag-scope`](https://specs.ipfs.tech/http-gateways/trustless-gateway/#dag-scope-request-query-parameter) for blocks at the terminus of the specified path and the blocks required to traverse path segments from root CID to the terminus. -- [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a DAG which name is the same as the terminus segment. Produces 400 Bad Request for content that is not UnixFS. -- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/). If the requested CID already has `dag-json` (0x0129) codec, data is validated as DAG-JSON before being returned as-is. Invalid DAG-JSON produces HTTP Error 500. -- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/). If the requested CID already has `dag-cbor` (0x71) codec, data is validated as DAG-CBOR before being returned as-is. Invalid DAG-CBOR produces HTTP Error 500. -- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec already is `json` (0x0200). Then, the raw JSON block can be returned as-is without any conversion. -- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec already is `cbor` (0x51). Then, the raw CBOR block can be returned as-is without any conversion. +- [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns a UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a DAG which name is the same as the terminus segment. Produces 406 Not Acceptable for content that is not UnixFS. +- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – Validates block data as DAG-JSON before returning it as-is. Invalid DAG-JSON produces 406 Not Acceptable. +- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – Validates block data as DAG-CBOR before returning it as-is. Invalid DAG-CBOR produces 406 Not Acceptable. +- [application/json](https://www.iana.org/assignments/media-types/application/json) – Validates block data as JSON before returning it as-is. Invalid JSON produces 406 Not Acceptable. +- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – Validates block data as CBOR before returning it as-is. Invalid CBOR produces 406 Not Acceptable. - [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record) – requests a verifiable :cite[ipns-record] to be returned. Produces 400 Bad Request if the content is not under the IPNS namespace, or contains a path. +- [text/html](https://html.spec.whatwg.org/) – returns a human-readable representation of the requested data which may include a link to download the raw data. :::note diff --git a/src/ipips/ipip-0524.md b/src/ipips/ipip-0524.md new file mode 100644 index 00000000..5ca03849 --- /dev/null +++ b/src/ipips/ipip-0524.md @@ -0,0 +1,91 @@ +--- +title: "IPIP-0524: Remove IPLD Data Model translations from Gateways" +date: 2025-11-27 +ipip: proposal +editors: + - name: Alex Potsides + github: achingbrain + url: https://achingbrain.net + affiliation: + name: Shipyard + url: https://ipshipyard.com +relatedIssues: + - https://github.com/ipfs/gateway-conformance/issues/200 +order: 524 +tags: ['ipips'] +--- + +## Summary + +Make IPFS HTTP Gateway responses easier to reason about by not requiring IPLD +Data Model translations + +## Motivation + +When sending an [Accept](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept) +header or [format](https://specs.ipfs.tech/http-gateways/path-gateway/#format-request-query-parameter) +query parameter to specify the response format of a request, the IPFS HTTP +Gateway specs [allow translation](https://specs.ipfs.tech/http-gateways/path-gateway/#accept-request-header) +of the requested content into the [IPLD Data Model](https://ipld.io/docs/data-model/). + +This adds significant complexity to HTTP Gateway implementations, since they +need to be able to translate between arbitrary data types and handle all the +various failure states. + +The conversions are also lossy due to differences in supported data types across +different formats so lack general-purpose utility and are ultimately something +that could be done on an interested client if required. + +## Detailed design + +An IPFS HTTP Gateway should make an attempt to verify that requested data can be +represented in the desired response format (e.g. it should attempt to parse the +block as JSON string or CBOR data when the relevant formats are requested, +though in the case of CBOR it does not need to support all [tags](https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml) +encountered in the block). + +If the block was loaded successfully but cannot be represented in the requested +format, the server should return a [406](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/406). + +This would be the outcome if, for example a DAG-JSON block was requested with +the `application/cbor` format. + +Where a human-readable rendering of the data is desired, the `text/html` format +can be requested. This would allow browsing DAG-PB data, for example. + +A [400](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/400) +may be returned if the request was invalid (for example an unsupported format +was requested). + +A [500](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/500) +may be returned in other circumstances. + +## Design rationale + +Simplifying the HTTP Gateway spec to remove these format translations and the +additional logic required makes it more straightforward to create new +implementations, and makes the returned data more transparent and so easier to +understand since the data is not modified to fit the output format. + +Clients that wish to translate between different data formats may request raw +blocks and do the translation themselves. + +### User benefit + +HTTP Gateways will be simpler to use having fewer options and they will be +encouraged to adapt their clients for their individual use-cases where more +advanced functionality is required. + +### Compatibility + +This is a breaking change. Data will no longer be translated between different +formats by the server and that logic will shift to interested clients. + +## Test fixtures + +Implementers can run the [gateway-conformance](https://github.com/ipfs/gateway-conformance/) +test suite. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From b1aae97c3186ad5be4771101831c092efffa5d5e Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 5 Feb 2026 18:48:28 +0100 Subject: [PATCH 2/5] docs: clarify IPIP-0524 validation is optional for matching codecs soften path-gateway.md wording: validation MAY be skipped when CID codec matches the requested format, since the response is equivalent to format=raw with a different Content-Type. --- src/http-gateways/path-gateway.md | 8 ++++---- src/ipips/ipip-0524.md | 8 +++++++- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index 8d807ec8..77c8a194 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -159,10 +159,10 @@ For example: - [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable raw [block](https://docs.ipfs.io/concepts/glossary/#block) to be returned - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned with implicit or explicit [`dag-scope`](https://specs.ipfs.tech/http-gateways/trustless-gateway/#dag-scope-request-query-parameter) for blocks at the terminus of the specified path and the blocks required to traverse path segments from root CID to the terminus. - [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns a UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a DAG which name is the same as the terminus segment. Produces 406 Not Acceptable for content that is not UnixFS. -- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – Validates block data as DAG-JSON before returning it as-is. Invalid DAG-JSON produces 406 Not Acceptable. -- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – Validates block data as DAG-CBOR before returning it as-is. Invalid DAG-CBOR produces 406 Not Acceptable. -- [application/json](https://www.iana.org/assignments/media-types/application/json) – Validates block data as JSON before returning it as-is. Invalid JSON produces 406 Not Acceptable. -- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – Validates block data as CBOR before returning it as-is. Invalid CBOR produces 406 Not Acceptable. +- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – Returns the block when CID codec is `dag-json`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. +- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – Returns the block when CID codec is `dag-cbor`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. +- [application/json](https://www.iana.org/assignments/media-types/application/json) – Returns the block when CID codec is `json`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. +- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – Returns the block when CID codec is `cbor`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. - [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record) – requests a verifiable :cite[ipns-record] to be returned. Produces 400 Bad Request if the content is not under the IPNS namespace, or contains a path. - [text/html](https://html.spec.whatwg.org/) – returns a human-readable representation of the requested data which may include a link to download the raw data. diff --git a/src/ipips/ipip-0524.md b/src/ipips/ipip-0524.md index 5ca03849..014076a1 100644 --- a/src/ipips/ipip-0524.md +++ b/src/ipips/ipip-0524.md @@ -1,6 +1,6 @@ --- title: "IPIP-0524: Remove IPLD Data Model translations from Gateways" -date: 2025-11-27 +date: 2026-02-05 ipip: proposal editors: - name: Alex Potsides @@ -44,6 +44,12 @@ block as JSON string or CBOR data when the relevant formats are requested, though in the case of CBOR it does not need to support all [tags](https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml) encountered in the block). +When the block's CID codec already matches the requested response format, +implementations MAY skip parsing the block data and return it as-is. This is +effectively equivalent to requesting `?format=raw` but with a codec-specific +`Content-Type` header. This optimization is particularly useful for root CID +requests that do not require path traversal. + If the block was loaded successfully but cannot be represented in the requested format, the server should return a [406](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/406). From b50dccee062e772ca206b6d9a8472920ef781835 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 5 Feb 2026 23:24:43 +0100 Subject: [PATCH 3/5] docs: reorganize IPIP-0524 design rationale and compatibility sections restructure to match idiomatic IPIP format: expand User benefit with cross-library interop argument, add Compatibility sub-headings (json independence, UnixFS exception, opt-in, implementation-defined), add Security section, move test list into Test fixtures. clarify `application/json` Accept handling for UnixFS files in path gateway spec. --- src/http-gateways/path-gateway.md | 10 ++- src/ipips/ipip-0524.md | 107 ++++++++++++++++++++++++------ 2 files changed, 94 insertions(+), 23 deletions(-) diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index 77c8a194..2db1242a 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -4,7 +4,7 @@ description: > The comprehensive low-level HTTP Gateway enables the integration of IPFS resources into the HTTP stack through /ipfs and /ipns namespaces, supporting both deserialized and verifiable response types. -date: 2025-10-13 +date: 2026-02-05 maturity: reliable editors: - name: Marcin Rataj @@ -36,6 +36,12 @@ thanks: affiliation: name: Protocol Labs url: https://protocol.ai/ + - name: Alex Potsides + github: achingbrain + url: https://achingbrain.net + affiliation: + name: Shipyard + url: https://ipshipyard.com xref: - url - trustless-gateway @@ -161,7 +167,7 @@ For example: - [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns a UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a DAG which name is the same as the terminus segment. Produces 406 Not Acceptable for content that is not UnixFS. - [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – Returns the block when CID codec is `dag-json`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. - [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – Returns the block when CID codec is `dag-cbor`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. -- [application/json](https://www.iana.org/assignments/media-types/application/json) – Returns the block when CID codec is `json`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. +- [application/json](https://www.iana.org/assignments/media-types/application/json) – For blocks with CID codec `json`, returns block data as `application/json`. Implementations MAY validate block data before returning. For deserialized UnixFS files that represent text files with valid JSON, implementations SHOULD allow serving the file content as `application/json` regardless of the CID codec being `dag-pb` or `raw`. SHOULD produce 406 Not Acceptable in all other cases. - [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – Returns the block when CID codec is `cbor`. Implementations MAY validate block data before returning. SHOULD produce 406 Not Acceptable when the CID codec does not match. - [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record) – requests a verifiable :cite[ipns-record] to be returned. Produces 400 Bad Request if the content is not under the IPNS namespace, or contains a path. - [text/html](https://html.spec.whatwg.org/) – returns a human-readable representation of the requested data which may include a link to download the raw data. diff --git a/src/ipips/ipip-0524.md b/src/ipips/ipip-0524.md index 014076a1..bce15f3c 100644 --- a/src/ipips/ipip-0524.md +++ b/src/ipips/ipip-0524.md @@ -9,6 +9,12 @@ editors: affiliation: name: Shipyard url: https://ipshipyard.com + - name: Marcin Rataj + github: lidel + url: https://lidel.org + affiliation: + name: Shipyard + url: https://ipshipyard.com relatedIssues: - https://github.com/ipfs/gateway-conformance/issues/200 order: 524 @@ -38,23 +44,18 @@ that could be done on an interested client if required. ## Detailed design -An IPFS HTTP Gateway should make an attempt to verify that requested data can be -represented in the desired response format (e.g. it should attempt to parse the -block as JSON string or CBOR data when the relevant formats are requested, -though in the case of CBOR it does not need to support all [tags](https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml) -encountered in the block). +When the block's CID codec matches the requested response format, +implementations MAY return the block as-is without parsing or validating it. +This is effectively equivalent to requesting `?format=raw` but with a +codec-specific `Content-Type` header. -When the block's CID codec already matches the requested response format, -implementations MAY skip parsing the block data and return it as-is. This is -effectively equivalent to requesting `?format=raw` but with a codec-specific -`Content-Type` header. This optimization is particularly useful for root CID -requests that do not require path traversal. +When the CID codec does not match the requested format, the gateway SHOULD +return a [406 Not Acceptable](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/406) +unless the server provides cross-codec conversion as an extra feature outside +of this specification. -If the block was loaded successfully but cannot be represented in the requested -format, the server should return a [406](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/406). - -This would be the outcome if, for example a DAG-JSON block was requested with -the `application/cbor` format. +For example, requesting a DAG-JSON block with the `application/cbor` format +would result in a 406 response. Where a human-readable rendering of the data is desired, the `text/html` format can be requested. This would allow browsing DAG-PB data, for example. @@ -78,19 +79,83 @@ blocks and do the translation themselves. ### User benefit -HTTP Gateways will be simpler to use having fewer options and they will be -encouraged to adapt their clients for their individual use-cases where more -advanced functionality is required. +For gateway operators and implementers, removing the requirement to perform +codec conversions server-side significantly reduces implementation complexity. + +For end users and application developers, the change makes gateway behavior +easier to reason about: a request either returns data deserialized according +to the rules of the CID's original codec, or fails with 406. This moves +conversion to userland, encouraging users to fetch raw blocks with +`?format=raw` and convert client-side, putting the application in full control +and producing deterministic results regardless of which gateway is used. + +This matters in practice because codec libraries do not behave identically. +[Cross-library dag-cbor tests (2026)](https://hyphacoop.github.io/dasl-testing/?group=tests-by-file&tag=dag-cbor) +show each implementation differs on edge cases like float handling, map key +ordering, and encoding strictness. Relying on server-side conversion means +the output depends on whichever library the gateway happens to use, which is +not a foundation for robust software. ### Compatibility -This is a breaking change. Data will no longer be translated between different -formats by the server and that logic will shift to interested clients. +Formally this is a breaking change: server-side IPLD Data Model translations +between codecs are removed. + +In practice, nobody could build reliable software +on top of conversion logic that behaved non-deterministically across gateways +written in different languages. Clients that needed data in a different +format often chose to fetch `?format=raw` and convert client-side already. + +This IPIP standardizes that robust real-world pattern and removes an +unreliable niche feature that has seen limited use. + +#### `json` and `dag-json` independence + +`application/json` and `application/vnd.ipld.dag-json` are now treated as +independent formats, each matching only their respective CID codec (`json` +0x0200 and `dag-json` 0x0129). The old behavior where `application/json` was +an alias for `application/vnd.ipld.dag-json` (falling back to dag-json +conversion) no longer applies. + +#### UnixFS interop exception for `Accept: application/json` + +Note: the codec match requirement and 406 behavior described above do not +apply to deserialized UnixFS file responses. Users commonly store valid JSON +as UnixFS files (with `dag-pb` or `raw` codec), and serving those files with +`Accept: application/json` is regular HTTP content serving, not codec +conversion. See the `application/json` entry in the +[Accept request header](https://specs.ipfs.tech/http-gateways/path-gateway/#accept-request-header) +section of the Path Gateway spec for normative requirements. + +#### Opt-in backward compatibility + +Implementations MAY offer an opt-in configuration flag to restore the old +codec conversion behavior for backward compatibility. + +#### Implementation-defined behavior + +- The content of the 406 error response body (e.g. actionable hints). +- Handling of `?format=json` / `Accept: application/json` on non-json-codec + content (like `dag-pb` UnixFS files). +- Whether to offer an opt-in flag for restoring codec conversion. +- Validation of block data when the CID codec matches the requested format. + +### Security + +No security implications. This change restricts gateway behavior (returning +406 instead of converting), which reduces attack surface. ## Test fixtures Implementers can run the [gateway-conformance](https://github.com/ipfs/gateway-conformance/) -test suite. +test suite v0.10 or later. The following behaviors are verified by the test suite: + +- Requesting a block in a format that differs from its CID codec (e.g. + `dag-pb` block with `?format=dag-json`) returns HTTP 406. +- Requesting a block in its native codec returns HTTP 200. +- `?format=raw` works for any codec. +- HTML rendering (`Accept: text/html`) of DAG-JSON/DAG-CBOR blocks is not + codec conversion and remains part of the spec. ### Copyright From 2bb0197d0c1ed2b87f6b180e49066e9d370c79c7 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 5 Feb 2026 23:56:22 +0100 Subject: [PATCH 4/5] docs: add 406 status code section, remove stale conversion wording add missing 406 Not Acceptable subsection to the Response Status Codes, remove "or converted to DAG-CBOR/DAG-JSON" from the Response Payload section to align with IPIP-0524 removing server-side codec conversions. --- src/http-gateways/path-gateway.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index 2db1242a..dfd95010 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -346,6 +346,23 @@ responses (such as CAR), once HTTP 200 OK status is sent, gateways cannot change it. If a child block is missing during streaming, the gateway SHOULD terminate the stream. Clients MUST verify response completeness. +### `406` Not Acceptable + +Returned when the requested response format does not match the CID's codec +and the gateway does not perform cross-codec conversion. + +For example, requesting `?format=dag-json` on a `dag-cbor` block, or +`?format=dag-cbor` on a `dag-pb` block, SHOULD return a 406 response. + +Similarly, requesting `?format=tar` for content that is not UnixFS SHOULD +return 406. + +Implementations MAY include an actionable hint in the response body (e.g., +suggesting the client fetch the raw block with `?format=raw` and convert +client-side). + +See :cite[ipip-0524] for details. + ### `410` Gone Error to indicate that request was formally correct, but this specific Gateway @@ -760,10 +777,10 @@ By default, implicit deserialized response type is based on `Accept` header and - Bytes representing a CBOR file, see [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) - Works exactly the same as `raw`, but returned `Content-Type` is `application/cbor` - DAG-JSON (0x0129) - - If the `Accept` header includes `text/html`, implementation should return a generated HTML with options to download DAG-JSON as-is, or converted to DAG-CBOR. + - If the `Accept` header includes `text/html`, implementation should return a generated HTML with an option to download DAG-JSON as-is. - Otherwise, response works exactly the same as `raw` block, but returned `Content-Type` is [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) - DAG-CBOR (0x71) - - If the `Accept` header includes `text/html`: implementation should return a generated HTML with options to download DAG-CBOR as-is, or converted to DAG-JSON. + - If the `Accept` header includes `text/html`: implementation should return a generated HTML with an option to download DAG-CBOR as-is. - Otherwise, response works exactly the same as `raw` block, but returned `Content-Type` is [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) The following response types require an explicit opt-in, can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header: From d4999d430cfc5dccdd678ccd00ba01bc7e090f6b Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Feb 2026 18:33:22 +0100 Subject: [PATCH 5/5] docs: add real-world traffic data, clarify title add ?format= usage breakdown from ipfs.io/dweb.link public gateways showing that no real-world traffic depends on cross-codec conversion. rename IPIP to use clearer terminology. --- src/ipips/ipip-0524.md | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/src/ipips/ipip-0524.md b/src/ipips/ipip-0524.md index bce15f3c..a833a9e6 100644 --- a/src/ipips/ipip-0524.md +++ b/src/ipips/ipip-0524.md @@ -1,6 +1,6 @@ --- -title: "IPIP-0524: Remove IPLD Data Model translations from Gateways" -date: 2026-02-05 +title: "IPIP-0524: Remove cross-codec conversion from HTTP Gateways" +date: 2026-02-06 ipip: proposal editors: - name: Alex Potsides @@ -109,6 +109,43 @@ format often chose to fetch `?format=raw` and convert client-side already. This IPIP standardizes that robust real-world pattern and removes an unreliable niche feature that has seen limited use. +#### Real-world `?format=` usage on `ipfs.io` and `dweb.link` + +A 24-hour sample of traffic on the `ipfs.io` and `dweb.link` public gateways +(Feb 2026) shows that only 4.5% of all requests use the `?format=` query +parameter, and the vast majority ask for `json`: + +| `?format=` value | % of requests with `format=` | +|------------------|------------------------------| +| `json` | 99.11% | +| `raw` | 0.86% | +| `dag-json` | 0.02% | +| `car` | 0.01% | +| other | <0.01% | + +Note: `ipfs.io` and `dweb.link` serve deserialized responses. Trustless +verifiable requests (`?format=raw`, `?format=car`) are redirected to +`trustless-gateway.link`, which is why those formats appear so rarely here. + +Looking at what those `?format=json` requests actually point at tells the +real story. The CID codec of the requested blocks breaks down as follows: + +| CID codec of requested block | % of `?format=json` | +|------------------------------|---------------------| +| `dag-pb` (CIDv0 `Qm...`) | 60.0% | +| `dag-pb` (CIDv1 `bafy...`) | 21.4% | +| `raw` (`bafk...`) | 18.6% | + +100% of `?format=json` requests are for blocks with `dag-pb` or `raw` codec. +None target the `json` codec (0x0200). In other words, these clients are +reading regular JSON files stored as UnixFS, not asking the gateway to convert +between IPLD codecs. The gateway serves them as plain HTTP file responses, +which is covered by the UnixFS interop exception described later in this IPIP. + +The remaining formats (`dag-json` and `car`) together account for less than +0.04% of `?format=` requests and do not depend on cross-codec conversion +either, since they request data in the block's native codec. + #### `json` and `dag-json` independence `application/json` and `application/vnd.ipld.dag-json` are now treated as