From 8a175f6b3cb745cc15f0e41172f6034a0a8f9af5 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 12 Oct 2022 13:58:50 +0200 Subject: [PATCH 1/9] ipip: error handling in gateways --- IPIP/0000-gateway-error-handling.md | 66 +++++++++++++++++++++++++++++ http-gateways/PATH_GATEWAY.md | 28 +++++++----- 2 files changed, 84 insertions(+), 10 deletions(-) create mode 100644 IPIP/0000-gateway-error-handling.md diff --git a/IPIP/0000-gateway-error-handling.md b/IPIP/0000-gateway-error-handling.md new file mode 100644 index 000000000..28d52ec97 --- /dev/null +++ b/IPIP/0000-gateway-error-handling.md @@ -0,0 +1,66 @@ +# IPIP 0000: Error Handling in HTTP Gateways + +- Start Date: 2022-10-12 +- Related Issues: + - [ipfs/kubo/pull/9333](https://github.com/ipfs/kubo/pull/9333) + - [mdn/browser-compat-data/issues/14703](https://github.com/mdn/browser-compat-data/issues/14703) + +## Summary + +Ensure error handling in web gateways is clear and consistent. + +## Motivation + +Web gateways provide different functionalities where users can download files. +The download of this files is streamed from the server to the client using HTTP. +However, there is no good way of presenting to the client an error that happens +during the stream. + +For example, if during the download of a TAR file, the server detects some error +and is not able to continue, the user can get a valid, yet incomplete TAR. However, +the user will not know that the TAR is incomplete. By introducing consistent error +handling, the server attempts to notify the user. + +## Detailed design + +If the server encounters an error before streaming the contents to the client, +the server should fail with the respective `4xx` or `5xx` HTTP status code. + +If the server encounters an error while streaming the contents, the server must +force-close the HTTP connection to the user. This way, the user will receive a +network error, making it clear that the downloaded file is not valid. + +## Test fixtures + +There are no relevant test fixures for this IPIP. + +## Design rationale + +Before starting to stream the body of the response, the server is able to set +an HTTP status code for the error. However, after the HTTP headers are set +and the body started being streamed, there are no clear ways in the HTTP +specification to show an error. Since the gateway is browser-first, it is +important to show an error and avoid users receiving an incomplete file. +Therefore, the server can force-close the HTTP connection, leading to a network +error. This tells the user that an error happened. + +### User benefit + +The user will know that an error happened while receiving the file. Otherwise, +the user might receive incomplete, but still valid, files that could be mistaken +but the real file. + +### Compatibility + +This RFC is backwards compatible. + +### Alternatives + +Using [`Trailer`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer) HTTP headers +was considered. However, trailer headers are [not supported in browsers](https://github.com/mdn/browser-compat-data/issues/14703). +In addition, even if trailer headers were supported in browsers, there is no clear +standard for which header would be used to indicate errors. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 35ad96dd4..42d431705 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -1,6 +1,6 @@ # Path Gateway Specification -![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) +![Status: Work In Progress](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) **Authors**: @@ -194,7 +194,6 @@ blocks. Gateway implementations SHOULD be smart enough to require only the minimal DAG subset necessary for handling the range request. - NOTE: for more advanced use cases such as partial DAG/CAR streaming, or non-UnixFS data structures, see the `selector` query parameter [proposal](https://github.com/ipfs/go-ipfs/issues/8769). @@ -256,7 +255,6 @@ for more details. - This is a powerful primitive that allows for fetching subsets of data in specific order, either as raw bytes, or a CAR stream. Think “HTTP range requests”, but for IPLD, and more powerful. --> - # HTTP Response ## Response Status Codes @@ -372,7 +370,6 @@ and CDNs, implementations should base it on both CID and response type: should be based on requested range in addition to CID and response format: `Etag: "bafy..foo.0-42` - ### `Cache-Control` (response header) Used for HTTP caching. @@ -433,6 +430,7 @@ or optional [`filename`](#filename-request-query-parameter) parameter) and magic bytes to improve the utility of produced responses. For example: + - detect plain text file and return `Content-Type: text/plain` instead of `application/octet-stream` - detect SVG image @@ -446,6 +444,7 @@ Returned when `download`, `filename` query parameter, or a custom response The first parameter passed in this header indicates if content should be displayed `inline` by the browser, or sent as an `attachment` that opens the “Save As” dialog: + - `Content-Disposition: inline` is the default, returned when request was made with `download=false` or a custom `filename` was provided with the request without any explicit `download` parameter. @@ -457,13 +456,14 @@ The remainder is an optional `filename` parameter that will be prefilled in the NOTE: when the `filename` includes non-ASCII characters, the header must include both ASCII and UTF-8 representations for compatibility with legacy user -agents and existing web browsers. +agents and existing web browsers. To illustrate, `?filename=testтест.pdf` should produce: `Content-Disposition inline; filename="test____.jpg"; filename*=UTF-8''test%D1%82%D0%B5%D1%81%D1%82.jpg` - - ASCII representation must have non-ASCII characters replaced with `_` - - UTF-8 representation must be wrapped in Percent Encoding ([RFC 3986, Section 2.1](https://www.rfc-editor.org/rfc/rfc3986.html#section-2.1)). - - NOTE: `UTF-8''` is not a typo – see [Examples in RFC5987](https://datatracker.ietf.org/doc/html/rfc5987#section-3.2.2) + +- ASCII representation must have non-ASCII characters replaced with `_` +- UTF-8 representation must be wrapped in Percent Encoding ([RFC 3986, Section 2.1](https://www.rfc-editor.org/rfc/rfc3986.html#section-2.1)). + - NOTE: `UTF-8''` is not a typo – see [Examples in RFC5987](https://datatracker.ietf.org/doc/html/rfc5987#section-3.2.2) `Content-Disposition` must be also set when a binary response format was requested: @@ -510,8 +510,9 @@ This header is more widely used in [SUBDOMAIN_GATEWAY.md](./SUBDOMAIN_GATEWAY.md Gateway MUST return a redirect when a valid UnixFS directory was requested without the trailing `/`, for example: + - response for `https://ipfs.io/ipns/en.wikipedia-on-ipfs.org/wiki` - (no trailing slash) will be HTTP 301 redirect with + (no trailing slash) will be HTTP 301 redirect with `Location: /ipns/en.wikipedia-on-ipfs.org/wiki/` ### `X-Ipfs-Path` (response header) @@ -614,7 +615,7 @@ IPLD data, starting from that data which the CID identified. **Note:** Other types of gateway may allow for passing CID by other means, such as `Host` header, changing the rules behind path splitting. (See [SUBDOMAIN_GATEWAY.md](./SUBDOMAIN_GATEWAY.md) -and [DNSLINK_GATEWAY.md](./DNSLINK_GATEWAY.md)). +and [DNSLINK_GATEWAY.md](./DNSLINK_GATEWAY.md)). ### Traversing remaining path @@ -628,6 +629,7 @@ low level logical pathing from IPLD: ### Handling traversal errors Gateway MUST respond with HTTP error when it is not possible to traverse the requested content path: + - [`404 Not Found`](#404-not-found) should be returned when the root CID is valid and traversable, but the DAG it represents does not include content path remainder. - Error response body should indicate which part of immutable content path (`/ipfs/{cid}/path/to/file`) is missing @@ -655,6 +657,7 @@ Implementations are encouraged to support pluggable denylists to allow IPFS node operators to opt into not hosting previously flagged content. Gateway MUST respond with HTTP error when requested CID is on any of active denylists: + - [410 Gone](#410-gone) returned when CID is denied for non-legal reasons, or when the exact reason is unknown - [451 Unavailable For Legal Reasons](#451-unavailable-for-legal-reasons) returned when denylist indicates that content was blocked on legal basis @@ -694,3 +697,8 @@ The usual optimizations involve: limiting the cost of a single page load. - The downside of this approach is that it will always be slower than skipping child block resolution. + +## Streaming errors + +Gateways MUST force-close HTTP connections if they detect an error while +streaming a file to avoid that users receive incomplete, yet valid, files. From a84fb63b6163a25e18ef57dcc0def7f34c045256 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 12 Oct 2022 14:03:29 +0200 Subject: [PATCH 2/9] ipip: cleanup streaming and should --- IPIP/0000-gateway-error-handling.md | 6 +++--- http-gateways/PATH_GATEWAY.md | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/IPIP/0000-gateway-error-handling.md b/IPIP/0000-gateway-error-handling.md index 28d52ec97..146062b44 100644 --- a/IPIP/0000-gateway-error-handling.md +++ b/IPIP/0000-gateway-error-handling.md @@ -1,4 +1,4 @@ -# IPIP 0000: Error Handling in HTTP Gateways +# IPIP 0000: Streaming Error Handling in HTTP Gateways - Start Date: 2022-10-12 - Related Issues: @@ -7,7 +7,7 @@ ## Summary -Ensure error handling in web gateways is clear and consistent. +Ensure streaming error handling in web gateways is clear and consistent. ## Motivation @@ -24,7 +24,7 @@ handling, the server attempts to notify the user. ## Detailed design If the server encounters an error before streaming the contents to the client, -the server should fail with the respective `4xx` or `5xx` HTTP status code. +the server must fail with the respective `4xx` or `5xx` HTTP status code (no change). If the server encounters an error while streaming the contents, the server must force-close the HTTP connection to the user. This way, the user will receive a diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 42d431705..06d0f4ecd 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -83,6 +83,7 @@ where client prefers to perform all validation locally. - [Best practices for HTTP caching](#best-practices-for-http-caching) - [Denylists](#denylists) - [Generated HTML with directory index](#generated-html-with-directory-index) + - [Streaming errors](#streaming-errors) # HTTP API From 4e7354422ba9373e660c520f2410aefc8543d1df Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 12 Oct 2022 17:06:22 +0200 Subject: [PATCH 3/9] Update IPIP/0000-gateway-error-handling.md Co-authored-by: Jorropo --- IPIP/0000-gateway-error-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-error-handling.md b/IPIP/0000-gateway-error-handling.md index 146062b44..0d39a60b6 100644 --- a/IPIP/0000-gateway-error-handling.md +++ b/IPIP/0000-gateway-error-handling.md @@ -27,7 +27,7 @@ If the server encounters an error before streaming the contents to the client, the server must fail with the respective `4xx` or `5xx` HTTP status code (no change). If the server encounters an error while streaming the contents, the server must -force-close the HTTP connection to the user. This way, the user will receive a +force-close the HTTP stream to the user. This way, the user will receive a network error, making it clear that the downloaded file is not valid. ## Test fixtures From 9480caaf9ac54fff385064fc0e2b704f2c34fe69 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Mon, 7 Nov 2022 15:15:10 +0100 Subject: [PATCH 4/9] add stream close details for http 1.1 and http 2.0 --- IPIP/0000-gateway-error-handling.md | 2 +- http-gateways/PATH_GATEWAY.md | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/IPIP/0000-gateway-error-handling.md b/IPIP/0000-gateway-error-handling.md index 0d39a60b6..678d34fd2 100644 --- a/IPIP/0000-gateway-error-handling.md +++ b/IPIP/0000-gateway-error-handling.md @@ -41,7 +41,7 @@ an HTTP status code for the error. However, after the HTTP headers are set and the body started being streamed, there are no clear ways in the HTTP specification to show an error. Since the gateway is browser-first, it is important to show an error and avoid users receiving an incomplete file. -Therefore, the server can force-close the HTTP connection, leading to a network +Therefore, the server can force-close the HTTP stream, leading to a network error. This tells the user that an error happened. ### User benefit diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 06d0f4ecd..d24814dde 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -701,5 +701,9 @@ The usual optimizations involve: ## Streaming errors -Gateways MUST force-close HTTP connections if they detect an error while -streaming a file to avoid that users receive incomplete, yet valid, files. +To avoid users receiving an incomplete, yet valid, files, the gateway MUST +close the HTTP stream if an error occurs while streaming a file to the client. +This can be done via the following mechanisms: + +- Sending a `RST` (reset) frame for HTTP/1.1 +- Sending a `RST_STREAM` (reset stream) frame for HTTP/2 From a20a665a67d6802435a824431476966bd9893892 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 8 Nov 2022 11:33:42 +0100 Subject: [PATCH 5/9] Update http-gateways/PATH_GATEWAY.md Co-authored-by: Jorropo --- http-gateways/PATH_GATEWAY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index d24814dde..271cb81ca 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -705,5 +705,5 @@ To avoid users receiving an incomplete, yet valid, files, the gateway MUST close the HTTP stream if an error occurs while streaming a file to the client. This can be done via the following mechanisms: -- Sending a `RST` (reset) frame for HTTP/1.1 +- Sending a `FIN` (close) frame for HTTP/1.1 - Sending a `RST_STREAM` (reset stream) frame for HTTP/2 From e7f6b5436d5d0463238d36f33609dee1bf7a8bb9 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 8 Nov 2022 11:34:15 +0100 Subject: [PATCH 6/9] Update http-gateways/PATH_GATEWAY.md Co-authored-by: Jorropo --- http-gateways/PATH_GATEWAY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 271cb81ca..ef50b849d 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -707,3 +707,4 @@ This can be done via the following mechanisms: - Sending a `FIN` (close) frame for HTTP/1.1 - Sending a `RST_STREAM` (reset stream) frame for HTTP/2 +- Sending a `CANCEL_PUSH` frame for HTTP/3, see RFC9114 section A.2.5.3. From 0fcdd020b3579c45d025614cd74b922e365c01e2 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Thu, 10 Nov 2022 13:48:20 +0100 Subject: [PATCH 7/9] ipip: add headers info --- ...ling.md => 0332-gateway-error-handling.md} | 0 http-gateways/PATH_GATEWAY.md | 25 +++++++++++++++++++ 2 files changed, 25 insertions(+) rename IPIP/{0000-gateway-error-handling.md => 0332-gateway-error-handling.md} (100%) diff --git a/IPIP/0000-gateway-error-handling.md b/IPIP/0332-gateway-error-handling.md similarity index 100% rename from IPIP/0000-gateway-error-handling.md rename to IPIP/0332-gateway-error-handling.md diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 339dd1260..c77312629 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -41,6 +41,7 @@ where client prefers to perform all validation locally. - [`Accept` (request header)](#accept-request-header) - [`Range` (request header)](#range-request-header) - [`Service-Worker` (request header)](#service-worker-request-header) + - [`X-Error-Behavior` (request header)](#x-error-behavior-request-header) - [Request Query Parameters](#request-query-parameters) - [`filename` (request query parameter)](#filename-request-query-parameter) - [`download` (request query parameter)](#download-request-query-parameter) @@ -74,6 +75,8 @@ where client prefers to perform all validation locally. - [`X-Ipfs-Roots` (response header)](#x-ipfs-roots-response-header) - [`X-Content-Type-Options` (response header)](#x-content-type-options-response-header) - [`X-Trace-Id` (response header)](#x-trace-id-response-header) + - [`X-On-Error` (response header)](#x-on-error-response-header) + - [`X-Stream-Error` (response header)](#x-stream-error-response-header) - [Response Payload](#response-payload) - [Appendix: notes for implementers](#appendix-notes-for-implementers) - [Content resolution](#content-resolution) @@ -218,6 +221,17 @@ Gateway should refuse attempts to register a service worker for entire Requests to these paths with `Service-Worker: script` MUST be denied by returning HTTP 400 Bad Request error. +### `X-Error-Behavior` (request header) + +Clients can request a different behavior when errors occur during an HTTP +stream. The possible values, and behaviors are defined as follows: + +- `reset` (default): resets the HTTP stream. +- `trailer`: a trailer header [`X-Stream-Error`](#x-stream-error-response-header) will +be sent with the error message. + +See the [streaming errors](#streaming-errors) section for more information. + ## Request Query Parameters All query parameters are optional. @@ -583,6 +597,17 @@ unique identifier to help in debugging errors and performance issues. A good practice is to always return it with HTTP error [status codes](#response-status-codes) >=`400`. +### `X-On-Error` (response header) + +Returned on streaming requests to indicate the server behavior in case an error happens. +The header can have the value `reset`, or `trailer`. See the [streaming errors](#streaming-errors) +section for more information. + +### `X-Stream-Error` (response header) + +Returned if the client set [`X-Error-Behavior`](#x-error-behavior-request-header) to `trailer`. This trailing header will +contain the error message that occurred while streaming a file. + ## Response Payload Data sent with HTTP response depends on the type of requested IPFS resource: From 4dd1293fa8f03b7bf92bbe6c6b92b082c773b4e8 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Thu, 10 Nov 2022 13:49:54 +0100 Subject: [PATCH 8/9] ipip: add header info --- IPIP/0332-gateway-error-handling.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/IPIP/0332-gateway-error-handling.md b/IPIP/0332-gateway-error-handling.md index 678d34fd2..15c7609ce 100644 --- a/IPIP/0332-gateway-error-handling.md +++ b/IPIP/0332-gateway-error-handling.md @@ -30,6 +30,12 @@ If the server encounters an error while streaming the contents, the server must force-close the HTTP stream to the user. This way, the user will receive a network error, making it clear that the downloaded file is not valid. +In addition, the server must set an header `X-On-Error: reset` to indicate that +connection will be reset if an error occurs. The client can opt-out of this behavior +by sending a `X-Error-Behavior: trailer`. In that case, the server will not +reset the connection, but send the `X-Stream-Error` header with the error +message. + ## Test fixtures There are no relevant test fixures for this IPIP. From c0c6af74d31dfbcfcc991a906479a510bb3eb490 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Thu, 10 Nov 2022 13:51:22 +0100 Subject: [PATCH 9/9] ipip: update design --- IPIP/0332-gateway-error-handling.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/IPIP/0332-gateway-error-handling.md b/IPIP/0332-gateway-error-handling.md index 15c7609ce..91b51f60c 100644 --- a/IPIP/0332-gateway-error-handling.md +++ b/IPIP/0332-gateway-error-handling.md @@ -29,12 +29,12 @@ the server must fail with the respective `4xx` or `5xx` HTTP status code (no ch If the server encounters an error while streaming the contents, the server must force-close the HTTP stream to the user. This way, the user will receive a network error, making it clear that the downloaded file is not valid. - In addition, the server must set an header `X-On-Error: reset` to indicate that -connection will be reset if an error occurs. The client can opt-out of this behavior -by sending a `X-Error-Behavior: trailer`. In that case, the server will not -reset the connection, but send the `X-Stream-Error` header with the error -message. +connection will be reset if an error occurs. + +The client can opt-out of this behavior by sending a `X-Error-Behavior: trailer`. +In that case, the server will not force-close the HTTP stream. Instead, the server +will send the trailing header `X-Stream-Error` header with the error message. ## Test fixtures