From 242c4763f31525746991eff0adaa87ad3e942296 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 29 Sep 2022 15:17:11 +0200 Subject: [PATCH 1/7] ipip: add dag-cbor to reframe over http --- .markdownlint.json | 1 + IPIP/0000-http-reframe-cbor.md | 71 +++++++++++++++++++++++++++++++ reframe/REFRAME_HTTP_TRANSPORT.md | 64 ++++++++++++++++++++++------ 3 files changed, 123 insertions(+), 13 deletions(-) create mode 100644 IPIP/0000-http-reframe-cbor.md diff --git a/.markdownlint.json b/.markdownlint.json index 192f4edc3..d9283c1ba 100644 --- a/.markdownlint.json +++ b/.markdownlint.json @@ -1,4 +1,5 @@ { + "fenced-code-language": false, "single-h1": false, "no-bare-urls": false, "no-emphasis-as-header": false, diff --git a/IPIP/0000-http-reframe-cbor.md b/IPIP/0000-http-reframe-cbor.md new file mode 100644 index 000000000..0a02d8372 --- /dev/null +++ b/IPIP/0000-http-reframe-cbor.md @@ -0,0 +1,71 @@ +# IPIP 0000: Add DAG-CBOR support to Reframe over HTTP + + + +- Start Date: 2022-09-29 +- Related Issues: + - https://github.com/ipld/edelweiss/issues/16#issuecomment-1074161577 + - https://github.com/ipfs/kubo/issues/8823 + +## Summary + + +This IPIP adds DAG-CBOR support to Reframe over HTTP. + +## Motivation + +We've been using Reframe in Kubo for a while and it is clear that Reframe +messages are not designed to be created or read by humans. + +The plaintext DAG-JSON representation of messages does not really bring +anything to the table (because both CIDs and Multiaddrs are in a format that +needs manual encoding/decoding anyway), + +## Detailed design + +We already support DAG-JSON, with its own content type. +The change here is to add support for requests and responses sent as DAG-CBOR, +with own content type: `application/vnd.ipfs.rpc+dag-cbor`. + +For details, see changes made to `reframe/REFRAME_HTTP_TRANSPORT.md` + +## Test fixtures + +TODO: add CIDs of sample DAG-CBOR messages after https://github.com/ipfs/go-delegated-routing implements it, and has own tests. + +## Design rationale + +IPFS stack aims to support both DAG-CBOR and DAG-JSON. Users can store JSON as +CBOR and vice versa. Having consitent support for both in Reframe not only +aligns with user expectations, but also allows us to save some bytes +(bandwidth, response caching requirements) by using a binary CBOR as the +production format. + +### User benefit + +User will be able to choose between binary and human-readable representation, +just like they do in other parts of IPFS/IPLD stack. + +### Compatibility + +Explain the upgrade considerations for existing implementations. + +This IPIP add DAG-CBOR next to already existing DAG-JSON. Preexisting clients +that only speak DAG-JSON will continue working, no change is required. + +### Security + +N/A, we will use the same DAG-CBOR encoder/decoder as the rest of the stack. + +### Alternatives + +Alternative is to do nothing, and end up with: + +- inconsitent user experience +- wasted bandwidth and cache storage + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/reframe/REFRAME_HTTP_TRANSPORT.md b/reframe/REFRAME_HTTP_TRANSPORT.md index c5a85ed78..9f70d6d49 100644 --- a/reframe/REFRAME_HTTP_TRANSPORT.md +++ b/reframe/REFRAME_HTTP_TRANSPORT.md @@ -16,36 +16,74 @@ The Reframe over HTTP protocol is defining the transport and message serialization mechanisms for sending Reframe messages over HTTP `POST` and `GET`, and provides guidance for implementers around HTTP caching. -# Organization of this document - -- [HTTP Transport Design](#http-transport-design) - - [HTTP Caching Considerations](#http-caching-considerations) - - [POST vs GET](#post-vs-get) - - [Avoiding sending the same response messages twice](#avoiding-sending-the-same-response-messages-twice) - - [Client controls for time-based caching](#client-controls-for-time-based-caching) - - [Rate-limiting non-cachable POST requests](#rate-limiting-non-cachable-post-requests) +## Organization of this document + +- [HTTP Endpoint](#http-endpoint) + - [Content type](#content-type) + - [HTTP methods](#http-methods) + - [Other notes](#other-notes) +- [HTTP Caching Considerations](#http-caching-considerations) + - [POST vs GET](#post-vs-get) + - [Avoiding sending the same response messages twice](#avoiding-sending-the-same-response-messages-twice) + - [Client controls for time-based caching](#client-controls-for-time-based-caching) + - [Rate-limiting non-cachable POST requests](#rate-limiting-non-cachable-post-requests) - [Implementations](#implementations) -# HTTP Transport Design +## HTTP Endpoint -All messages sent in HTTP body MUST be encoded as DAG-JSON and use explicit content type `application/vnd.ipfs.rpc+dag-json; version=1` +``` +https://rpc-service.example.net/reframe +``` + +URL of a Reframe endpoint must end with `/reframe` path. + +### Content type + +Requests SHOULD be sent with explicit `Accept` and `Content-Type` HTTP headers specifying the body format. + +All messages sent in HTTP body MUST be encoded as either: + +- [DAG-CBOR](https://ipld.io/specs/codecs/dag-cbor/spec/), and use explicit content type `application/vnd.ipfs.rpc+dag-cbor; version=1` + - **This is a CBOR (binary) format for use in production.** + - CBOR request MUST include HTTP header: `Accept: application/vnd.ipfs.rpc+dag-cbor; version=1` + - CBOR request AND response MUST include header: `Content-Type: application/vnd.ipfs.rpc+dag-cbor; version=1` +- [DAG-JSON](https://ipld.io/specs/codecs/dag-json/spec/), and use explicit content type `application/vnd.ipfs.rpc+dag-json; version=1` + - **This is a human-readable plain text format for use in testing and debugging.** + - JSON request MUST include header: `Accept: application/vnd.ipfs.rpc+dag-json; version=1` + - JSON request AND response MUST include header: `Content-Type: application/vnd.ipfs.rpc+dag-json; version=1` + +Implementations SHOULD error when an explicit content type is missing, but MAY decide to implement some defaults instead. +The rules around implicit content type are as follows: + +- Requests without a matching `Content-Type` header MAY be interpreted as DAG-JSON. +- Requests without a matching `Accept` header MAY produce a DAG-JSON response. +- Responses without a matching `Content-Type` header MAY be interpreted as DAG-JSON. + +### HTTP methods Requests MUST be sent as either: +- `GET /reframe/{mbase64url-dag-cbor}` + - Cachable HTTP `GET` requests with message passed as DAG-CBOR in HTTP path segment, encoded as URL-safe [`base64url` multibase](https://docs.ipfs.io/concepts/glossary/#base64url) string + - DAG-CBOR in multibase `base64url` is used (even when request body is DAG-JSON) because JSON may include characters that are not safe to be used in URLs, and percent-encoding or base-encoding a big JSON query may take too much space. + - Suitable for sharing links, sending bigger messages, and when a query result MUST benefit from HTTP caching (see _HTTP Caching Considerations_ below). - `GET /reframe?q={percent-encoded-dag-json}` - DAG-JSON is supported via a `?q` query parameter, and the value MUST be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding) - Suitable for sharing links, sending smaller messages, testing and debugging. - `POST /reframe` - - Ephemeral HTTP `POST` request with message passed as DAG-JSON in HTTP request body + - Ephemeral HTTP `POST` request with DAG-JSON or DAG-CBOR message passed in HTTP request body - Suitable for bigger messages, and when HTTP caching should be skipped for the most fresh results Servers MUST support `GET` for methods marked as cachable and MUST support `POST` for all methods (both cachable and not-cachable). This allows servers to rate-limit `POST` when cachable `GET` could be used instead, and enables clients to use `POST` as a fallback in case there is a technical problem with bigger Reframe messages not fitting in a `GET` URL. See "Caching Considerations" section. +### Other notes + If a server supports HTTP/1.1, then it MAY send chunked-encoded messages. Clients supporting HTTP/1.1 MUST accept chunked-encoded responses. Requests and Responses MUST occur over a single HTTP call instead of the server being allowed to dial back the client with a response at a later time. The response status code MUST be 200 if the RPC transaction succeeds, even when there's an error at the application layer, and a non-200 status code if the RPC transaction fails. -If a server chooses to respond to a single request message with a group of messages in the response it should do so as a set of `\n` delimited DAG-JSON messages (i.e. `{Response1}\n{Response2}...`). +If a server chooses to respond to a single request message with a group of DAG-JSON messages in the response it should do so as a set of `\n` delimited DAG-JSON messages (i.e. `{Response1}\n{Response2}...`). +DAG-CBOR responses require no special handling, as they are already self-delimiting due to the nature of the CBOR encoding. Requests and responses MUST come with `version=1` as a _Required Parameter_ in the `Accept` and `Content-Type` HTTP headers. @@ -99,6 +137,6 @@ Retry-After: 3600 too many POST requests: consider switching to cachable GET or try again later (see Retry-After header) ``` -# Implementations +## Implementations https://github.com/ipfs/go-delegated-routing From 6e7ca3b94960729371acf89d2ab5b3522102eb8b Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 14 Oct 2022 00:34:13 +0200 Subject: [PATCH 2/7] Apply suggestions from review Co-authored-by: Gus Eggert --- IPIP/0000-http-reframe-cbor.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/IPIP/0000-http-reframe-cbor.md b/IPIP/0000-http-reframe-cbor.md index 0a02d8372..b6cd83eab 100644 --- a/IPIP/0000-http-reframe-cbor.md +++ b/IPIP/0000-http-reframe-cbor.md @@ -29,7 +29,7 @@ We already support DAG-JSON, with its own content type. The change here is to add support for requests and responses sent as DAG-CBOR, with own content type: `application/vnd.ipfs.rpc+dag-cbor`. -For details, see changes made to `reframe/REFRAME_HTTP_TRANSPORT.md` +For details, see changes made to `reframe/REFRAME_HTTP_TRANSPORT.md`. ## Test fixtures @@ -38,9 +38,9 @@ TODO: add CIDs of sample DAG-CBOR messages after https://github.com/ipfs/go-dele ## Design rationale IPFS stack aims to support both DAG-CBOR and DAG-JSON. Users can store JSON as -CBOR and vice versa. Having consitent support for both in Reframe not only +CBOR and vice versa. Having consistent support for both in Reframe not only aligns with user expectations, but also allows us to save some bytes -(bandwidth, response caching requirements) by using a binary CBOR as the +(bandwidth, response caching requirements) by using a binary CBOR as the production format. ### User benefit @@ -63,7 +63,7 @@ N/A, we will use the same DAG-CBOR encoder/decoder as the rest of the stack. Alternative is to do nothing, and end up with: -- inconsitent user experience +- inconsistent user experience - wasted bandwidth and cache storage ### Copyright From e1448c2eea5be30c62f86c0e81b94e59fe6cae11 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 19 Oct 2022 13:48:01 +0200 Subject: [PATCH 3/7] reframe(http): cachable method name on URL path --- reframe/REFRAME_HTTP_TRANSPORT.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/reframe/REFRAME_HTTP_TRANSPORT.md b/reframe/REFRAME_HTTP_TRANSPORT.md index 9f70d6d49..73373438f 100644 --- a/reframe/REFRAME_HTTP_TRANSPORT.md +++ b/reframe/REFRAME_HTTP_TRANSPORT.md @@ -63,16 +63,20 @@ The rules around implicit content type are as follows: Requests MUST be sent as either: -- `GET /reframe/{mbase64url-dag-cbor}` +- `GET /reframe/{CachableRequestMethodName}/{request-as-mbase64url-dag-cbor}` - Cachable HTTP `GET` requests with message passed as DAG-CBOR in HTTP path segment, encoded as URL-safe [`base64url` multibase](https://docs.ipfs.io/concepts/glossary/#base64url) string + - Cachable method name is placed on the URL path, allowing for different caching strategies per method, and custom routing/scaling per method, if needed. - DAG-CBOR in multibase `base64url` is used (even when request body is DAG-JSON) because JSON may include characters that are not safe to be used in URLs, and percent-encoding or base-encoding a big JSON query may take too much space. - Suitable for sharing links, sending bigger messages, and when a query result MUST benefit from HTTP caching (see _HTTP Caching Considerations_ below). -- `GET /reframe?q={percent-encoded-dag-json}` + - DAG-CBOR response is the implicit default, unless explicit `Accept` header is passed +- `GET /reframe?q={percent-encoded-request-as-dag-json}` - DAG-JSON is supported via a `?q` query parameter, and the value MUST be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding) - Suitable for sharing links, sending smaller messages, testing and debugging. + - DAG-JSON response is the implicit default, unless explicit `Accept` header is passed - `POST /reframe` - - Ephemeral HTTP `POST` request with DAG-JSON or DAG-CBOR message passed in HTTP request body + - Ephemeral HTTP `POST` request with DAG-JSON or DAG-CBOR message passed in HTTP request body and a mandatory `Content-Type` header informing endpoint how to parse the body - Suitable for bigger messages, and when HTTP caching should be skipped for the most fresh results + - Response type is the same as request, unless explicit `Accept` header is passed Servers MUST support `GET` for methods marked as cachable and MUST support `POST` for all methods (both cachable and not-cachable). This allows servers to rate-limit `POST` when cachable `GET` could be used instead, and enables clients to use `POST` as a fallback in case there is a technical problem with bigger Reframe messages not fitting in a `GET` URL. See "Caching Considerations" section. From 58d09375a6f9d800c4bf7244b0bb81678c2c5707 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 19 Oct 2022 15:01:11 +0200 Subject: [PATCH 4/7] reframe(http): always have method in the URL makes sense for POST too: alows for scaling, routing different methods to different backend services --- reframe/REFRAME_HTTP_TRANSPORT.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reframe/REFRAME_HTTP_TRANSPORT.md b/reframe/REFRAME_HTTP_TRANSPORT.md index 73373438f..216035468 100644 --- a/reframe/REFRAME_HTTP_TRANSPORT.md +++ b/reframe/REFRAME_HTTP_TRANSPORT.md @@ -63,17 +63,17 @@ The rules around implicit content type are as follows: Requests MUST be sent as either: -- `GET /reframe/{CachableRequestMethodName}/{request-as-mbase64url-dag-cbor}` +- `GET /reframe/{method}/{request-as-mbase64url-dag-cbor}` - Cachable HTTP `GET` requests with message passed as DAG-CBOR in HTTP path segment, encoded as URL-safe [`base64url` multibase](https://docs.ipfs.io/concepts/glossary/#base64url) string - - Cachable method name is placed on the URL path, allowing for different caching strategies per method, and custom routing/scaling per method, if needed. + - Cachable `method` name is placed on the URL path, allowing for different caching strategies per `method`, and custom routing/scaling per `method`, if needed. - DAG-CBOR in multibase `base64url` is used (even when request body is DAG-JSON) because JSON may include characters that are not safe to be used in URLs, and percent-encoding or base-encoding a big JSON query may take too much space. - Suitable for sharing links, sending bigger messages, and when a query result MUST benefit from HTTP caching (see _HTTP Caching Considerations_ below). - DAG-CBOR response is the implicit default, unless explicit `Accept` header is passed -- `GET /reframe?q={percent-encoded-request-as-dag-json}` +- `GET /reframe/{method}?q={percent-encoded-request-as-dag-json}` - DAG-JSON is supported via a `?q` query parameter, and the value MUST be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding) - Suitable for sharing links, sending smaller messages, testing and debugging. - DAG-JSON response is the implicit default, unless explicit `Accept` header is passed -- `POST /reframe` +- `POST /reframe/{method}` - Ephemeral HTTP `POST` request with DAG-JSON or DAG-CBOR message passed in HTTP request body and a mandatory `Content-Type` header informing endpoint how to parse the body - Suitable for bigger messages, and when HTTP caching should be skipped for the most fresh results - Response type is the same as request, unless explicit `Accept` header is passed From 0c57718044828cbb6ba86aa1cb8ce948c063dfff Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 19 Oct 2022 15:22:01 +0200 Subject: [PATCH 5/7] reframe(http): v2 of wire protocol, removed etag Requiring Etag meant implementations naively buffered entire streaming response. Relaxing this. --- reframe/REFRAME_HTTP_TRANSPORT.md | 54 +++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 17 deletions(-) diff --git a/reframe/REFRAME_HTTP_TRANSPORT.md b/reframe/REFRAME_HTTP_TRANSPORT.md index 216035468..fdcf685bd 100644 --- a/reframe/REFRAME_HTTP_TRANSPORT.md +++ b/reframe/REFRAME_HTTP_TRANSPORT.md @@ -24,8 +24,9 @@ serialization mechanisms for sending Reframe messages over HTTP `POST` and - [Other notes](#other-notes) - [HTTP Caching Considerations](#http-caching-considerations) - [POST vs GET](#post-vs-get) - - [Avoiding sending the same response messages twice](#avoiding-sending-the-same-response-messages-twice) - - [Client controls for time-based caching](#client-controls-for-time-based-caching) + - [Etag](#etag) + - [Last-Modified](#last-modified) + - [Cache-Control](#cache-control) - [Rate-limiting non-cachable POST requests](#rate-limiting-non-cachable-post-requests) - [Implementations](#implementations) @@ -43,14 +44,14 @@ Requests SHOULD be sent with explicit `Accept` and `Content-Type` HTTP headers s All messages sent in HTTP body MUST be encoded as either: -- [DAG-CBOR](https://ipld.io/specs/codecs/dag-cbor/spec/), and use explicit content type `application/vnd.ipfs.rpc+dag-cbor; version=1` +- [DAG-CBOR](https://ipld.io/specs/codecs/dag-cbor/spec/), and use explicit content type `application/vnd.ipfs.rpc+dag-cbor; version=2` - **This is a CBOR (binary) format for use in production.** - - CBOR request MUST include HTTP header: `Accept: application/vnd.ipfs.rpc+dag-cbor; version=1` - - CBOR request AND response MUST include header: `Content-Type: application/vnd.ipfs.rpc+dag-cbor; version=1` -- [DAG-JSON](https://ipld.io/specs/codecs/dag-json/spec/), and use explicit content type `application/vnd.ipfs.rpc+dag-json; version=1` + - CBOR request MUST include HTTP header: `Accept: application/vnd.ipfs.rpc+dag-cbor; version=2` + - CBOR request AND response MUST include header: `Content-Type: application/vnd.ipfs.rpc+dag-cbor; version=2` +- [DAG-JSON](https://ipld.io/specs/codecs/dag-json/spec/), and use explicit content type `application/vnd.ipfs.rpc+dag-json; version=2` - **This is a human-readable plain text format for use in testing and debugging.** - - JSON request MUST include header: `Accept: application/vnd.ipfs.rpc+dag-json; version=1` - - JSON request AND response MUST include header: `Content-Type: application/vnd.ipfs.rpc+dag-json; version=1` + - JSON request MUST include header: `Accept: application/vnd.ipfs.rpc+dag-json; version=2` + - JSON request AND response MUST include header: `Content-Type: application/vnd.ipfs.rpc+dag-json; version=2` Implementations SHOULD error when an explicit content type is missing, but MAY decide to implement some defaults instead. The rules around implicit content type are as follows: @@ -76,7 +77,7 @@ Requests MUST be sent as either: - `POST /reframe/{method}` - Ephemeral HTTP `POST` request with DAG-JSON or DAG-CBOR message passed in HTTP request body and a mandatory `Content-Type` header informing endpoint how to parse the body - Suitable for bigger messages, and when HTTP caching should be skipped for the most fresh results - - Response type is the same as request, unless explicit `Accept` header is passed + - Response type is the same as `Content-Type` of the request, unless explicit `Accept` header is passed Servers MUST support `GET` for methods marked as cachable and MUST support `POST` for all methods (both cachable and not-cachable). This allows servers to rate-limit `POST` when cachable `GET` could be used instead, and enables clients to use `POST` as a fallback in case there is a technical problem with bigger Reframe messages not fitting in a `GET` URL. See "Caching Considerations" section. @@ -89,7 +90,7 @@ Requests and Responses MUST occur over a single HTTP call instead of the server If a server chooses to respond to a single request message with a group of DAG-JSON messages in the response it should do so as a set of `\n` delimited DAG-JSON messages (i.e. `{Response1}\n{Response2}...`). DAG-CBOR responses require no special handling, as they are already self-delimiting due to the nature of the CBOR encoding. -Requests and responses MUST come with `version=1` as a _Required Parameter_ in the `Accept` and `Content-Type` HTTP headers. +Requests and responses MUST come with `version=2` as a _Required Parameter_ in the `Accept` and `Content-Type` HTTP headers. Note: This version header is what allows the transport to more easily evolve over time (e.g. if it was desired to change the transport to support other encodings than DAG-JSON, utilize headers differently, move the request data from the body, etc.). Not including the version number is may lead to incompatibility with future versions of the transport. @@ -107,23 +108,42 @@ Use of `GET` endpoint is not mandatory, but suggested if a Reframe deployment expects to handle the same message query multiple times, and want to leverage existing HTTP tooling to maximize HTTP cache hits. -### Avoiding sending the same response messages twice +### Etag -Implementations MUST always return strong +For small responses. + +Implementations MAY return [`Etag`](https://httpwg.org/specs/rfc7232.html#header.etag) HTTP header based -on digest of DAG-JSON response messages. This allows clients to send -inexpensive conditional requests with +on a digest of response messages ONLY when `Etag` generation does not require +buffering bigger response in memory before sending it to the client. + +In other words, do not use `Etag` if it will block a big, streaming response. +Streaming responses should use `Last-Modified` instead. + +`Etag` allows clients to send inexpensive conditional requests with [`If-None-Match`](https://httpwg.org/specs/rfc7232.html#header.if-none-match) header, which will skip when the response message did not change. -### Client controls for time-based caching +### Last-Modified + +For streaming responses. -Implementations can also return (optional) +Implementations SHOULD return [`Last-Modified`](https://httpwg.org/specs/rfc7232.html#header.last-modified) -HTTP header, allowing clients to send conditional requests with +HTTP header with bigger, streaming responses. + +This allows clients to send conditional requests with [`If-Modified-Since`](https://httpwg.org/specs/rfc7232.html#header.if-modified-since) header to specify their acceptance for stale (cached) responses. +### Cache-Control + +Implementations MAY return custom `Cache-Control` per Reframe method, +when a specific cache window makes sense in the context of specific method. + +It is also acceptable to leave it out and let reverse HTTP provies / CDNs to +set it. Value will depend on use case, and expected load. + ### Rate-limiting non-cachable POST requests HTTP endpoint can return status code From 72739d64b6077308fb743962dec9cd46c16ebf78 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 20 Oct 2022 14:38:08 +0200 Subject: [PATCH 6/7] reframe(http): rename q to json making it self-describing, reducing developer confusion we experienced last week when folks started implementing version=1 --- reframe/REFRAME_HTTP_TRANSPORT.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reframe/REFRAME_HTTP_TRANSPORT.md b/reframe/REFRAME_HTTP_TRANSPORT.md index fdcf685bd..dcad275ac 100644 --- a/reframe/REFRAME_HTTP_TRANSPORT.md +++ b/reframe/REFRAME_HTTP_TRANSPORT.md @@ -70,8 +70,8 @@ Requests MUST be sent as either: - DAG-CBOR in multibase `base64url` is used (even when request body is DAG-JSON) because JSON may include characters that are not safe to be used in URLs, and percent-encoding or base-encoding a big JSON query may take too much space. - Suitable for sharing links, sending bigger messages, and when a query result MUST benefit from HTTP caching (see _HTTP Caching Considerations_ below). - DAG-CBOR response is the implicit default, unless explicit `Accept` header is passed -- `GET /reframe/{method}?q={percent-encoded-request-as-dag-json}` - - DAG-JSON is supported via a `?q` query parameter, and the value MUST be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding) +- `GET /reframe/{method}?json={percent-encoded-request-as-dag-json}` + - DAG-JSON is supported via a `?json` query parameter, and the value MUST be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding) - Suitable for sharing links, sending smaller messages, testing and debugging. - DAG-JSON response is the implicit default, unless explicit `Accept` header is passed - `POST /reframe/{method}` From 0256c2b79ee6ca9c4ea58056fbd751fc7d35bef4 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 20 Oct 2022 14:48:52 +0200 Subject: [PATCH 7/7] ipip(reframe-http): update to reflect version=2 Added things beyond DAG-CBOR, IPIP had to be updated --- IPIP/0000-http-reframe-cbor.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/IPIP/0000-http-reframe-cbor.md b/IPIP/0000-http-reframe-cbor.md index b6cd83eab..552034a8f 100644 --- a/IPIP/0000-http-reframe-cbor.md +++ b/IPIP/0000-http-reframe-cbor.md @@ -21,7 +21,12 @@ messages are not designed to be created or read by humans. The plaintext DAG-JSON representation of messages does not really bring anything to the table (because both CIDs and Multiaddrs are in a format that -needs manual encoding/decoding anyway), +needs manual encoding/decoding anyway), and the utility is limited to debugging +and use in examples. + +We've also identified some HTTP caching and scaling issues due to all methods +sharing the same URL path and the way `Etag` header is generated, and how +it made streaming responses impossible. ## Detailed design @@ -29,6 +34,10 @@ We already support DAG-JSON, with its own content type. The change here is to add support for requests and responses sent as DAG-CBOR, with own content type: `application/vnd.ipfs.rpc+dag-cbor`. +We change the URL to include method name on the path. This allows deployments +to scale better: set different HTTP cache control policies, or route different +methods to different backend services. + For details, see changes made to `reframe/REFRAME_HTTP_TRANSPORT.md`. ## Test fixtures @@ -48,12 +57,16 @@ production format. User will be able to choose between binary and human-readable representation, just like they do in other parts of IPFS/IPLD stack. +DAG-JSON is the implicit default, improving ergonomics when debugging with `curl` +in CLI or fetching response via regular web browser + ### Compatibility -Explain the upgrade considerations for existing implementations. +IPFS / IPLD stack already includes both DAG-CBOR and DAG-JSON libraries. +The `version` parameter of the HTTP wire protocol is bumped to `2`. -This IPIP add DAG-CBOR next to already existing DAG-JSON. Preexisting clients -that only speak DAG-JSON will continue working, no change is required. +Reframe endpoints that care about backward-compatibility with Kubo 0.16 +can keep support for requests sent with `version=1`. ### Security @@ -65,6 +78,7 @@ Alternative is to do nothing, and end up with: - inconsistent user experience - wasted bandwidth and cache storage +- difficult deployment and scaling (all methods under same endpoint) ### Copyright