From 1af20b068150fe4bd612a46c205c03e87cf671b2 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 29 Aug 2025 00:17:30 +0000 Subject: [PATCH 1/7] A104: xDS Extension Config Discovery Service (ECDS) --- A104-xds-ecds.md | 169 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 A104-xds-ecds.md diff --git a/A104-xds-ecds.md b/A104-xds-ecds.md new file mode 100644 index 000000000..6f3f17c20 --- /dev/null +++ b/A104-xds-ecds.md @@ -0,0 +1,169 @@ +A104: xDS Extension Config Discovery Service (ECDS) +---- +* Author(s): @markdroth +* Approver: @ejona86, @dfawley +* Status: {Draft, In Review, Ready for Implementation, Implemented} +* Implemented in: +* Last updated: 2025-08-28 +* Discussion at: (filled after thread exists) + +## Abstract + +gRPC willl support the xDS Extension Config Discovery Service (ECDS), +which allows filter configs to be split into separate xDS resources +rather than being inlined in the LDS resource. + +## Background + +The xDS HTTP filter support described in [A39] supports filter configs +inlined in the LDS resource. ECDS is an API whereby the LDS resource +will instead tell the data plane to fetch the config for a given filter +from a separate xDS resource. This is useful in cases where it is +desirable to get the resource from a different control plane via xDS +federation, as described in [A47]. + +### Related Proposals: +* [A39: xDS HTTP Filter Support][A39] +* [A47: xDS Federation][A47] +* [A74: xDS Config Tears][A74] + +[A39]: A39-xds-http-filters.md +[A47]: A47-xds-federation.md +[A74]: A74-xds-config-tears.md + +## Proposal + +When validating the [`HttpConnectionManager.http_filters` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L436), +if the [`config_discovery` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1243) +is set instead of the [`typed_config` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1233), +that indicates that the filter config should be fetched via ECDS. + +The ECDS `DiscoveryRequest` will use +[`envoy.config.core.v3.TypedExtensionConfig`](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/extension.proto#L20) +as the resource type. The resource name will come from the [`http_filter.name` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1223). + +### LDS Resource Validation Changes + +Inside of the `HttpFilter.config_discovery` field, gRPC will look at +the following fields: +- [config_source](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L267): + This field is ignored. TODO: should we require this to be a "self" + ConfigSource? Not sure that makes sense if we are going to share + config with Envoy, although that might not be an issue for LDS. +- [default_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L272): + This field is ignored. +- [apply_default_config_without_warming](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L277): + This field is ignored. +- [type_urls](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L281C19-L281C28): + When ECDS returns the `TypedExtensionConfig` resource, it will contain + an extension of some arbitrary type. If this field is set, it + specifies a allowlist of types that are acceptable. If this list is + non-empty and the returned extension type is not on the list, then + the extension will not be accepted. However, note that we cannot NACK + the ECDS resource in this case, because it's entirely possible that + the same ECDS resource will be used in a different filter instance + where its type *is* in the allowlist. Instead, we will look at the + `HttpFilter.is_optional` field: if set to true, then we will skip the + filter; otherwise, we will instead use a filter that will fail all + data plane RPCs. + + TODO: should we just not support this field at all, to make this + failure mode go away? + +The parsed LDS resource will need to represent ECDS configuration for +individual filters. For example, in C-core, the representation of a +filter today looks like this: + +```c++ +struct HttpFilter { + std::string name; + XdsHttpFilterImpl::FilterConfig config; +}; +``` + +We will change this to instead look like this: + +```c++ +struct HttpFilter { + std::string name; + std::variant config; +}; +``` + +### ECDS Resource Validation + +We will implement a new xDS resource type for `TypedExtensionConfig` +resources. The [`name` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/extension.proto#L23) +will indicate the resource name when not wrapped in a [`Resource` +wrapper](https://github.com/envoyproxy/envoy/blob/ee9d3b13f0e73930236b9371554e3f3ac3fcf7c4/api/envoy/service/discovery/v3/discovery.proto#L386). + +Although in principle, ECDS allows fetching extensions of *any* type, +gRPC will initially support it only for HTTP filters. As a result, we +can hard-code the use the xDS HTTP filter registry when validating ECDS +resources. We will validate the contents of the [`typed_config` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/extension.proto#L31) +via the xDS HTTP filter registry, just as we do today for the +[`HttpFilter.typed_config` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1233). + +The parsed representation of an ECDS resource will be the same as the +representation of an HTTP filter in the parsed LDS resource. For +example, in C-core, the representation will be the `HttpFilter` struct +shown above. + +### gRPC Client-Side Changes + +The XdsDependencyManager (see [A74]) will be changed to handle ECDS. +Specifically, when it receives the LDS resource, it will look through the +list of HTTP filters to see if any of them use ECDS. For those that do, +it will start a watch for the specified ECDS resource. It will not +return any result to the xds resolver until all ECDS resources have +been obtained. + +A new field will be added to the `XdsConfig` struct returned by the +XdsDependencyManager to contain all fetched ECDS resources. This field +will be a map from resource name to parsed ECDS resource. The xds +resolver will use that map to get the filter config for any filter in +the LDS resource that uses ECDS. + +### gRPC Server-Side Changes + +TODO: Figure out what server-side changes are needed + +### Composite Filter Changes + +TODO: Do we need to support ECDS in the composite filter? + +### Temporary environment variable protection + +Support for ECDS will be guarded by the `GRPC_EXPERIMENTAL_XDS_ECDS` +environment variable. This guard will be removed once the feature passes +interop tests. + +## Rationale + +### Handling Non-Allowlisted Extension Types + +Envoy currently lacks a caching XdsClient layer. As a result, its +documented behavior when the ECDS resource contains a non-allowlisted type +is to reject that update. That approach won't work for gRPC, because we +*do* have a caching XdsClient layer, and the same resource may be watched +by some other filter instance, where the type *is* in the allowlist. +Therefore, gRPC will either skip the filter or fail data plane RPCs, +depending on the value of `is_optional`. + +Note that if we do fail data plane RPCs, this is an instance of a +relatively rare class of config problems where the individual resource +is valid in isolation and therefore cannot be NACKed, but a problem does +appear when we try to combine the resources into a cohesive config. +We do not currently have a good solution for detecting this class of +problems. + +## Implementation + +Will be implemented in C-core, Java, Go, and Node. From 94388220ad2a38e7ccadb5ee63dd5bb5670a767c Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 29 Aug 2025 23:39:49 +0000 Subject: [PATCH 2/7] add a lot of details --- A104-xds-ecds.md | 149 ++++++++++++++++++++++++++++------------------- 1 file changed, 89 insertions(+), 60 deletions(-) diff --git a/A104-xds-ecds.md b/A104-xds-ecds.md index 6f3f17c20..849aa8aa2 100644 --- a/A104-xds-ecds.md +++ b/A104-xds-ecds.md @@ -4,7 +4,7 @@ A104: xDS Extension Config Discovery Service (ECDS) * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-08-28 +* Last updated: 2025-08-29 * Discussion at: (filled after thread exists) ## Abstract @@ -22,58 +22,71 @@ from a separate xDS resource. This is useful in cases where it is desirable to get the resource from a different control plane via xDS federation, as described in [A47]. -### Related Proposals: +The ECDS `DiscoveryRequest` uses +[`envoy.config.core.v3.TypedExtensionConfig`](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/extension.proto#L20) +as the resource type, and the extension configured within the +`TypedExtensionConfig` dictates what type of filter to use. This allows +the control plane to dynamically choose the type of the filter. + +### Related Proposals: * [A39: xDS HTTP Filter Support][A39] * [A47: xDS Federation][A47] * [A74: xDS Config Tears][A74] +* [A103: xDS Composite Filter][A103] (pending) [A39]: A39-xds-http-filters.md [A47]: A47-xds-federation.md [A74]: A74-xds-config-tears.md +[A103]: https://github.com/grpc/proposal/pull/511 + +TODO: add Updated-by: header to A103 ## Proposal -When validating the [`HttpConnectionManager.http_filters` +We will support configuring ECDS in two places: +- In LDS, in the HttpConnectionManager config, as part of the list of + xDS HTTP filters. +- In the xDS composite filter (see [A103]), as part of the configuration + of the filter to choose based on the match criteria. + +### Validation of `ExtensionConfigSource` Proto + +ECDS is configured via an [`ExtensionConfigSource` +proto](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L266). +The presence of this message will indicate that ECDS should be used, but +gRPC will not actually use any of the fields inside of this proto, each +for different reasons: +- [config_source](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L267): + We will use xDS federation (see [A47]) to determine what server to + fetch the resource from based on the resource name, so there is no + need to look at this field. (In the past, we have required that + various ConfigSource fields specify the "self" type, but this + convention predates xDS federation and does not seem to serve any + value anymore.) +- [default_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L272) + and + [apply_default_config_without_warming](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L277): + We don't currently have any use-case for setting a default config, so + we will not support these fields. We can consider adding support for + these fields in the future if a use-case comes up. +- [type_urls](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L281C19-L281C28): + We don't have a use-case for restricting the set of filter types that + may be returned, and supporting this would introduce the possibility + of a failure mode that cannot be reflected in a NACK, which could + compromise reliability (see [Rationale](#rationale) below for details). + +### LDS Resource Validation Changes + +When validating the entries in the [`HttpConnectionManager.http_filters` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L436), if the [`config_discovery` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1243) is set instead of the [`typed_config` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1233), that indicates that the filter config should be fetched via ECDS. - -The ECDS `DiscoveryRequest` will use -[`envoy.config.core.v3.TypedExtensionConfig`](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/extension.proto#L20) -as the resource type. The resource name will come from the [`http_filter.name` +In this case, the ECDS resource name comes from the [`http_filter.name` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1223). -### LDS Resource Validation Changes - -Inside of the `HttpFilter.config_discovery` field, gRPC will look at -the following fields: -- [config_source](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L267): - This field is ignored. TODO: should we require this to be a "self" - ConfigSource? Not sure that makes sense if we are going to share - config with Envoy, although that might not be an issue for LDS. -- [default_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L272): - This field is ignored. -- [apply_default_config_without_warming](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L277): - This field is ignored. -- [type_urls](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L281C19-L281C28): - When ECDS returns the `TypedExtensionConfig` resource, it will contain - an extension of some arbitrary type. If this field is set, it - specifies a allowlist of types that are acceptable. If this list is - non-empty and the returned extension type is not on the list, then - the extension will not be accepted. However, note that we cannot NACK - the ECDS resource in this case, because it's entirely possible that - the same ECDS resource will be used in a different filter instance - where its type *is* in the allowlist. Instead, we will look at the - `HttpFilter.is_optional` field: if set to true, then we will skip the - filter; otherwise, we will instead use a filter that will fail all - data plane RPCs. - - TODO: should we just not support this field at all, to make this - failure mode go away? - The parsed LDS resource will need to represent ECDS configuration for individual filters. For example, in C-core, the representation of a filter today looks like this: @@ -89,11 +102,28 @@ We will change this to instead look like this: ```c++ struct HttpFilter { + struct EcdsConfig {}; + std::string name; std::variant config; }; ``` +### xDS Composite Filter Validation Changes + +As per [A103], the xDS composite filter is configured using a +[`ExecuteFilterAction` +proto](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L49). +In that proto, if the [`dynamic_config` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L59) +is set instead of the [`typed_config` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L54), +that indicates that the filter config should be fetched via ECDS. +In this case, the ECDS resource name comes from the [`DynamicConfig.name` +field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L39). + +TODO: document parsed representation changes + ### ECDS Resource Validation We will implement a new xDS resource type for `TypedExtensionConfig` @@ -113,17 +143,24 @@ field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb The parsed representation of an ECDS resource will be the same as the representation of an HTTP filter in the parsed LDS resource. For -example, in C-core, the representation will be the `HttpFilter` struct -shown above. +example, in C-core, the representation will be the +`XdsHttpFilterImpl::FilterConfig` struct mentioned above. ### gRPC Client-Side Changes The XdsDependencyManager (see [A74]) will be changed to handle ECDS. Specifically, when it receives the LDS resource, it will look through the list of HTTP filters to see if any of them use ECDS. For those that do, -it will start a watch for the specified ECDS resource. It will not -return any result to the xds resolver until all ECDS resources have -been obtained. +it will start a watch for the specified ECDS resource. + +TODO: don't want XDM to have specific knowledge of the composite filter, +so how will it know to start a watch for the required ECDS resource? + +The XdsDependencyManager will not return any result to the xds resolver +until all ECDS resources have been obtained. If an ECDS resource fails +validation, the XdsDependencyManager will return a transient error to +the xds resolver, which will cause the channel to stick with its +previous good config, if any. A new field will be added to the `XdsConfig` struct returned by the XdsDependencyManager to contain all fetched ECDS resources. This field @@ -134,10 +171,7 @@ the LDS resource that uses ECDS. ### gRPC Server-Side Changes TODO: Figure out what server-side changes are needed - -### Composite Filter Changes - -TODO: Do we need to support ECDS in the composite filter? +(should have essentially the same behavior as XDM on client side) ### Temporary environment variable protection @@ -147,22 +181,17 @@ interop tests. ## Rationale -### Handling Non-Allowlisted Extension Types - -Envoy currently lacks a caching XdsClient layer. As a result, its -documented behavior when the ECDS resource contains a non-allowlisted type -is to reject that update. That approach won't work for gRPC, because we -*do* have a caching XdsClient layer, and the same resource may be watched -by some other filter instance, where the type *is* in the allowlist. -Therefore, gRPC will either skip the filter or fail data plane RPCs, -depending on the value of `is_optional`. - -Note that if we do fail data plane RPCs, this is an instance of a -relatively rare class of config problems where the individual resource -is valid in isolation and therefore cannot be NACKed, but a problem does -appear when we try to combine the resources into a cohesive config. -We do not currently have a good solution for detecting this class of -problems. +We will not support the `ExtensionConfigSource.type_urls` field, because +that would introduce the possibility of a failure without a NACK to inform +the control plane of the problem. The XdsClient needs to decide whether +to NACK when it first recieves the ECDS resource and checks whether +it's valid, and only if it is will it update its cache and report it +to the watchers. However, only the watchers know the context in which +the ECDS resource is being requested, which is where the allowlist of +filter types would be configured. It's entirely possible that the same +ECDS resource could be used in two different filter instances, where +its type is allowed in one but disallowed in another, and we wouldn't +discover that until we send the update to the watchers. ## Implementation From 0ee69dbcba5fa9f895df075ae6bd40e1297175b1 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Sat, 30 Aug 2025 00:18:38 +0000 Subject: [PATCH 3/7] more details on composite filter --- A104-xds-ecds.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/A104-xds-ecds.md b/A104-xds-ecds.md index 849aa8aa2..ed0a28ad5 100644 --- a/A104-xds-ecds.md +++ b/A104-xds-ecds.md @@ -122,7 +122,27 @@ that indicates that the filter config should be fetched via ECDS. In this case, the ECDS resource name comes from the [`DynamicConfig.name` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L39). -TODO: document parsed representation changes +The validation rules for the fields in the `ExecuteFilterAction` proto +will now be (first match wins): +- If `dynamic_config` is set, that gets used. +- If `filter_chain` is set, that gets used. +- If `typed_config` is set, that gets used. +- Otherwise, the config is considered invalid. + +Note that `dynamic_config` is being set as higher precedence to the other +fields, because gRPC will already support the other two fields, so this +provides a path for control planes to migrate to the new field without +breaking existing gRPC clients. However, control plane operators should +be aware that Envoy currently requires that exactly one of `typed_config` +or `dynamic_config` are present, so a config setting both of those is +likely to be rejected by Envoy. + +The parsed representation of the composite filter will change in a similar +way to the parsed LDS representation. As per [A103], the actions in the +matcher tree can have two possible values, a parsed filter config or an +indication to skip the filter. With the addition of ECDS support, a +third possible value will be added, which is the ECDS resource name to +fetch. ### ECDS Resource Validation From aa868135f1916a59e06ca0dafbda71ae120d8dfc Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 4 Sep 2025 23:44:00 +0000 Subject: [PATCH 4/7] fleshed out a lot of plumbing details --- A104-xds-ecds.md | 122 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 100 insertions(+), 22 deletions(-) diff --git a/A104-xds-ecds.md b/A104-xds-ecds.md index ed0a28ad5..690d617c2 100644 --- a/A104-xds-ecds.md +++ b/A104-xds-ecds.md @@ -4,7 +4,7 @@ A104: xDS Extension Config Discovery Service (ECDS) * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-08-29 +* Last updated: 2025-09-04 * Discussion at: (filled after thread exists) ## Abstract @@ -49,13 +49,19 @@ We will support configuring ECDS in two places: - In the xDS composite filter (see [A103]), as part of the configuration of the filter to choose based on the match criteria. +In order to support that second case in a general-purpose way, we will +change the interface for defining xDS HTTP filters such that when a +filter parses its config, it may return an indication of any ECDS +resources that it may require, even transitively. + ### Validation of `ExtensionConfigSource` Proto -ECDS is configured via an [`ExtensionConfigSource` +In both LDS and in the composite filter, ECDS is configured via an +[`ExtensionConfigSource` proto](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L266). -The presence of this message will indicate that ECDS should be used, but -gRPC will not actually use any of the fields inside of this proto, each -for different reasons: +The presence of this message will indicate that ECDS should be used, +but gRPC will not actually use any of the fields inside of this proto, +each for different reasons: - [config_source](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/config/core/v3/config_source.proto#L267): We will use xDS federation (see [A47]) to determine what server to fetch the resource from based on the resource name, so there is no @@ -75,8 +81,36 @@ for different reasons: of a failure mode that cannot be reflected in a NACK, which could compromise reliability (see [Rationale](#rationale) below for details). +### xDS HTTP Filter API Changes + +As per [A39], when gRPC sees an HTTP filter config in either LDS or RDS, +it will look up the filter implementation in the xDS HTTP filter +registry by the protobuf type of the filter config. It will then ask that +filter implementation to parse the filter config. + +We will change the xDS HTTP filter implementation API such that when it +parses the config, it has the ability to return a set of ECDS resource +names that it depends on. Note that this will work transitively for +cases like the composite filter: if the composite filter uses the xDS +HTTP filter registry to parse a nested filter config, then any ECDS +resource needed anywhere in that tree will be returned from the +composite filter's parsing function. + +Note that in order to avoid potential circular dependencies or stack +overflows, we will impose a maximum recursion depth of 8 when parsing +HTTP filter configs. + ### LDS Resource Validation Changes +We will add a new field to the parsed representation of the +`HttpConnectionManager` config that will indicate the set of ECDS resource +names that are needed as dependencies. For example, in C-core, it will +look like this: + +``` +std::set ecds_resources_needed; +``` + When validating the entries in the [`HttpConnectionManager.http_filters` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L436), if the [`config_discovery` @@ -102,13 +136,41 @@ We will change this to instead look like this: ```c++ struct HttpFilter { - struct EcdsConfig {}; + struct UseEcds {}; std::string name; - std::variant config; + std::variant config; }; ``` +If the `config` field contains a `UseEcds` struct, that means that the +config for the filter should be obtained from an ECDS resource with name +from the `name` field. In this case, the name of the ECDS resource will +be added to the `ecds_resources_needed` field in the parsed +`HttpConnectionManager` config. + +If the `config` field contains an inline filter config rather than a +`UseEcds` struct, then we will use the xDS HTTP filter registry to parse +the specified filter config, just like we do today. In this case, any +ECDS resources needed that the filter parsing returned will be added to +the `ecds_resources_needed` field in the parsed `HttpConnectionManager` +config. + +### RDS Resource Validation Changes + +Just like in LDS, we will add a new `ecds_resources_needed` field to +the parsed representation of the `RouteConfiguration` resource that will +indicate the set of ECDS resource names that are needed as dependencies. +This field will have the same type as the corresponding field described +in the `HttpConnectionManager` config above. + +As described in [xDS HTTP Filter API Changes](#xds-http-filter-api-changes) +above, as we use the xDS HTTP filter registry to parse the configs in +the various `typed_per_filter_config` fields, the xDS HTTP filter +implementation will return a list of ECDS resource names that it depends +on. This list will be added to the `ecds_resources_needed` field in the +parsed `RouteConfiguration` resource. + ### xDS Composite Filter Validation Changes As per [A103], the xDS composite filter is configured using a @@ -161,30 +223,46 @@ via the xDS HTTP filter registry, just as we do today for the [`HttpFilter.typed_config` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1233). -The parsed representation of an ECDS resource will be the same as the -representation of an HTTP filter in the parsed LDS resource. For -example, in C-core, the representation will be the -`XdsHttpFilterImpl::FilterConfig` struct mentioned above. +The parsed representation of an ECDS resource will have two fields in it: +- The parsed representation of the enclosed filter config, in the same + form as encoded in the parsed `HttpConnectionManager` config. +- A list of dependent ECDS resources. + +As an example, in C-core, the ECDS parsed representation will look +something like this: + +```c++ +struct XdsEcdsResource { + XdsHttpFilterImpl::FilterConfig config; + std::set ecds_resources_needed; +}; +``` ### gRPC Client-Side Changes The XdsDependencyManager (see [A74]) will be changed to handle ECDS. -Specifically, when it receives the LDS resource, it will look through the -list of HTTP filters to see if any of them use ECDS. For those that do, -it will start a watch for the specified ECDS resource. - -TODO: don't want XDM to have specific knowledge of the composite filter, -so how will it know to start a watch for the required ECDS resource? +ECDS resources will be added to the dependency tree in the following +cases: +- From the `ecds_resources_needed` field in a parsed LDS resource. +- From the `ecds_resources_needed` field in a parsed + `RouteConfiguration` resource (regardless of whether it is inlined + into LDS or obtained via RDS). +- From the `ecds_resources_needed` field in a parsed ECDS resource. + +Whenever it sees an update that changes the set of ECDS resources +needed, it will start watches for any ECDS resource that it is not +already watching, and it will stop watches for any ECDS resource that it +had been watching but that is no longer referenced by the configuration. The XdsDependencyManager will not return any result to the xds resolver -until all ECDS resources have been obtained. If an ECDS resource fails -validation, the XdsDependencyManager will return a transient error to -the xds resolver, which will cause the channel to stick with its -previous good config, if any. +until all ECDS resources have been obtained. If an ECDS resource returns +a data error, the XdsDependencyManager will return an error to +the xds resolver, just as if the LDS or RDS resources had seen the +error. A new field will be added to the `XdsConfig` struct returned by the XdsDependencyManager to contain all fetched ECDS resources. This field -will be a map from resource name to parsed ECDS resource. The xds +will be a map from resource name to the parsed ECDS resource. The xds resolver will use that map to get the filter config for any filter in the LDS resource that uses ECDS. From d480292b5876613b8f9ceee3704440bdba572a7b Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 4 Sep 2025 23:59:30 +0000 Subject: [PATCH 5/7] terminal filter handling --- A104-xds-ecds.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/A104-xds-ecds.md b/A104-xds-ecds.md index 690d617c2..93f49305b 100644 --- a/A104-xds-ecds.md +++ b/A104-xds-ecds.md @@ -156,6 +156,15 @@ ECDS resources needed that the filter parsing returned will be added to the `ecds_resources_needed` field in the parsed `HttpConnectionManager` config. +Note that when a filter's configuration is obtained from a separate ECDS +resource, that means that the `HttpConnectionManager` validation code +will not know the type of the filter and therefore will not know whether +the filter is a terminal filter. To avoid causing problems after config +validation time, we will support ECDS only for non-terminal filters. +This means that in the LDS validation logic, it will be considered an +error if the last filter in the chain (which must be terminal) uses ECDS +instead of an inline config. + ### RDS Resource Validation Changes Just like in LDS, we will add a new `ecds_resources_needed` field to @@ -223,6 +232,10 @@ via the xDS HTTP filter registry, just as we do today for the [`HttpFilter.typed_config` field](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1233). +As mentioned above, we support ECDS for only non-terminal filters. +Therefore, if the filter implementation for the filter type indicates +that it is a terminal filter, the ECDS resource will be NACKed. + The parsed representation of an ECDS resource will have two fields in it: - The parsed representation of the enclosed filter config, in the same form as encoded in the parsed `HttpConnectionManager` config. From 3384327cb4f02cd401309e2d11703596b82d5986 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 5 Sep 2025 00:58:47 +0000 Subject: [PATCH 6/7] clarify recursion limit --- A104-xds-ecds.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/A104-xds-ecds.md b/A104-xds-ecds.md index 93f49305b..97a388bb4 100644 --- a/A104-xds-ecds.md +++ b/A104-xds-ecds.md @@ -97,8 +97,9 @@ resource needed anywhere in that tree will be returned from the composite filter's parsing function. Note that in order to avoid potential circular dependencies or stack -overflows, we will impose a maximum recursion depth of 8 when parsing -HTTP filter configs. +overflows, we will impose a maximum recursion depth of 8 when expanding +ECDS dependencies. This will be done both on the client side (in the +XdsDependencyManager) and on the server side. ### LDS Resource Validation Changes From 917a8284ce04684dff91aaf6e4edbe3bf5d451a6 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 5 Sep 2025 01:03:16 +0000 Subject: [PATCH 7/7] add TODO --- A104-xds-ecds.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/A104-xds-ecds.md b/A104-xds-ecds.md index 97a388bb4..9c0fae670 100644 --- a/A104-xds-ecds.md +++ b/A104-xds-ecds.md @@ -101,6 +101,11 @@ overflows, we will impose a maximum recursion depth of 8 when expanding ECDS dependencies. This will be done both on the client side (in the XdsDependencyManager) and on the server side. +TODO: Do we need to take into account the full height of the HTTP +filter config stack, even if some are inlined and others use ECDS? Or +are we okay with imposing separate limits on inline and ECDS resolution, +so that max is really the product of the two? + ### LDS Resource Validation Changes We will add a new field to the parsed representation of the