aep-8905 native sidecar support

Author: jklaw90

vertical-pod-autoscaler/enhancements/8905-native-sidecar-support/README.md

@@ -0,0 +1,74 @@
+# KEP-8905: Native Sidecar Support
+
+<!-- toc -->
+- [Summary](#summary)
+   - [Goals](#goals)
+   - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Design Details](#design-details)
+   - [Recommendations](#recommendations)
+   - [Update / Admission](#update--admission)
+   - [Test Plan](#test-plan)
+   - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This proposal adds support for native sidecar containers (init containers with `restartPolicy: Always`) in Vertical Pod Autoscaler.
+
+Kubernetes 1.28 introduced native sidecar containers. These are init containers that start before the main containers and continue running during the lifecycle of the Pod. VPA currently supports standard containers and regular init containers, but it should also support recommending resources for these new native sidecar containers to ensure they are right-sized.
+Addresses [issue #7229](https://github.com/kubernetes/autoscaler/issues/7229)
+
+### Goals
+
+- Allow VPA Recommender to generate resource recommendations for native sidecar containers.
+- Ensure VPA Updater and Admission Controller can apply recommendations to native sidecar containers.
+
+### Non-Goals
+
+- Support for sidecar containers in Kubernetes versions older than 1.28.
+
+## Proposal
+
+The proposal is to introduce a new feature gate `NativeSidecar` in VPA. When enabled, VPA components will recognize and handle native sidecar containers.
+
+## Design Details
+
+### Recommendations
+
+The `ClusterFeeder` in Recommender is updated to identify init containers with `restartPolicy: Always` as native sidecars. Native sidecars will be treated exactly like normal containers from recommenders perspective. The VPA custom resource definition will remain the same due to the fact container names are unique within a pod. All recommendation status updates for native sidecars will added with other container recommendations.
+
+### Update / Admission
+
+The patch generation logic is updated to target `/spec/initContainers` for native sidecar containers. Updater and Admission will work almost entirely the same due to the unique container naming of a pod they will be able to find which container needs updates from the VPA status.
+
+### Test Plan
+
+The following test scenarios will be added to e2e tests.
+
+- Admission applies recommendations to native sidecars.
+- Updater will update sidecar container resources in-place or evict.
+- Admission will patch sidecar container resources.
+- When the feature gate `NativeSidecar` is false VPA components will not modify native sidecars.
+
+### Upgrade / Downgrade Strategy
+
+#### Upgrade
+
+On upgrade of the VPA to 1.6.0 (tentative release version), nothing will change,
+VPAs will continue to work as before.
+
+Users can use the new `NativeSidecar` by enabling the alpha Feature Gate (which defaults to disabled)
+by passing `--feature-gates=NativeSidecar=true` to the VPA components.
+
+#### Downgrade
+
+On downgrade of VPA from 1.6.0 (tentative release version), nothing will change.
+VPAs will continue to work as previously. Checkpoints may contain sidecar resource information until updated, but updater and admission will modify sidecar resources.
+
+## Alternatives
+
+### Treat as Standard Containers
+
+We could treat them as standard containers, but they are technically init containers in the Pod spec, so the patch path would be incorrect (`/spec/containers` vs `/spec/initContainers`).