v2.29.1

Changes by Kind

Bug or Regression

• Fix Calico apiserver RBAC permissions for Kubernetes 1.33+ (#12695, @k8s-infra-cherrypick-robot)
• Fix Cilium loadBalancer.mode rendering in Kubespray values template. (#12705, @k8s-infra-cherrypick-robot)
• Fix(calico): Add missed rbac verb watch for hostendpoints (#12644, @k8s-infra-cherrypick-robot)
• Removing external etcd member (not stacked with control plane) should now work without erroring out because the node is not in the kubernetes cluster (#12685, @k8s-infra-cherrypick-robot)

Components

• kubernetes 1.33.7
• etcd 3.5.25
• docker 28.3
• containerd 2.1.5
• cri-o 1.33.7
• cni-plugins 1.8.0
• calico 3.30.5
• cilium 1.18.4
• flannel 0.27.3
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.2.2
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.12.0
• ingress-nginx 1.13.3
• argocd 2.14.5
• helm 3.18.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.32
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

v2.29.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  /etc/hosts/ is no longer populated with all cluster nodes (#12382, @VannTen)
• Action required
  Add support for coredns_affinity to change affinity of coredns deployments, defaulting to the upstream coredns deployment's one.
  The coredns deployment's node affinity has been removed, so the coredns pods will no longer be scheduled into control-planes by default. (#11994, @HoKim98)
• Action required
  Remove support for weave network plugin (#12230, @anshuman-agarwala)
• Action required
  The tag 'master' is removed, replaced by the tag 'control-plane' (#12228, @VannTen)
• Action required
  conntrack_modules is removed; the list of conntrack modules to try to load is instead hardcoded, since there is no reason to have any other values. (#12475, @VannTen)
• Action required
  drop support for cri-o on ubuntu20. (#12233, @VannTen)

Changes by Kind

Feature

• A new sysctl_ignoreerrors value has been added, it allows to ignore errors about unknown keys that may be raised by sysctl (#12514, @bidorffOL)
• A new configuration option kubelet_static_pod_path has been added which can be used to configure path of static pod manifests OR even to disable staticPodPath setting in kubelet by setting it as empty (STIG recommendation for worker nodes) (#12433, @shaleenbathla)
• Add cilium_install_extra_flags variable (#12262, @tmurakam)
• Add external_openstack_lbaas_member_subnet_id: str (not set by default), to define a specific member-subnet-id for the openstack load balancers (#12267, @voondo)
• Add support for containerd_extra_runtime_args variable to allow injection of additional runtime configuration options into containerd CRI plugin section. (#12247, @Ujstor)
• Add support for kubeadm_image_repo variable to change kubernetes core image repository (e.g. kube-apiserver, kube-proxy). (#12128, @HoKim98)
• Add the possibility to use any values from Cilium Helm Chart (#12375, @cleman95)
• Added Prometheus Operator CRDs installation (#12441, @tico88612)
• Adds support for installing containerd as a static binary.
  Bump containerd to 2.1.3, runc to 1.3.0, nerdctl to 2.1.2 (#12377, @yankay)
• Bump ansible to 10.7.0 (#11924, @tico88612)
• Calico supports nftable mode (#12255, @tico88612)
• Control plane health check retries for apiserver, scheduler, and controller-manager are now configurable via control_plane_health_retries (default: 60). (#12452, @aman4433)
• Feat: Support certificate validity period config in kubeadm v1beta4 (#12272, @ErikJiang)
• Feat: add support for crio additional mounts (#12561, @mahendra77024)
• Introduce crio_runtime_switch boolean to allow users to switch the crio runtime by removing pods and stopping crio and kubelet during upgrade ; otherwise crio has problems when trying to work with pods created with the old runtime. (#12008, @mahendra77024)
• Introduced coredns_replicas to alter coredns deployment replicas when enable_dns_autoscaler is set to false. (#12387, @clwluvw)
• Redeploy coredns and nodelocaldns when their configurations change. (#12401, @atobaum)
• Remove --auth-anonymous if kube_api_anonymous_auth is undefined. (#12353, @psychomantys)
• Support Debian 13 Trixie (#12456, @tico88612)
• Support for custom header configuration in containerd registry mirrors via inventory and role variables. Users can now specify headers (e.g., Authorization) for registry mirrors in hosts.toml. (#12368, @pando85)
• Support kubernetes v1.33.1 (#12199, @tmurakam)
• Update cni-plugin to 1.8.0 (#12551, @tmurakam)
• Update load balancers versions to Nginx 1.28.0, Haproxy 3.1.7 (#12178, @guoard)
• Upgrade external snapshot CRD to v0.15.0 (#12308, @tico88612)
• Upgrade multus cni from 4.1.0 to 4.2.2 (#12495, @ThisIsQasim)
• [calico] Update default calico to v3.30.3 (#12523, @tmurakam)
• [flannel] upgrade to 0.26.7 (#12260, @tico88612)
• [ingress-nginx] upgrade controller to version 1.13.3 (#12604, @mzaian)

Design

• Show node to be upgraded/uncordoned in upgrade/uncordon confirmation prompt when using upgrade_node_confirm or upgrade_node_post_upgrade_confirm (#12399, @MatthiasLohr)

Bug or Regression

• Add argocd_install_checksum: str, to define the checksum of argocd_install_url (#12266, @voondo)
• Add missing addresses in kube-apiserver certificate SAN. (#12413, @hhk7734)
• Bugfix: skip etcd cert extraction if cilium identity uses crd (#12565, @mahendra77024)
• Fix Cilium installation issues (caused by templating syntax errors) when certain non-default features (encryption, etc.) are enabled (#12280, @spantaleev)
• Fix Hubble-Relay peer discovery in clusters using non-default cluster name by properly configuring clusterDomain in Cilium Helm values (#12346, @mertcancam)
• Fix cilium installation role to render cilium_config_extra_vars into helm values (#12335, @atobaum)
• Fix cilium_policy_audit_mode variable (#12569, @guoard)
• Fix error when using kubeadm_ignore_preflight_errors: ['all'] (#12606, @VannTen)
• Fix ingress-nginx DaemonSet and Service templates rendering TCP/UDP ports as strings, which prevented correct export of TCP/UDP services via NGINX ingress controller. (#12442, @MahdadGhasemian)
• Fix invalid PodSecurity admission configuration when kube_pod_security_use_default: false (#12439, @AMacedoP)
• Fix scale.yml problems with cached IP facts (#12243, @fox0430)
• Fix the Cilium cluster, which is upgraded from 2.27 to 2.28 will break
  Fix helm release re-use message when installing repeatedly (#12254, @tico88612)
• Fix the issue of etcd node addition failure caused by incorrect ETCD_INITIAL_CLUSTER configuration. (#12342, @liuxu623)
• Fix(kubeadm): Conditionally add --skip-phases flag for v1.32.0+ (#12351, @ErikJiang)
• Fix: A timeout occurs when running the offline deployment script using Podman. (#11962, @DearJey)
• Fix: When running ./manage-offline-container-images.sh register with using Podman, getting the image_id fails and the script is interrupted. (#11961, @DearJey)
• Fix: kubeadm secondary nodes use file discovery validation failed (#12132, @tico88612)
• Fixed a looping timeout bug when deleting an entire cluster (#12300, @chadswen)
• Fixed cilium_enable_bgp_control_plane config (#12430, @XuhuiSun95)
• Fixed packages installation on Alma/Rocky Linux when behind a proxy (#12264, @root-expert)
• Fixes a syntax error that made the '_bgp_config' an 'AnsibleUnsafeText' instead of a 'dict', which caused the "Calico | Process BGP Configuration" step to fail (#12258, @mathgaming)
• Make APT updates its package cache before dist-upgrade (#12465, @guoard)
• Nodelocaldns capabilities only use NET_ADMIN, not privileged (#12398, @tico88612)
• [reset] When flush_iptables: true, set IPv4/IPv6 default policies (INPUT/FORWARD/OUTPUT) to ACCEPT before flushing and delete user-defined chains to ensure a clean, non-locking reset. (#12552, @sasantk)

Other (Cleanup or Flake)

• Change the EOL Debian backports apt package to archive.debian.org (#12434, @tico88612)
• Dnsmasq directories are no longer cleaned up (#12380, @wangsifei99)
• Fix netcheck etcd image tag align with the etcd current version (#12402, @wangsifei99)
• Remove Ubuntu 20.04 support (#12301, @tico88612)
• When using timer based kubeadm certs renewal, only renew if certificates are near expiration (#12194, @panpan0000)

Components

• kubernetes 1.33.5
• etcd 3.5.22
• docker 28.3
• containerd 2.1.4
• cri-o 1.33.4
• cni-plugins 1.8.0
• calico 3.30.3
• cilium 1.18.2
• flannel 0.27.3
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.2.2
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.12.0
• ingress-nginx 1.13.3
• argocd 2.14.5
• helm 3.18.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.32
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

v2.28.1

Changes by Kind

Bug or Regression

• Add argocd_install_checksum: str, to define the checksum of argocd_install_url (#12467, @k8s-infra-cherrypick-robot)
• Add missing addresses in kube-apiserver certificate SAN. (#12505, @k8s-infra-cherrypick-robot)
• Fix Cilium installation issues (caused by templating syntax errors) when certain non-default features (encryption, etc.) are enabled (#12283, @k8s-infra-cherrypick-robot)
• Fix Hubble-Relay peer discovery in clusters using non-default cluster name by properly configuring clusterDomain in Cilium Helm values (#12374, @k8s-infra-cherrypick-robot)
• Fix cilium installation role to render cilium_config_extra_vars into helm values (#12338, @k8s-infra-cherrypick-robot)
• Fix invalid PodSecurity admission configuration when kube_pod_security_use_default: false (#12478, @k8s-infra-cherrypick-robot)
• Fix the Cilium cluster, which is upgraded from 2.27 to 2.28 will break
  Fix helm release re-use message when installing repeatedly (#12324, @k8s-infra-cherrypick-robot)
• Fix the issue of etcd node addition failure caused by incorrect ETCD_INITIAL_CLUSTER configuration. (#12352, @k8s-infra-cherrypick-robot)
• Fix(kubeadm): Conditionally add --skip-phases flag for v1.32.0+ (#12354, @k8s-infra-cherrypick-robot)
• Fix: When running ./manage-offline-container-images.sh register with using Podman, getting the image_id fails and the script is interrupted. (#12316, @k8s-infra-cherrypick-robot)
• Fixed cilium_enable_bgp_control_plane config (#12432, @k8s-infra-cherrypick-robot)
• Fixes a syntax error that made the '_bgp_config' an 'AnsibleUnsafeText' instead of a 'dict', which caused the "Calico | Process BGP Configuration" step to fail (#12394, @k8s-infra-cherrypick-robot)

Components

• kubernetes 1.32.8
• etcd 3.5.22
• docker 28.0
• containerd 2.0.6
• cri-o 1.32.0
• cni-plugins 1.4.1
• calico 3.29.5
• cilium 1.17.7
• flannel 0.22.0
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.1.0
• weave 2.8.7
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.11.3
• ingress-nginx 1.12.1
• argocd 2.14.5
• helm 3.16.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.24
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

v2.27.1

Changes by Kind

Feature

• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. (#11884, @k8s-infra-cherrypick-robot)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. (#12080, @tico88612)

Documentation

• Fix sample inventory for the reserved resource (#11922, @k8s-infra-cherrypick-robot)

Failing Test

• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl (#11956, @k8s-infra-cherrypick-robot)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (#12015, @chadswen)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary (#11957, @k8s-infra-cherrypick-robot)
• Fix: When running ./manage-offline-container-images.sh register with using Podman, getting the image_id fails and the script is interrupted. (#12314, @k8s-infra-cherrypick-robot)
• Install symlinks parroting as other control plane nodes etcd certificates (and key) on all control plane nodes, to make kubeadm works (#12192, @k8s-infra-cherrypick-robot)
• Make fallback_ip cacheable in facts (#12182, @guoard)
• [calico] Fix kubecontrollersconfigurations list permission (#12039, @k8s-infra-cherrypick-robot)

Components

• kubernetes v1.31.9
• etcd v3.5.21
• docker v26.1
• containerd v1.7.27
• cri-o v1.31.6
• cni-plugins v1.4.1
• calico v3.29.4
• cilium v1.15.9
• flannel v0.22.0
• kube-ovn v1.12.21
• kube-router v2.0.0
• multus v3.8
• weave v2.8.7
• kube-vip v0.8.0
• cert-manager v1.15.3
• coredns v1.11.3
• ingress-nginx v1.12.1
• krew v0.4.4
• argocd v2.11.0
• helm v3.16.4
• metallb v0.13.9
• registry v2.8.1
• cephfs-provisioner v2.1.0-k8s1.11
• rbd-provisioner v2.1.1-k8s1.11
• aws-ebs-csi-plugin v0.5.0
• azure-csi-plugin v1.10.0
• cinder-csi-plugin v1.30.0
• gcp-pd-csi-plugin v1.9.2
• local-path-provisioner v0.0.24
• local-volume-provisioner v2.5.0
• node-feature-discovery v0.16.4

v2.28.0

Announcement

⚠️ This is the last version of RHEL 8 that we support, and it will be deprecated in the next release, see #11872 for a discussion and reasons why.
⚠️ We have removed the Weave CNI test in previous versions and will remove it in the next release because the project has been deprecated.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Krew installation support is removed (#11824, @VannTen)
• Action required
  You should remove the leading 'v' of all explicit version of components deployed by kubespray (most notably kube_version) (#11890, @VannTen)
• Action required
  etcd_kubeadm_enabled (was deprecated) is removed. You should remove it from your inventory (#11901, @VannTen)
• gateway_api_experimental_channel is deprecated, please use gateway_api_channel and set experimental. (#11763, @tico88612)

Changes by Kind

Feature

• Add Kubernetes 1.32.x hash (#12161, @tmurakam) (#11885, @yankay) (#12003, @mzaian) (#12052, @0ekk)
• Add containerd 2.0.x hash (#11845, @mzaian) (#12011, @mzaian)
• Update runc binary to v1.2.4
  Set containerd_limit_open_file_num to 1048576 so it's configurable. (#11845, @mzaian)
• Update runc binary to v1.2.5 (#12011, @mzaian)
• Make nerdctl 2.0.3 default (#11913, @mzaian)
• Add deploy_coredns: bool (true by default), to let kubespray deploy or not coredns in kube-system (#12218, @ant31)
• Add option ubuntu_stop_unattended_upgrades to stop Ubuntu unattended upgrades (#12174, @0ekk)
• Add support for ranges: (start‑stop or single start) as an additional way to define Cilium LoadBalancer IP pools, alongside the existing cidrs: field. (#12140, @Kimcheolhui)
• Adds the script controb/offline/upload2artifactory.py for offline environments. (#11886, @bbaassssiiee)
• ArgoCD updated to version 2.14.5 to maintain compatibility with Kubernetes version 1.31. (#12041, @farshadasadpour)
• Automatically publish ingress-nginx service address if manual address is not specified and ingress-nginx is not using host network (#11879, @ThisIsQasim)
• Bump node-local-dns (k8s-dns-node-cache) image (#11981, @sathieu)
• Cilium CNI installation replaces Jinja template with Cilium CLI
  cilium_agent_custom_args and cilium_operator_custom_args are deprecated, please use cilium_agent_extra_args and cilium_operator_extra_args.
  cilium_identity_allocation_mode default change to crd.
  cilium_enable_host_legacy_routing default change to false.
  Add CIlium hubble export advanced flow log settings (cilium_hubble_export_file_max_backups, cilium_hubble_export_file_max_size_mb, cilium_hubble_export_dynamic_enabled and cilium_hubble_export_dynamic_config_content)
  Deprecated cilium_ipsec_node_encryption, replace it with cilium_encryption_node_encryption (#12101, @tico88612)
• Default etcd snapshot count to 10000 (#11997, @ErikJiang)
• Enable_dual_stack_networks deprecated, refact network stack with separate ipv4 and ipv6 (#11953, @borislitv)
• Ensure metrics port exists for nodelocaldns/nodelocaldns-second daemonsets (#11998, @Rickkwa)
• Fix cilium network plugin config issue deploying cilium 1.17 (#11986, @pedro-peter)
• For RHEL hosts, checking for subscription status timeout after rh_subscription_check_timeout (default to 3 minutes) (#12115, @VannTen)
• Gateway API can be brought forward before the CNI installation. (#12189, @tico88612)
• Improve ntp package conflict handling (#12212, @ErikJiang)
• Increase the control plane memory requirement to 2GB (#11864, @yankay)
• Network: Fix calico-kube-controller can't list the tiers resources (#12169, @cyclinder)
• Setting up a Docker image service for offline installation on a Mac (#11960, @diguage)
• Support containerd registry mirror certificate configuration (#11857, @KubeKyrie)
• Support kube-proxy nftables mode (#12060, @yankay)
• Terraform upcloud: Add possibility to setup cluster using nodes with no public IPs (#11696, @Xartos)
• Terraform: Added support for UpCloud routers and gateways (#11386, @Xartos)
• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. (#11883, @tico88612)
• Tolerations of cilium-operator deployments can be defined using the cilium_operator_tolerations group_var (#12200, @felipe88alves)
• Update default crio capabilities to allow rancher to start (#11989, @jvkassi)
• Update CI test from AlmaLinux8 to AlmaLinux9 (#11889, @yankay)
• Update kube-vip to v0.8.9 (#11983, @sathieu)
• Upgrade OpenStack Cloud Controller Manager to v1.32.0 (#12121, @tico88612)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. (#12075, @farshadasadpour)
• Upgrade kube-router to 2.1.1 (#12066, @VannTen)
• Upgrade load balancers image version to Nginx 1.27, Haproxy 3.1. (#11928, @guoard)
• Upgrade the default Docker version to 28.0 (#12070, @tico88612)
• Users can now configure hubble-export-file-max-backups and hubble-export-file-max-size-mb through the Kubespray inventory. (#12072, @ErmolenkoMaxim)
• [calico] Update default calico to v3.29.2 (#12012, @mzaian)
• [kubernetes/control-plane] Added support for structured AuthorizationConfiguration files. (#11852, @chadswen)

Documentation

• Fix documentation for offline usage by adding the 'v' prefix in download urls (#12166, @tmurakam)
• Fix path to facts.yml in node facts refresh section (#12177, @guoard)
• Fix sample inventory for the reserved resource (#11895, @anshuman-agarwala)
• No longer reserve outdated cephfs-provisioner installation and documentation (#12113, @tico88612)
• No longer reserve outdated rbd-provisioner installation and documentation (#12114, @tico88612)
• Our CRI-O default capabilities remove NET_RAW and SYS_CHROOT. (#12018, @tico88612)

Failing Test

• Add dns_autoscaler_affinity and remove in-place values. (#12165, @tico88612)
• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl (#11948, @yankay)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (#12015, @chadswen)
• Enable NRI by default on containerd (following containerd defaults) (#12152, @ShinyaIshitobi)
• File download.url's are masked unless the extra var unsafe_show_logs is true. (#11959, @bbaassssiiee)
• Fix a bug where kubeadm_certificate_key was not defined if control plane nodes were not in correct order (#11875, @Xartos)
• Fix a bug where custom TCP/UDP ports were not exposed by the ingress-nginx-controller container and service. (#11850, @commx)
• Fix broken calico Typha template when using both calico_ipam_host_local and typha_secure (#11917, @c-romeo)
• Fix broken dhclient hooks when using resolvconf (#11946, @kyrbrbik)
• Fix control plane pods deletion with proper shell quoting (#11943, @iptizer)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary (#11952, @RaulButuc)
• Fix hubble-ui deployment to not renders tls volume when the cilium_hubble_tls_generate option not configured. (#12143, @atobaum)
• Fix scale.yml problems with cached IP facts (#12020, @0ekk)
• Fix: Using the ./manage-offline-container-images.sh register command does not create a new container but registers the image in the existing container registry. (#11964, @DearJey)
• Fix: arm64 checksums for youki and kata-containers (#12173, @ErikJiang)
• Fix: missing 'v' prefix in offline image tags (#12086, @ErikJiang)
• Fix: prevent kubeadm to override coredns configuration/deployment on upgrade (#12028, @sathieu)
• Fixed an issue where the second and subsequent parameters in kubelet_cpu_manager_policy_options were ignored due to incorrect indentation. (#12123, @HoKim98)
• Fixed kube-vip to use kube-vip/kube-vip-iptables image instead of kube-vip/kube-vip when lb_fwdmethod or kube_vip_lb_fwdmethod is set to masquerade (#12145, @aviral-agarwal)
• Install symlinks parroting as other control plane nodes etcd certificates (and key) on all control plane nodes, to make kubeadm works (#12181, @VannTen)
• Kubelet-csr-approver moves to regular application installation (#12141, @tico88612)
• New Boolean default variable leave_etc_backup_files: true, set to false for uncluttered /etc directory on target nodes. (#11937, @bbaassssiiee)
• [calico] Fix kubecontrollersconfigurations list permission (#12035, @darkobas2)

Other (Cleanup or Flake)

• Binary checksums are no longer overridable from inventories or host facts (#12234, @VannTen)
• Calico-node pods no longer have a cpu limit by default (#11914, @VannTen)
• Enhance safety and validation mechanisms in the node removal process (#12085, @farshadasadpour)
• Heketi playbook (contrib) is removed. (#12091, @VannTen)
• Kubectl bash completion and alias available for Suse operation systems family (#11860, @noama-nv)
• Kubespray-defaults role is renamed to kubespray_defaults (#12202, @VannTen)
• Remove contrib/kvm-setup and contrib/mitogen. (#12093, @VannTen)
• Rename role bootstrap-os to bootstrap_os (#12203, @VannTen)
• Update KUBESPRAY_VERSION for v2.27.0 (#11854, @yankay)
• Update containerd.options key name (#12170, @flpanbin)
• Upgrade CI for openSuse 15.6 (#12074, @yankay)

Component versions

• kubernetes 1.32.5
• etcd 3.5.16
• docker 28.0
• containerd 2.0.5
• cri-o 1.32.0
• cni-plugins 1.4.1
• calico 3.29.3
• cilium 1.17.3
• flannel 0.22.0
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.1.0
• weave 2.8.7
• kube-vip 0.8.0
• cert-man...

v2.27.0

Urgent Upgrade Notes

No, really, you MUST read this before you upgrade

• Action required
  Change kubeadm_patches format to use an array of inline patch instead of patch files.
  See the example for new format. (#11521, @VannTen)
• Action required
  Removes the generation of static tokens for every node in the cluster when kube_token_auth: true (#11567, @VannTen)
• Action required
  The kubelet_node_{config_extra_args,custom_flags} are removed. Use kubelet_{config_extra_args,custom_flags} in <your_inventory>/group_vars/kube_node.yml.
  The {kube,system}_master_{cpu,memory,ephemeral-storage,pid} are removed. Use the {kube,system}_{cpu,memory,ephemeral-storage,pid} variables in <your_inventory>/group_vars/kube_control_plane.yml. kubelet_custom_flags` can no longer be a string, an array is required. (#10643, @VannTen)
• Action required
  k8s_cluster group is now automatically defined, it can be removed from your inventory if you're not using it for group_vars (#11559, @VannTen)
• Action required
  kubeadm_ignore_preflight_errors is introduced to ignore specific preflight checks from kubeadm. The previous was effectively all, so some errors might surface during upgrade, in which cases, users should add the ones they choose to ignore to that variable. (#11710, @VannTen)

Container-Managers

• Containerd upgrade to 1.7.24 (#11833, @tico88612)
  Add hashes containerd 1.7.23 (#11642, @robertvolkmann)
  Add hashes containerd 1.7.22 default (#11554, @foobaar)
• Runc upgrade to v1.2.3 (#11833, @tico88612)
  Runc add version v1.1.14 (#11574, @janosbabik)

API Change

• If you use CRI-O and want to keep runc as your container default runtime when you upgrade cluster, you must set runc_enable: true and crio_default_runtime: "runc".
  Make CRI-O's default runtime configurable
  CRI-O v1.31 default runtime change to crun
  Crun upgrade to 1.17
  Skopeo upgrade to v1.16.1 (#11601, @tico88612)

Feature

• Make Kubernetes v1.31.4 default
  Add hashes for Kubernetes 1.31.4, 1.30.8 and 1.29.12 (#11828, @tico88612)
  Add hashes for Kubernetes 1.31.3, 1.30.7 and 1.29.11 (#11737, @tico88612)
  Add hashes for Kubernetes 1.31.2, 1.30.6 and 1.29.10 (#11662, @robertvolkmann)
  Add hashes for Kubernetes 1.31.1 and 1.31.0 (#11533, @philipsabri)
  Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 (#11581, @DirkTheDaring)
• Add CI for openeuler 24.03
  Add CI Image for openeuler 24.03, 22.03 (#11689, @yankay)
• Add ResourceQuota AdmissionController plugin Configuration (#11814, @chadswen)
• Add a new CRI-O crio_root variable (#11692, @toliger)
• Add external Oracle cloud infrastructure cloud controller manager (#11378, @tico88612)
• Add optional support for Host Firewall and PolicyAuditMode features in Cilium (#11230, @ledroide)
• Add support Fedora 39/40 (#11573, @tico88612)
• Add support to use existing fips with terraform OpenStack (#11558, @anders-elastisys)
• Add the support of network isolation configuration in Multus. (#11605, @Sispheor)
• Added support for using ntpsec (#11665, @davidumea)
• Adds ingress_nginx_service_annotations variable to allow setting annotations for ingress-nginx controller service (#11544, @ThisIsQasim)
• Adds nodelocaldns_additional_configs variable (#11657, @0x4c6565)
• Allow disabling cilium hubble-ui using cilium_enable_hubble_ui variable (#10939, @pedro-peter)
• Allow to skip network configuration by setting kube_network_plugin value to none (#11844, @ant31)
• Configuration can now be supplied to ImagePolicyWebhook and PodNodeSelector admission plugins (#11471, @VannTen)
• Feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition (#11570, @mirwan)
• Feat: Kubeadm config API support v1beta4 (#11674, @tico88612)
• Iproute is installed before gathering facts (needed for getting ansible_default_ipv4) (#11816, @0ekk)
• Partial Support of Cilium v1.16+ - kube-proxy replacement var changes
  Add optional support for configuring BGP Control Plane, IP Load Balancer Pools , Legacy BGP Peer Config v1 and BGP Config v2 features in Cilium (#11620, @logicsys)
• [cilium] Make cilium 1.15.9 default (#11593, @foobaar)
• Make cri-dockerd log level configurable (#11646, @mirwan)
• Remove support Fedora 37/38 (#11600, @tico88612)
• Reset operation: remove /var/log/containers and disable service auto-boot, make sure that multi-user.target.wants is deleted. (#11501, @leeonfu)
• Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 (#11757, @ErikJiang)
• Update crictl to version v1.31.1 for Kubernetes 1.31
  Update crictl to version v1.30.1 for Kubernetes 1.30 (#11661, @robertvolkmann)
• Update multus to v4.1.0 (#11434, @ThisIsQasim)
• Upgrade CoreDNS version to v1.11.3 (#11653, @tico88612)
• Upgrade OpenStack Cloud Controller Manager to v1.31.1 (#11738, @tico88612)
• Upgrade pause container to 3.10 (#11695, @tico88612)
• [calico] Update default calico to v3.29.1 (#11798, @mzaian)
• [cert-manager] upgrade to v1.15.3 (#11668, @tico88612)
• [cri-o] Switch binaries to libexecdir
  Update youki version to 0.4.1 to fix ci. (#11584, @yankay)
• [etcd] Default version to 3.5.16 for 1.28, 1.29, 1.30, 1.31 (#11572, @janosbabik)
• [helm] Upgrade to v3.16.4, add 3.16.x checksum (#11832, @tico88612)
• [ingress-nginx] upgrade controller to version 1.12.0 (#11846, @mzaian)
• [need notice] update containerd max_container_log_line_size default value to 16384 (#11585, @KubeKyrie)
• [nerdctl] Default version to 1.7.7 (#11575, @janosbabik)

Documentation

• No longer support in-tree cloud provider, please delete or write external to the cloud_provider variable. (#11633, @tico88612)
• Remove inventory_builder scripts and contrib/dind (#11748, @VannTen)
• Update dns-stack.md reference in docs/ansible/vars.md (#11745, @emmanuel-ferdman)

Failing Test

• Fix CI: elastx cleanup security groups (#11411, @yankay)
• Update rockylinux9 base image to 9.5 (#11744, @yankay)

Bug or Regression

• Action required
  Running kubespray with --limit without cached facts is no longer supported. Improves the scaling for large clusters. (#11598, @VannTen)
• Always copy cert generation script to first etcd to pick up fixes on existing clusters (#11612, @VannTen)
• Fix Cilium agent permission can't read loadbalancerippools and secrets (#11466, @foobaar)
• Fix calico dual stack installation when using ip and ip6. (#11770, @VannTen)
• Fix collection usage for calico and other configuration depending on .sh and .conf files in Kubespray (#11707, @VannTen)
• Fix format of kubeadm-config v1beta4 (#11709, @VannTen)
• Fix kube-vip container securityContext (#11647, @KubeKyrie)
• Fix openEuler system packages installation (#11688, @VannTen)
• Fix pretty-printing (in kubectl) of nodelocaldns and coredns configmap when using dns_upstream_forward_extra_opts with an empty value option. (#11694, @VannTen)
• Fix spurious failure with 'localhost' when using scale.yml --limit <some nodes> (#11817, @VannTen)
• Fix task naming in bootstrap-os (#11714, @ErikJiang)
• Fix terraform.py on python >=3.12 (#11773, @enrico9034)
• Fix the check for cached data when using --limit (#11693, @VannTen)
• Fix the usage of --limit when using legacy groups (#11577, @VannTen)
• Fix usage of admission plugins configuration. (#11779, @VannTen)
• Fix using the default network manager in reset.yml (#11678, @KubeKyrie)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11631, @tico88612)
• Fixed: VSphere CSI and CPI drivers and are now retrieved from registry.k8s.io instead of gcr.io, as they have been deleted from the latter. Only a few recent versions are available in the new repository; if you have pinned vsphere_csi_controller, vsphere_csi_driver_image_tag or vsphere_syncer_image_tag to a version older than v3.1.2, please check if that version is available from the new repository. The same goes for external_vsphere_cloud_controller_image_tag which can no longer be latest, and should align with the running version of Kubernetes. It now defaults to v1.31.0. (#11564, @luringens)
• HA etcd cluster keeps quorum during upgrades. (#11677, @VannTen)
• Kubeadm images (kube-controller-manager,kube-scheduler,kube-apiserver,kube-proxy) are properly downloaded, including when using the download cache. (#11741, @VannTen)
• Make sure kubespray-defaults can be executed successfully by executing bootstrap-os first (#11441, @huangkevin404)
• Make upcloud csi_driver use the correct pull secret (#11597, @VannTen)
• Modifies Helm parameters wait and atomic to be set to false when using kube_network_plugin=cni to prevent deployment issues with kubelet-csr-approver. (#11704, @M-JavadHeydarpour)
• Remove invalid extraArgs entry and update template file reference (#11703, @agravgaard)
• Update calico-nopde template and remove flexvol-driver initContainer (#11634, @KubeKyrie)
• Use correct version for community.general collection (#11724, @VannTen)

Other (Cleanup or Flake)

• Cleanup older terminology, replace "master" with "control plane" (#11394, @bogd)
• Drop support for Kubernetes 1.28.x minimum version now is 1.29.x
  Drop support for CRI-O 1.28.x minimum version now is 1.29.x (#11609, @yankay)
• Fix roles/download/tasks/download_file.yml task name typo (#11684, @dmncmn)
• Optimize CA cert hash calculation with community.crypto (#11758, @ErikJiang)
• Remove pip install . support and rpm spec file (#11760, @VannTen)
• Replace deprecated unarchive.copy with unarchive.remote_src (#11207, @Payback159)
• Update KUBESPRAY_VERSION for v2.26.0 (#11511, @yankay)
• containerd_use_config_path is removed as kubespray now always use containerd config_path configuration. (#11755, @VannTen)

v2.25.1

Changes by Kind

Deprecation / Removal

• Deprecating support for Centos7 and Weave ; they are not tested anymore (#11428, @k8s-infra-cherrypick-robot)
• Remove Debian 10 support. (#11429, @k8s-infra-cherrypick-robot)ra-cherrypick-robot)

Feature

• Default to Kubernetes v1.29.10 (#11666, @enneitex)
• Add hashes for Kubernetes v1.29.9 (#11599, @robertvolkmann), Add hashes for Kubernetes v1.29.7 (#11408, @mzaian), Add hashes for Kubernetes v1.29.6 (#11355, @tico88612)
• Default to etcd v3.5.16 (#11576, @janosbabik)
• [ingress-nginx] upgrade controller to version 1.11.2 (#11464, @k8s-infra-cherrypick-robot)
• Allow to run kubespray with an empty kube_node group, to provision only the control plane (#11249, @k8s-infra-cherrypick-robot)
• CentOS 7 yum repo baseurl update (#11365, @k8s-infra-cherrypick-robot)
• Check CentOS-Base.repo exists for CentOS 7 (#11438, @k8s-infra-cherrypick-robot)
• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11339, @k8s-infra-cherrypick-robot)

Applications

• [helm] Upgrade to v3.14.4 (#11599, @robertvolkmann)

Network

• [calico] Update default calico to v3.27.4
  [calico] Fix high cpu load due to XDP program in iptables (#11476, @mzaian)

Container-Managers

• [containerd] Default to v1.7.22
  [nerdctl] Upgrade to 1.7.7
  [runc] Upgrade to v1.1.14 (#11576, @janosbabik)

Bug or Regression

• Always copy cert generation script to first etcd to pick up fixes on existing clusters (#11616, @k8s-infra-cherrypick-robot)
• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11332, @k8s-infra-cherrypick-robot)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11638, @k8s-infra-cherrypick-robot)

v2.24.3

Changes by Kind

API Change

• Default to kubernetes v1.28.14
  Default to etcd v3.5.16
  Default to containerd v1.7.22
  Default to cri-o v1.28.10
  Default to nerdctl 1.7.7
  Default to runc v1.1.14 (#11516, @VannTen)

Feature

• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11361, @k8s-infra-cherrypick-robot)
• Check CentOS-Base.repo exists for CentOS 7 (#11546, @tico88612)

Other (Cleanup or Flake)

• Update KUBESPRAY_VERSION in galaxy.yml to v2.24.3 and Update Readme.md to v2.24.1 (#11385, @yankay)

v2.26.0

Deprecation / Removal

• Deprecating support for Centos7; they are not tested anymore (#11344, @ant31)
• Remove Debian 10 support. (#11347, @tico88612)
• Remove the kubeadm_version which is always equal to kube_version (#11473, @VannTen)
• Drop support for Kubernetes 1.27.x minimum version now is 1.28.x (#11221, @mzaian)
• if you were previously only setting serializeImagePulls: false to have unlimited parallel pulls, you will need to set kubelet_max_parallel_images_pulls to a suitable value instead (#11094, @tu1h)

Feature / Major Changes

• Make kubernetes v1.30.4 default (#11455, @kokyhm)
• Add hashes for Kubernetes v1.30.3 default (#11391, @tico88612), Add hashes for Kubernetes v1.30.2 default (#11343, @tmurakam), Add hashes for Kubernetes 1.30.0, 1.30.1 and 1.30.2 (#11261, @tmurakam), Add hashes for kubernetes 1.29.7, 1.28.[11-12] (#11407, @mzaian)
• Add option ubuntu_kernel_unattended_upgrades_disabled to control unattended-upgrades for Linux kernel and all packages start with linux- on Ubuntu (#11296, @tu1h)
• Added option to configure dependencies for kubelet.service (#11297, @ledroide)
• Adds the possibility to add extra arguments to the various containers in the cinder-csi plugin.(#11169, @Payback159)
• Allow to run kubespray with an empty kube_node group, to provision only the control plane (#11248, @VannTen)
• CentOS 7 yum repo baseurl update (#11360, @tico88612)
• Check CentOS-Base.repo exists for CentOS 7 (#11402, @tu1h)
• Check if peers is defined when peering with routers (#11259, @ehsan310)
• OpenStack Cloud Controller Manager upgrade to 1.30.0 (#11358, @tico88612)
• Rename systemd module to systemd_service (#11396, @tu1h)
• User has the ability to configure calico-kube-controllers log level (#11335, @mirwan)
• User has the ability to configure local_volume_provisioner log level (#11336, @mirwan)
• User has the ability to configure netchecker components log levels (#11334, @mirwan)
• You can now disable installing OS dependencies using system's package manager by skipping system-packages tag. (#10872, @hedayat)
• kubelet_max_parallel_image_pulls represents the maximum number of image pulls in parallel (#11094, @tu1h)
• Update reset task to support Tencent OS (reset_restart_network_service_name) (#11459, @KubeKyrie)
• Add conditional checking on ubuntu kernel unattended_upgrades disabling (#11479, @tu1h)

Applications

• Bump Cinder CSI Plugin to v1.30.0 (#11374, @tico88612)
• Bump upcloud csi driver to v1.1.0 in order to enable csi volume snapshots. (#11303, @Elias-elastisys)
• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11310, @mochizuki875)
• Update kube-vip to v0.8.0 (#11156, @jisnardo)
• [cert-manager] upgrade to v1.14.7 (#11341, @tico88612)
• [cert-manager] add support v1.13.6 (#11279, @tico88612)
• [ingress-nginx] upgrade controller to version 1.11.2 (#11463, @mzaian)
• [helm] Upgrade to v3.15.4, add 3.15.x, and drop 3.13.x (#11486, @yankay)
• Add support for LB in UpCloud private zone (#11260, @davidumea)
• Bump UpCloud terraform module to v5.6.0
  UpCloud servers specify server groups to be apart of, eliminates manual rescheduling. (#11311, @robinAwallace)
• Update node-feature-discovery to v0.16.4 (#11250, @mzaian)
• Allow for configuring etcd progress notify interval and default set to 5s (#11499, @liuxu623)
• Support Gateway API CRDs install (#11376, @tico88612)
• Increase ansible timeout to 300 (#11354, @rptaylor)

Network

• [calico] Change calico default version to v3.28.1, add v3.28.0 and checksum , Update calico apiserver deployment to use new readiness probe (#11234, @ehsan310)
• [calico] add calico support v3.27.4 to fix high cpu load due to XDP program in iptables (#11476, @ehsan310)
• Add cilium_hubble_event_buffer_capacity & cilium_hubble_event_queue_size vars (#10943, @pedro-peter)
• [network] bump cni version to v1.4.0 (#10698, @cyclinder)
• Change weave CNI to community version and upgrade to the latest version (2.8.7) (#11228, @tico88612)
• [kube-ovn] update to v1.12.21 (#11445, @oilbeater)

Container-Managers

• [containerd] Make containerd 1.7.21 default (#11478, @yankay)
• [containerd] added debug config variables (#11080, @spnngl)
• [containerd] fixes wrong templating for tracing config (#11372, @ugur99)
  [runc] Upgrade to v1.1.13 (#11413, @mzaian)
• Update docker cli version 26.1.2 (#11291, @ErikJiang)

Documentation

• Update RELEASE.md for upgrading k8s (#11321, @yankay)

Bug or Regression

• Delete /etc/NetworkManager/conf.d/dns.conf on reset. (#11440, @HoKim98)
• Fix Hetzner kubernetes group names (#11232, @jmaccabee13)
• Fix: skip multus when not defined (#10934, @darkobas2)
• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11309, @mochizuki875)
• Provide missing advertise-address flag to kube-apiserver (#11387, @derselbst)
• Update reset task to support Kylin OS (reset_restart_network_service_name) (#11406, @KubeKyrie)
• Updated indentation in cni-kube-ovn.yml.j2 (L658) (#11357, @sanshah1211)
• Fix CI with fail docker pull in gitlab runner by change DOCKER_HOST (#11315, @yankay)
• Fix etcd not starting up when using a custom access address (#11388, @derselbst)
• Fix the Auto Bump PR is blocked by the label do-not-merge/release-note-label-needed by adding dependabot release-note-none label. (#11256, @yankay)
• Fix kube_reserved so it only controls kubeReservedCgroup . (#11367, @rptaylor)
• Disables reconfiguring the cluster during upgrade (remove --config option from kubeadm upgrade apply) (#11352, @tmurakam)
• Fix error in boostrap-os when git does not handle symlinks (#11508, @VannTen)
• Fix static kube-apiserver advertise address based on first control plane (#11457, @Seljuke)
• Fix incorrect member matching when removing etcd nodes (#11488, @ErikJiang)
• Fix double pop of access_ip (#11435, @rptaylor)
• Fix use super-admin.conf for kube-vip on first master when it exists to support initial k8s v1.29+ installation with kube-vip enabled (#11422, @Seljuke)

Other (Cleanup or Flake)

• Contrib playbooks are no longer included in the ansible kubespray collection (#11239, @VannTen)
• Reduced required python packages in requirements.txt (#11199, @itayporezky)
• Fix openstack cleanup by change the delete security_group order (#11299, @yankay)
• RHEL 7, Centos 7 and derivatives are no longer supported. (#11246, @VannTen)
• Use TasksMask=infinity on ostree systems for docker systemd service (#11493, @VannTen)

Supported Components

• Core
  • kubernetes v1.30.4
  • etcd v3.5.12
  • docker v26.1
  • containerd v1.7.21
  • cri-o v1.30.3 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.28.1
  • cilium v1.15.4
  • flannel v0.22.0
  • kube-ovn v1.12.21
  • kube-router v2.0.0
  • multus v3.8
  • weave v2.8.7
  • kube-vip v0.8.0
• Application
  • cert-manager v1.14.7
  • coredns v1.11.1
  • ingress-nginx v1.11.2
  • krew v0.4.4
  • argocd v2.11.0
  • helm v3.15.4
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.30.0
  • gcp-pd-csi-plugin v1.9.2
  • local-path-provisioner v0.0.24
  • local-volume-provisioner v2.5.0
  • node-feature-discovery v0.16.4

Known issues

• Upgrade of clusters with external etcd can be problematic (in particular long lived clusters, as this is not reproducible on cluster created by v2.25.1) ; see #11500 (comment) and the previous discussion for details and a workaround

Notes

• Deprecating support for Centos7
• The Ansible version has been upgrade to 9.8.0
• Change weave CNI to community version https://github.com/weaveworks/weave

Maintainers

Great respect for joining maintainers 🎉

• Adding myself (VannTen) as approver #11483 @VannTen
• Add tico88612 as reviewer #11453 @tico88612
• owners: move ant31 from emeritus_approvers to approvers #11247 @ant31

v2.24.2

Changes by Kind

Feature

• Make kubernetes v1.28.10 default (#11269, @mzaian)
• Revert 'Support CoreDNS use host network & config CoreDNS port' (#10617, @liuxu623)
• User has a possibility to modify Service type with "ingress_nginx_service_type" property in addons. (#11330, @mochizuki875)

Bug or Regression

• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11331, @mochizuki875)
• Fix CentOS 7 yum repo baseurl update (#11364, @tico88612 )

Other (Cleanup or Flake)

• Remove the archived debian apt repository when installing docker-engine (#11215, @VannTen )
• Update KUBESPRAY_VERSION in galaxy.yml v2.24.1 (#10961, @yankay)