upgrade from 1.32 to 1.33 failing #12738
Description
What happened?
tried to upgrade k8s from 1.32 to 1.33.5
and encountered upgrade failure
After that kubelet fails to start with below message
I1122 07:45:05.091542 258703 flags.go:64] FLAG: --address="0.0.0.0"
I1122 07:45:05.091633 258703 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
I1122 07:45:05.091651 258703 flags.go:64] FLAG: --anonymous-auth="true"
I1122 07:45:05.091662 258703 flags.go:64] FLAG: --application-metrics-count-limit="100"
I1122 07:45:05.091683 258703 flags.go:64] FLAG: --authentication-token-webhook="false"
I1122 07:45:05.091713 258703 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
I1122 07:45:05.091725 258703 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
I1122 07:45:05.091736 258703 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
I1122 07:45:05.091747 258703 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
I1122 07:45:05.091757 258703 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
I1122 07:45:05.091768 258703 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/bootstrap-kubelet.conf"
I1122 07:45:05.091778 258703 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
I1122 07:45:05.091788 258703 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
I1122 07:45:05.091798 258703 flags.go:64] FLAG: --cgroup-root=""
I1122 07:45:05.091807 258703 flags.go:64] FLAG: --cgroups-per-qos="true"
I1122 07:45:05.091817 258703 flags.go:64] FLAG: --client-ca-file=""
I1122 07:45:05.091827 258703 flags.go:64] FLAG: --cloud-config=""
I1122 07:45:05.091837 258703 flags.go:64] FLAG: --cloud-provider=""
I1122 07:45:05.091847 258703 flags.go:64] FLAG: --cluster-dns="[]"
I1122 07:45:05.091886 258703 flags.go:64] FLAG: --cluster-domain=""
I1122 07:45:05.091904 258703 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet-config.yaml"
I1122 07:45:05.091921 258703 flags.go:64] FLAG: --config-dir=""
I1122 07:45:05.091936 258703 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
I1122 07:45:05.091941 258703 flags.go:64] FLAG: --container-log-max-files="5"
I1122 07:45:05.091959 258703 flags.go:64] FLAG: --container-log-max-size="10Mi"
I1122 07:45:05.091964 258703 flags.go:64] FLAG: --container-runtime-endpoint="unix:///run/containerd/containerd.sock"
I1122 07:45:05.091968 258703 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
I1122 07:45:05.091972 258703 flags.go:64] FLAG: --containerd-namespace="k8s.io"
I1122 07:45:05.091976 258703 flags.go:64] FLAG: --contention-profiling="false"
I1122 07:45:05.091979 258703 flags.go:64] FLAG: --cpu-cfs-quota="true"
I1122 07:45:05.091982 258703 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
I1122 07:45:05.091986 258703 flags.go:64] FLAG: --cpu-manager-policy="none"
I1122 07:45:05.091989 258703 flags.go:64] FLAG: --cpu-manager-policy-options=""
I1122 07:45:05.092006 258703 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
I1122 07:45:05.092010 258703 flags.go:64] FLAG: --enable-controller-attach-detach="true"
I1122 07:45:05.092013 258703 flags.go:64] FLAG: --enable-debugging-handlers="true"
I1122 07:45:05.092016 258703 flags.go:64] FLAG: --enable-load-reader="false"
I1122 07:45:05.092020 258703 flags.go:64] FLAG: --enable-server="true"
I1122 07:45:05.092023 258703 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
I1122 07:45:05.092028 258703 flags.go:64] FLAG: --event-burst="100"
I1122 07:45:05.092031 258703 flags.go:64] FLAG: --event-qps="50"
I1122 07:45:05.092035 258703 flags.go:64] FLAG: --event-storage-age-limit="default=0"
I1122 07:45:05.092039 258703 flags.go:64] FLAG: --event-storage-event-limit="default=0"
I1122 07:45:05.092043 258703 flags.go:64] FLAG: --eviction-hard=""
I1122 07:45:05.092047 258703 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
I1122 07:45:05.092050 258703 flags.go:64] FLAG: --eviction-minimum-reclaim=""
I1122 07:45:05.092054 258703 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
I1122 07:45:05.092058 258703 flags.go:64] FLAG: --eviction-soft=""
I1122 07:45:05.092061 258703 flags.go:64] FLAG: --eviction-soft-grace-period=""
I1122 07:45:05.092065 258703 flags.go:64] FLAG: --exit-on-lock-contention="false"
I1122 07:45:05.092068 258703 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
I1122 07:45:05.092072 258703 flags.go:64] FLAG: --experimental-mounter-path=""
I1122 07:45:05.092075 258703 flags.go:64] FLAG: --fail-cgroupv1="false"
I1122 07:45:05.092079 258703 flags.go:64] FLAG: --fail-swap-on="true"
I1122 07:45:05.092082 258703 flags.go:64] FLAG: --feature-gates=""
I1122 07:45:05.092086 258703 flags.go:64] FLAG: --file-check-frequency="20s"
I1122 07:45:05.092090 258703 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
I1122 07:45:05.092094 258703 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
I1122 07:45:05.092097 258703 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
I1122 07:45:05.092101 258703 flags.go:64] FLAG: --healthz-port="10248"
I1122 07:45:05.092104 258703 flags.go:64] FLAG: --help="false"
I1122 07:45:05.092108 258703 flags.go:64] FLAG: --hostname-override="aed1-ksp-01"
I1122 07:45:05.092112 258703 flags.go:64] FLAG: --housekeeping-interval="10s"
I1122 07:45:05.092115 258703 flags.go:64] FLAG: --http-check-frequency="20s"
I1122 07:45:05.092119 258703 flags.go:64] FLAG: --image-credential-provider-bin-dir=""
I1122 07:45:05.092122 258703 flags.go:64] FLAG: --image-credential-provider-config=""
I1122 07:45:05.092126 258703 flags.go:64] FLAG: --image-gc-high-threshold="85"
I1122 07:45:05.092130 258703 flags.go:64] FLAG: --image-gc-low-threshold="80"
I1122 07:45:05.092133 258703 flags.go:64] FLAG: --image-service-endpoint=""
I1122 07:45:05.092137 258703 flags.go:64] FLAG: --kernel-memcg-notification="false"
I1122 07:45:05.092140 258703 flags.go:64] FLAG: --kube-api-burst="100"
I1122 07:45:05.092143 258703 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I1122 07:45:05.092147 258703 flags.go:64] FLAG: --kube-api-qps="50"
I1122 07:45:05.092150 258703 flags.go:64] FLAG: --kube-reserved=""
I1122 07:45:05.092154 258703 flags.go:64] FLAG: --kube-reserved-cgroup=""
I1122 07:45:05.092157 258703 flags.go:64] FLAG: --kubeconfig="/etc/kubernetes/kubelet.conf"
I1122 07:45:05.092161 258703 flags.go:64] FLAG: --kubelet-cgroups=""
I1122 07:45:05.092164 258703 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
I1122 07:45:05.092167 258703 flags.go:64] FLAG: --lock-file=""
I1122 07:45:05.092171 258703 flags.go:64] FLAG: --log-cadvisor-usage="false"
I1122 07:45:05.092174 258703 flags.go:64] FLAG: --log-flush-frequency="5s"
I1122 07:45:05.092177 258703 flags.go:64] FLAG: --log-json-info-buffer-size="0"
I1122 07:45:05.092184 258703 flags.go:64] FLAG: --log-json-split-stream="false"
I1122 07:45:05.092187 258703 flags.go:64] FLAG: --log-text-info-buffer-size="0"
I1122 07:45:05.092190 258703 flags.go:64] FLAG: --log-text-split-stream="false"
I1122 07:45:05.092194 258703 flags.go:64] FLAG: --logging-format="text"
I1122 07:45:05.092198 258703 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
I1122 07:45:05.092201 258703 flags.go:64] FLAG: --make-iptables-util-chains="true"
I1122 07:45:05.092205 258703 flags.go:64] FLAG: --manifest-url=""
I1122 07:45:05.092208 258703 flags.go:64] FLAG: --manifest-url-header=""
I1122 07:45:05.092213 258703 flags.go:64] FLAG: --max-open-files="1000000"
I1122 07:45:05.092221 258703 flags.go:64] FLAG: --max-pods="110"
I1122 07:45:05.092225 258703 flags.go:64] FLAG: --maximum-dead-containers="-1"
I1122 07:45:05.092228 258703 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
I1122 07:45:05.092231 258703 flags.go:64] FLAG: --memory-manager-policy="None"
I1122 07:45:05.092235 258703 flags.go:64] FLAG: --minimum-container-ttl-duration="0s"
I1122 07:45:05.092238 258703 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
I1122 07:45:05.092242 258703 flags.go:64] FLAG: --node-ip="10.124.64.9"
I1122 07:45:05.092245 258703 flags.go:64] FLAG: --node-labels=""
I1122 07:45:05.092250 258703 flags.go:64] FLAG: --node-status-max-images="50"
I1122 07:45:05.092254 258703 flags.go:64] FLAG: --node-status-update-frequency="10s"
I1122 07:45:05.092257 258703 flags.go:64] FLAG: --oom-score-adj="-999"
I1122 07:45:05.092261 258703 flags.go:64] FLAG: --pod-cidr=""
I1122 07:45:05.092264 258703 flags.go:64] FLAG: --pod-infra-container-image="registry.k8s.io/pause:3.10"
I1122 07:45:05.092268 258703 flags.go:64] FLAG: --pod-manifest-path=""
I1122 07:45:05.092271 258703 flags.go:64] FLAG: --pod-max-pids="-1"
I1122 07:45:05.092275 258703 flags.go:64] FLAG: --pods-per-core="0"
I1122 07:45:05.092278 258703 flags.go:64] FLAG: --port="10250"
I1122 07:45:05.092281 258703 flags.go:64] FLAG: --protect-kernel-defaults="false"
I1122 07:45:05.092286 258703 flags.go:64] FLAG: --provider-id=""
I1122 07:45:05.092289 258703 flags.go:64] FLAG: --qos-reserved=""
I1122 07:45:05.092292 258703 flags.go:64] FLAG: --read-only-port="10255"
I1122 07:45:05.092296 258703 flags.go:64] FLAG: --register-node="true"
I1122 07:45:05.092299 258703 flags.go:64] FLAG: --register-schedulable="true"
I1122 07:45:05.092302 258703 flags.go:64] FLAG: --register-with-taints=""
I1122 07:45:05.092307 258703 flags.go:64] FLAG: --registry-burst="10"
I1122 07:45:05.092310 258703 flags.go:64] FLAG: --registry-qps="5"
I1122 07:45:05.092313 258703 flags.go:64] FLAG: --reserved-cpus=""
I1122 07:45:05.092317 258703 flags.go:64] FLAG: --reserved-memory=""
I1122 07:45:05.092321 258703 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
I1122 07:45:05.092324 258703 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
I1122 07:45:05.092328 258703 flags.go:64] FLAG: --rotate-certificates="false"
I1122 07:45:05.092332 258703 flags.go:64] FLAG: --rotate-server-certificates="false"
I1122 07:45:05.092335 258703 flags.go:64] FLAG: --runonce="false"
I1122 07:45:05.092338 258703 flags.go:64] FLAG: --runtime-cgroups="/system.slice/containerd.service"
I1122 07:45:05.092342 258703 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
I1122 07:45:05.092345 258703 flags.go:64] FLAG: --seccomp-default="false"
I1122 07:45:05.092349 258703 flags.go:64] FLAG: --serialize-image-pulls="true"
I1122 07:45:05.092352 258703 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
I1122 07:45:05.092356 258703 flags.go:64] FLAG: --storage-driver-db="cadvisor"
I1122 07:45:05.092359 258703 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
I1122 07:45:05.092363 258703 flags.go:64] FLAG: --storage-driver-password="root"
I1122 07:45:05.092367 258703 flags.go:64] FLAG: --storage-driver-secure="false"
I1122 07:45:05.092371 258703 flags.go:64] FLAG: --storage-driver-table="stats"
I1122 07:45:05.092374 258703 flags.go:64] FLAG: --storage-driver-user="root"
I1122 07:45:05.092377 258703 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
I1122 07:45:05.092381 258703 flags.go:64] FLAG: --sync-frequency="1m0s"
I1122 07:45:05.092384 258703 flags.go:64] FLAG: --system-cgroups=""
I1122 07:45:05.092387 258703 flags.go:64] FLAG: --system-reserved=""
I1122 07:45:05.092391 258703 flags.go:64] FLAG: --system-reserved-cgroup=""
I1122 07:45:05.092394 258703 flags.go:64] FLAG: --tls-cert-file=""
I1122 07:45:05.092398 258703 flags.go:64] FLAG: --tls-cipher-suites="[]"
I1122 07:45:05.092402 258703 flags.go:64] FLAG: --tls-min-version=""
I1122 07:45:05.092405 258703 flags.go:64] FLAG: --tls-private-key-file=""
I1122 07:45:05.092408 258703 flags.go:64] FLAG: --topology-manager-policy="none"
I1122 07:45:05.092412 258703 flags.go:64] FLAG: --topology-manager-policy-options=""
I1122 07:45:05.092415 258703 flags.go:64] FLAG: --topology-manager-scope="container"
I1122 07:45:05.092419 258703 flags.go:64] FLAG: --v="2"
I1122 07:45:05.092424 258703 flags.go:64] FLAG: --version="false"
I1122 07:45:05.092428 258703 flags.go:64] FLAG: --vmodule=""
I1122 07:45:05.092433 258703 flags.go:64] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
I1122 07:45:05.092438 258703 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
I1122 07:45:05.092540 258703 feature_gate.go:385] feature gates: {map[]}
I1122 07:45:05.097742 258703 server.go:530] "Kubelet version" kubeletVersion="v1.33.5"
I1122 07:45:05.097766 258703 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1122 07:45:05.097803 258703 feature_gate.go:385] feature gates: {map[]}
I1122 07:45:05.097915 258703 feature_gate.go:385] feature gates: {map[]}
I1122 07:45:05.098071 258703 server.go:956] "Client rotation is on, will bootstrap in background"
I1122 07:45:05.099206 258703 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
I1122 07:45:05.099329 258703 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
I1122 07:45:05.099741 258703 server.go:1013] "Starting client certificate rotation"
I1122 07:45:05.100352 258703 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
I1122 07:45:05.100442 258703 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2026-11-12 09:29:36 +0000 UTC" deadline="2026-08-21 08:07:27.365334155 +0000 UTC"
I1122 07:45:05.100471 258703 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="6528h22m22.264865135s"
I1122 07:45:05.100663 258703 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/ssl/ca.crt"
I1122 07:45:05.102373 258703 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/ssl/ca.crt"
I1122 07:45:05.106898 258703 log.go:25] "Validated CRI v1 runtime API"
I1122 07:45:05.108188 258703 log.go:25] "Validated CRI v1 image API"
I1122 07:45:05.113513 258703 server.go:1446] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
I1122 07:45:05.114226 258703 fs.go:135] Filesystem UUIDs: map[5BCE-FF6A:/dev/sdc15 7b8fd397-9408-4d16-99f2-e43f417bba31:/dev/sda1 8c0a4742-2f51-40b4-b659-357cfb0bb2a3:/dev/sdc1 fa5cd571-79c1-4faf-ab44-d0cd05f9836a:/dev/dm-0]
I1122 07:45:05.114264 258703 fs.go:136] Filesystem partitions: map[/dev/mapper/vol_db-log_vol_db:{mountpoint:/var/lib/containerd major:252 minor:0 fsType:ext4 blockSize:0} /dev/sda1:{mountpoint:/mnt major:8 minor:1 fsType:ext4 blockSize:0} /dev/sdc1:{mountpoint:/ major:8 minor:33 fsType:ext4 blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:27 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containerd/io.containerd.grpc.v1.cri/sandboxes/2f48c29a0a8c40358fb53f074b99bbe19e785acff4020b70f57542b7e9664be8/shm:{mountpoint:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/2f48c29a0a8c40358fb53f074b99bbe19e785acff4020b70f57542b7e9664be8/shm major:0 minor:66 fsType:tmpfs blockSize:0} /run/containerd/io.containerd.grpc.v1.cri/sandboxes/38418e0879d202ebb94a719d331b4488a2297c8817c33ff138224c69b4be84e0/shm:{mountpoint:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/38418e0879d202ebb94a719d331b4488a2297c8817c33ff138224c69b4be84e0/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containerd/io.containerd.grpc.v1.cri/sandboxes/63633b7f103b342f2cef85112a71c2456e7b7ad820d9aa153922cc739b356e79/shm:{mountpoint:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/63633b7f103b342f2cef85112a71c2456e7b7ad820d9aa153922cc739b356e79/shm major:0 minor:116 fsType:tmpfs blockSize:0} /run/containerd/io.containerd.grpc.v1.cri/sandboxes/80c559d06c0cb9471211424e2e14d9a2ce06f0efdaddf87f9f26e9fc9ce305c5/shm:{mountpoint:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/80c559d06c0cb9471211424e2e14d9a2ce06f0efdaddf87f9f26e9fc9ce305c5/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containerd/io.containerd.grpc.v1.cri/sandboxes/920365f3170d99a72d54cd298888476fbe5571563b6234931cddf475715270bc/shm:{mountpoint:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/920365f3170d99a72d54cd298888476fbe5571563b6234931cddf475715270bc/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containerd/io.containerd.grpc.v1.cri/sandboxes/ba387387dd5ae2a45da88462696fb93b40ced8f92f80392d1eb857d459f549e0/shm:{mountpoint:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/ba387387dd5ae2a45da88462696fb93b40ced8f92f80392d1eb857d459f549e0/shm major:0 minor:130 fsType:tmpfs blockSize:0} /run/containerd/io.containerd.grpc.v1.cri/sandboxes/e7c2c438774ef228a3658f13169c2e93c21dfb795a4d38dec5ff0fb50db482b4/shm:{mountpoint:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/e7c2c438774ef228a3658f13169c2e93c21dfb795a4d38dec5ff0fb50db482b4/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/lock:{mountpoint:/run/lock major:0 minor:28 fsType:tmpfs blockSize:0} /run/snapd/ns:{mountpoint:/run/snapd/ns major:0 minor:26 fsType:tmpfs blockSize:0} /run/user/1001:{mountpoint:/run/user/1001 major:0 minor:125 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0acc1c04-28fd-4706-aec9-c992d802730c/volumes/kubernetes.io~projected/kube-api-access-dn59r:{mountpoint:/var/lib/kubelet/pods/0acc1c04-28fd-4706-aec9-c992d802730c/volumes/kubernetes.io~projected/kube-api-access-dn59r major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eb5fada-f9e2-4128-9820-91e6f422818f/volumes/kubernetes.io~projected/kube-api-access-wh4gh:{mountpoint:/var/lib/kubelet/pods/1eb5fada-f9e2-4128-9820-91e6f422818f/volumes/kubernetes.io~projected/kube-api-access-wh4gh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e81e13a-688f-4d53-8706-63898acdae9a/volumes/kubernetes.io~projected/kube-api-access-4zbls:{mountpoint:/var/lib/kubelet/pods/7e81e13a-688f-4d53-8706-63898acdae9a/volumes/kubernetes.io~projected/kube-api-access-4zbls major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b42e497-b5b0-4214-a2a8-f45cb5d122ed/volumes/kubernetes.io~projected/kube-api-access-mkdcn:{mountpoint:/var/lib/kubelet/pods/9b42e497-b5b0-4214-a2a8-f45cb5d122ed/volumes/kubernetes.io~projected/kube-api-access-mkdcn major:0 minor:64 fsType:tmpfs blockSize:0} overlay_0-114:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/e7c2c438774ef228a3658f13169c2e93c21dfb795a4d38dec5ff0fb50db482b4/rootfs major:0 minor:114 fsType:overlay blockSize:0} overlay_0-117:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/62699b1ea44078936eaddcfa46ab621065ced0ae9d747d880e58e81354afe2a8/rootfs major:0 minor:117 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/6b03f6e86d4c301ea53b5499d058426526ba1c424ea09b84eeddce03c9586306/rootfs major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/63633b7f103b342f2cef85112a71c2456e7b7ad820d9aa153922cc739b356e79/rootfs major:0 minor:121 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/a90df596d0a0c96e30eb7bebe1ddbb2b9cb2215a6055f31a7b45038db304a55c/rootfs major:0 minor:126 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/ba387387dd5ae2a45da88462696fb93b40ced8f92f80392d1eb857d459f549e0/rootfs major:0 minor:137 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/b7bd812471d39bda70d167cf4fd493f7fda031880e60ff1fbb5439aa61ea0bd3/rootfs major:0 minor:143 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/80c559d06c0cb9471211424e2e14d9a2ce06f0efdaddf87f9f26e9fc9ce305c5/rootfs major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/fc52c3da03b573a1f61ba4a5c2eb5f55c2ab7b7610fc3e637b733e51ebb20570/rootfs major:0 minor:53 fsType:overlay blockSize:0} overlay_0-56:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/33d11afea46db795a2ff615a74679af19b7d1ee5e1408aa9757eca08075b8f7b/rootfs major:0 minor:56 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/920365f3170d99a72d54cd298888476fbe5571563b6234931cddf475715270bc/rootfs major:0 minor:57 fsType:overlay blockSize:0} overlay_0-59:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/38418e0879d202ebb94a719d331b4488a2297c8817c33ff138224c69b4be84e0/rootfs major:0 minor:59 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/1861537e7672c21fcfa09908b73ce5a6bc9495cc0a3ca6fb38740af766605930/rootfs major:0 minor:98 fsType:overlay blockSize:0} overlay_0-99:{mountpoint:/run/containerd/io.containerd.runtime.v2.task/k8s.io/2f48c29a0a8c40358fb53f074b99bbe19e785acff4020b70f57542b7e9664be8/rootfs major:0 minor:99 fsType:overlay blockSize:0}]
I1122 07:45:05.118306 258703 manager.go:217] Machine: {Timestamp:2025-11-22 07:45:05.118149082 +0000 UTC m=+0.062277364 CPUVendorID:GenuineIntel NumCores:2 NumPhysicalCores:2 NumSockets:1 CpuFrequency:2593907 MemoryCapacity:7272910848 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:046d1dfdcbac4ea0a7e8240316e23fd0 SystemUUID:c257801f-5ded-3642-99dd-f872b4b065c4 BootID:d4cd3ba5-be29-424e-87f8-c197d156c9e1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:27 Capacity:3636453376 Type:vfs Inodes:887806 HasInodes:true} {Device:/var/lib/kubelet/pods/1eb5fada-f9e2-4128-9820-91e6f422818f/volumes/kubernetes.io~projected/kube-api-access-wh4gh DeviceMajor:0 DeviceMinor:65 Capacity:6362746880 Type:vfs Inodes:887806 HasInodes:true} {Device:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/ba387387dd5ae2a45da88462696fb93b40ced8f92f80392d1eb857d459f549e0/shm DeviceMajor:0 DeviceMinor:130 Capacity:67108864 Type:vfs Inodes:887806 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:overlay_0-56 DeviceMajor:0 DeviceMinor:56 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/38418e0879d202ebb94a719d331b4488a2297c8817c33ff138224c69b4be84e0/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:887806 HasInodes:true} {Device:overlay_0-59 DeviceMajor:0 DeviceMinor:59 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/var/lib/kubelet/pods/7e81e13a-688f-4d53-8706-63898acdae9a/volumes/kubernetes.io~projected/kube-api-access-4zbls DeviceMajor:0 DeviceMinor:63 Capacity:6362746880 Type:vfs Inodes:887806 HasInodes:true} {Device:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/63633b7f103b342f2cef85112a71c2456e7b7ad820d9aa153922cc739b356e79/shm DeviceMajor:0 DeviceMinor:116 Capacity:67108864 Type:vfs Inodes:887806 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/run/lock DeviceMajor:0 DeviceMinor:28 Capacity:5242880 Type:vfs Inodes:887806 HasInodes:true} {Device:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/2f48c29a0a8c40358fb53f074b99bbe19e785acff4020b70f57542b7e9664be8/shm DeviceMajor:0 DeviceMinor:66 Capacity:67108864 Type:vfs Inodes:887806 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/920365f3170d99a72d54cd298888476fbe5571563b6234931cddf475715270bc/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:887806 HasInodes:true} {Device:/var/lib/kubelet/pods/0acc1c04-28fd-4706-aec9-c992d802730c/volumes/kubernetes.io~projected/kube-api-access-dn59r DeviceMajor:0 DeviceMinor:55 Capacity:6362746880 Type:vfs Inodes:887806 HasInodes:true} {Device:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/e7c2c438774ef228a3658f13169c2e93c21dfb795a4d38dec5ff0fb50db482b4/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:887806 HasInodes:true} {Device:overlay_0-99 DeviceMajor:0 DeviceMinor:99 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/dev/sdc1 DeviceMajor:8 DeviceMinor:33 Capacity:51835101184 Type:vfs Inodes:6451200 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/var/lib/kubelet/pods/9b42e497-b5b0-4214-a2a8-f45cb5d122ed/volumes/kubernetes.io~projected/kube-api-access-mkdcn DeviceMajor:0 DeviceMinor:64 Capacity:209715200 Type:vfs Inodes:887806 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:727293952 Type:vfs Inodes:887806 HasInodes:true} {Device:/run/snapd/ns DeviceMajor:0 DeviceMinor:26 Capacity:727293952 Type:vfs Inodes:887806 HasInodes:true} {Device:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/80c559d06c0cb9471211424e2e14d9a2ce06f0efdaddf87f9f26e9fc9ce305c5/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:887806 HasInodes:true} {Device:overlay_0-117 DeviceMajor:0 DeviceMinor:117 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/dev/mapper/vol_db-log_vol_db DeviceMajor:252 DeviceMinor:0 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:14685315072 Type:vfs Inodes:917504 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:20953251840 Type:vfs Inodes:1310720 HasInodes:true} {Device:/run/user/1001 DeviceMajor:0 DeviceMinor:125 Capacity:727289856 Type:vfs Inodes:177561 HasInodes:true}] DiskMap:map[252:0:{Name:dm-0 Major:252 Minor:0 Size:21470642176 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:15032385536 Scheduler:mq-deadline} 8:16:{Name:sdb Major:8 Minor:16 Size:21474836480 Scheduler:none} 8:32:{Name:sdc Major:8 Minor:32 Size:53687091200 Scheduler:mq-deadline}] NetworkDevices:[{Name:eth0 MacAddress:00:22:48:10:1d:61 Speed:50000 Mtu:1500} {Name:kube-ipvs0 MacAddress:52:0f:ab:29:e2:b1 Speed:0 Mtu:1500} {Name:nodelocaldns MacAddress:02:40:a1:da:57:92 Speed:0 Mtu:1500} {Name:vxlan.calico MacAddress:66:9a:b8:34:dc:95 Speed:50000 Mtu:1450}] Topology:[{Id:0 Memory:7272910848 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I1122 07:45:05.118500 258703 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
I1122 07:45:05.118545 258703 manager.go:233] Version: {KernelVersion:6.14.0-1014-azure ContainerOsVersion:Ubuntu 24.04.3 LTS DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
I1122 07:45:05.118673 258703 server.go:782] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
I1122 07:45:05.118920 258703 container_manager_linux.go:267] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I1122 07:45:05.118956 258703 container_manager_linux.go:272] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"aed1-ksp-01","RuntimeCgroupsName":"/system.slice/containerd.service","SystemCgroupsName":"","KubeletCgroupsName":"/system.slice/kubelet.service","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":{"cpu":"100m","ephemeral-storage":"500Mi","memory":"256Mi","pid":"1k"},"SystemReserved":{"cpu":"500m","ephemeral-storage":"500Mi","memory":"512Mi","pid":"1k"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":-1,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
I1122 07:45:05.119141 258703 topology_manager.go:138] "Creating topology manager with none policy"
I1122 07:45:05.119156 258703 container_manager_linux.go:303] "Creating device plugin manager"
I1122 07:45:05.119172 258703 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
I1122 07:45:05.119208 258703 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
I1122 07:45:05.119253 258703 state_mem.go:36] "Initialized new in-memory state store"
I1122 07:45:05.119358 258703 server.go:1261] "Using root directory" path="/var/lib/kubelet"
I1122 07:45:05.119429 258703 kubelet.go:480] "Attempting to sync node with API server"
I1122 07:45:05.119453 258703 kubelet.go:375] "Adding static pod path" path="/etc/kubernetes/manifests"
I1122 07:45:05.119472 258703 file.go:68] "Watching path" path="/etc/kubernetes/manifests"
I1122 07:45:05.119486 258703 kubelet.go:386] "Adding apiserver pod source"
I1122 07:45:05.119499 258703 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
I1122 07:45:05.119610 258703 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
I1122 07:45:05.119628 258703 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
I1122 07:45:05.125689 258703 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="containerd" version="v2.1.5" apiVersion="v1"
I1122 07:45:05.125981 258703 dynamic_serving_content.go:116] "Loaded a new cert/key pair" name="kubelet-server-cert-files::/var/lib/kubelet/pki/kubelet.crt::/var/lib/kubelet/pki/kubelet.key"
I1122 07:45:05.126161 258703 kubelet.go:935] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
I1122 07:45:05.126263 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
I1122 07:45:05.126275 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
I1122 07:45:05.126282 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
I1122 07:45:05.126292 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
I1122 07:45:05.126299 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
I1122 07:45:05.126304 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
I1122 07:45:05.126311 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
I1122 07:45:05.126317 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
I1122 07:45:05.126324 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
I1122 07:45:05.126331 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
I1122 07:45:05.126349 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
I1122 07:45:05.126361 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
I1122 07:45:05.126387 258703 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
I1122 07:45:05.127690 258703 nodeshutdown_manager_linux.go:108] "Creating node shutdown manager" shutdownGracePeriodRequested="1m0s" shutdownGracePeriodCriticalPods="20s" shutdownGracePeriodByPodPriority=[{"Priority":0,"ShutdownGracePeriodSeconds":40},{"Priority":2000000000,"ShutdownGracePeriodSeconds":20}]
I1122 07:45:05.132333 258703 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
E1122 07:45:05.132432 258703 run.go:72] "command failed" err="failed to run Kubelet: failed to create kubelet: create user namespace manager: kubelet mappings: user: lookup username kubelet: connection refused"
What did you expect to happen?
upgrade fails at below step
TASK [kubernetes/control-plane : Kubeadm | Upgrade first control plane node to 1.33.5]
How can we reproduce it (as minimally and precisely as possible)?
upgrade from 1.32 to 1.33
OS
Ubuntu 24
Version of Ansible
ansible==10.7.0
Version of Python
Python 3.10.12
Version of Kubespray (commit)
2.29.0
Network plugin used
calico
Full inventory with variables
[kube_control_plane:children]
ksp
[etcd:children]
ksp
[kube_node:children]
ksp
ksw
[calico_rr]
[k8s_cluster:children]
kube_control_plane
kube_node
Command used to invoke ansible
ansible-playbook -i /etc/ansible/hosts -e serial=1 -i ../hosts.yml ./upgrade-cluster.yml --become --become-user=root -e kube_version=1.33.5 -e 'proxy_env={}' -e persistent_volumes_enabled=true -e metrics_server_enabled=true -e ingress_nginx_enabled=true -e ingress_nginx_insecure_port=80 -e ingress_nginx_secure_port=443 -e ingress_nginx_host_network=true --extra-vars '{"ingress_nginx_configmap": {"annotations-risk-level": "Critical", "large-client-header-buffers": "8 128k", "max-worker-connections": "40960", "proxy-body-size": "100m"}}' -e kube_controller_node_monitor_grace_period=40s -e kube_apiserver_pod_eviction_unreachable_timeout_seconds=60 -e kube_apiserver_pod_eviction_not_ready_timeout_seconds=60 -e drain_grace_period=180 -e docker_iptables_enabled=true -e resolvconf_mode=docker_dns -e calico_network_backend=vxlan -e calico_vxlan_mode=Always -e calico_ipip_mode=Never -e kubelet_max_pods=200
Output of ansible run
TASK [kubernetes/control-plane : Kubeadm | Upgrade first control plane node to 1.33.5] ********************************************************************************************************************************************************************
fatal: [aed1-ksp-01]: FAILED! => {"changed": true, "cmd": ["timeout", "-k", "600s", "600s", "/usr/local/bin/kubeadm", "upgrade", "apply", "-y", "v1.33.5", "--config=/etc/kubernetes/kubeadm-config.yaml", "--skip-phases=addon/coredns"], "delta": "0:05:05.007382", "end": "2025-11-22 03:08:29.363688", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2025-11-22 03:03:24.356306", "stderr": "W1122 03:03:24.390815 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeproxy.config.k8s.io/v1alpha1, Kind=KubeProxyConfiguration\nW1122 03:03:24.390875 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubelet.config.k8s.io/v1beta1, Kind=KubeletConfiguration\nW1122 03:03:24.390881 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=InitConfiguration\nW1122 03:03:24.390884 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=ClusterConfiguration\nW1122 03:03:24.420824 122469 utils.go:69] The recommended value for \"clusterDNS\" in \"KubeletConfiguration\" is: [10.233.0.10]; the provided value is: [169.254.25.10]\nerror execution phase control-plane: couldn't complete the static pod upgrade: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: static Pod hash for component kube-apiserver on Node aed1-ksp-01 did not change after 5m0s: context deadline exceeded\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["W1122 03:03:24.390815 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeproxy.config.k8s.io/v1alpha1, Kind=KubeProxyConfiguration", "W1122 03:03:24.390875 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubelet.config.k8s.io/v1beta1, Kind=KubeletConfiguration", "W1122 03:03:24.390881 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=InitConfiguration", "W1122 03:03:24.390884 122469 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=ClusterConfiguration", "W1122 03:03:24.420824 122469 utils.go:69] The recommended value for \"clusterDNS\" in \"KubeletConfiguration\" is: [10.233.0.10]; the provided value is: [169.254.25.10]", "error execution phase control-plane: couldn't complete the static pod upgrade: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: static Pod hash for component kube-apiserver on Node aed1-ksp-01 did not change after 5m0s: context deadline exceeded", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[upgrade] Reading configuration from the \"kubeadm-config\" ConfigMap in namespace \"kube-system\"...\n[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.\n[upgrade/preflight] Running preflight checks\n[upgrade] Running cluster health checks\n[upgrade/preflight] You have chosen to upgrade the cluster version to \"v1.33.5\"\n[upgrade/versions] Cluster version: v1.32.5\n[upgrade/versions] kubeadm version: v1.33.5\n[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster\n[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection\n[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'\n[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version \"v1.33.5\" (timeout: 5m0s)...\n[upgrade/staticpods] Writing new Static Pod manifests to \"/etc/kubernetes/tmp/kubeadm-upgraded-manifests2551077473\"\n[upgrade/staticpods] Preparing for \"kube-apiserver\" upgrade\n[upgrade/staticpods] Renewing apiserver certificate\n[upgrade/staticpods] Renewing apiserver-kubelet-client certificate\n[upgrade/staticpods] Renewing front-proxy-client certificate\n[upgrade/staticpods] Moving new manifest to \"/etc/kubernetes/manifests/kube-apiserver.yaml\" and backing up old manifest to \"/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-11-22-03-03-28/kube-apiserver.yaml\"\n[upgrade/staticpods] Waiting for the kubelet to restart the component\n[upgrade/staticpods] This can take up to 5m0s", "stdout_lines": ["[upgrade] Reading configuration from the \"kubeadm-config\" ConfigMap in namespace \"kube-system\"...", "[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.", "[upgrade/preflight] Running preflight checks", "[upgrade] Running cluster health checks", "[upgrade/preflight] You have chosen to upgrade the cluster version to \"v1.33.5\"", "[upgrade/versions] Cluster version: v1.32.5", "[upgrade/versions] kubeadm version: v1.33.5", "[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster", "[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection", "[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'", "[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version \"v1.33.5\" (timeout: 5m0s)...", "[upgrade/staticpods] Writing new Static Pod manifests to \"/etc/kubernetes/tmp/kubeadm-upgraded-manifests2551077473\"", "[upgrade/staticpods] Preparing for \"kube-apiserver\" upgrade", "[upgrade/staticpods] Renewing apiserver certificate", "[upgrade/staticpods] Renewing apiserver-kubelet-client certificate", "[upgrade/staticpods] Renewing front-proxy-client certificate", "[upgrade/staticpods] Moving new manifest to \"/etc/kubernetes/manifests/kube-apiserver.yaml\" and backing up old manifest to \"/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-11-22-03-03-28/kube-apiserver.yaml\"", "[upgrade/staticpods] Waiting for the kubelet to restart the component", "[upgrade/staticpods] This can take up to 5m0s"]}
Anything else we need to know
No response