Kubernetes Upgrade Debt: Skew, Dead APIs, Safe Paths

Upgrade debt does not announce itself. It accumulates as a series of individually reasonable deferrals — a release freeze to protect, a controller nobody wants to touch, a quarter spent on something more visible — and then arrives at once, as a cluster three minors behind a version whose patch stream has stopped.

Search for guidance and you get one shape of answer: run a scanner, fix the deprecated apiVersions, upgrade. That step is real and it is not the binding one. The calendar, the component ordering and the ecosystem around the cluster decide whether it lands.

Upgrade Debt Is a Schedule, Not an Incident

Two published numbers set the entire budget. Kubernetes releases happen approximately three times per year. And the community supports each active patch release series "for a period of roughly fourteen (14) months" — twelve standard months, then two months of maintenance mode, during which Release Managers cut releases only for CVE-assigned vulnerabilities, dependency issues including base-image updates, and critical core-component issues.

Divide one into the other and three minors carry patches at any given moment. That is the whole of your slack: the version you run can sit roughly two minors behind the newest and still have a patch stream. It is fixed, and you cannot negotiate it — only spend it.

As of 30 July 2026 the active branches are 1.34, 1.35 and 1.36. Branch 1.34 enters maintenance mode on 27 August 2026 and reaches end of life on 27 October 2026; 1.35 on 28 December 2026 and 28 February 2027; 1.36 on 28 April 2027 and 28 June 2027. Those are published targets rather than guarantees, so plan the cadence against them and re-read the page each rung.

One hygiene note that costs nothing: pin against release tags, not the documentation page. Kubernetes 1.36.0 was published on 22 April 2026 and the current 1.36 patch, v1.36.3, on 23 July 2026 — yet when we checked on 30 July 2026, the kubernetes.io patch-releases page still described 1.36.3 as the *next* patch. Documentation lags its own tags.

Read the Skew Policy as a Scheduling Constraint

The version-skew policy is usually filed under compatibility. Read it as a scheduling document and it hands you the order of operations for a fleet.

kube-apiserver is the reference; everything else is measured against it. kube-controller-manager, kube-scheduler and cloud-controller-manager must not be newer than the apiservers they talk to and may be at most one minor older, to allow live upgrades. The kubelet must not be newer and may be up to three minors older — against an apiserver at 1.36, kubelets are supported at 1.36, 1.35, 1.34 and 1.33. kubectl is the one component permitted to be ahead: within one minor either side, so 1.37, 1.36 and 1.35.

That produces an order, not a preference: control plane first, workers trailing. And because the kubelet window is three minors wide, a node pool can legitimately lag by three releases — which is how fleets end up with an apiserver at 1.36 and workers untouched since 1.33, one rung from falling out of policy.

Then the clause most guides omit, the one governing the change window itself. If version skew exists between kube-apiserver instances in an HA cluster, the allowed range for other components narrows to the older instance. The policy's own example: with apiservers at 1.36 and 1.35, kube-proxy is supported at 1.35, 1.34 and 1.33 — 1.36 is not. Mid-upgrade, an HA control plane sits precisely in that state, so moving kube-proxy alongside the second apiserver opens a window where the fleet is out of policy. Stage them.

Two rules define the ladder itself. The skew policy states that "project policies for API deprecation and API change guidelines require kube-apiserver to not skip minor versions when upgrading, even in single-instance clusters." kubeadm's upgrade documentation says it flatly: "Skipping MINOR versions when upgrading is unsupported." So 1.34 to 1.36 in one move is not a path, however much downtime you were willing to buy.

It is worth naming the rule this is *not*. Deprecation Rule #4b says the preferred and storage version for a group "may not advance until after a release has been made that supports both the new version and the previous version," so an operator can upgrade and roll back "without converting anything to the new API version." That governs how APIs evolve, not the order in which you upgrade a cluster.

Click to zoom

The Pre-Upgrade Scan, and the Layer It Cannot See

Scan in layers, because no single source sees all the exposure. The live-cluster layer is the strongest signal and the most under-used. Since v1.19, a request to a deprecated REST endpoint returns a Warning header, adds a k8s.io/deprecated: "true" annotation to the audit event, and sets an apiserver_requested_deprecated_apis gauge to 1. That gauge carries labels for group, version, resource and subresource — joinable to apiserver_request_total — plus a removed_release label naming the release in which the API stops being served. Not a guess about your manifests: a list of what is actually being called.

The repository layer catches what the metric cannot: manifests not applied today but applied the moment a pipeline syncs — which, in a GitOps fleet, is every manifest in the repo. Pluto is the tool to reach for; it scopes itself to deprecated apiVersions "in their code repositories and their helm releases," and released v5.24.1 in July 2026. kubent, its stablemate in every upgrade guide, deserves a check before you trust its silence: no tagged release newer than 0.7.3, from August 2024. Tags are not code — but a version table not re-cut in two years tells you less than you would like.

Each layer is blind to the other. The metric cannot see an unapplied manifest, resets when the apiserver process restarts — so a freshly-rolled control plane reports a clean sheet it has not earned — and is per-instance: in HA, scrape every apiserver, not whichever one your client reached. Pluto cannot see an in-cluster controller reaching a deprecated endpoint through a client library, and scopes itself to apiVersions, not CRD schema drift. Run both.

#!/usr/bin/env bash
set -euo pipefail

MANIFESTS="${MANIFESTS:-./clusters}"

# Layer 1 — component skew, straight off the live objects.
kubectl version -o json | jq -r '.serverVersion.gitVersion, .clientVersion.gitVersion'
kubectl get nodes -o json |
  jq -r '.items[] | "\(.metadata.name)\t\(.status.nodeInfo.kubeletVersion)"'

# Layer 2 — what is actually calling a deprecated endpoint, and which
# release drops it. Resets on apiserver restart: read it before you roll.
kubectl get --raw /metrics |
  grep '^apiserver_requested_deprecated_apis' ||
  echo "none recorded since this apiserver process started"

# Layer 3 — what a pipeline would reapply the moment it syncs.
pluto detect-files -d "$MANIFESTS"
pluto detect-helm

echo "Layer 4 is manual: read the CHANGELOG for the target minor, not only the guide."

And then the layer nearly everyone misses. The Deprecated API Migration Guide is the canonical removal reference, and its newest "Removed APIs by release" chapter is v1.32, with no chapter for v1.33 through v1.36 as of 30 July 2026. That v1.32 entry is flowcontrol.apiserver.k8s.io/v1beta3 for FlowSchema and PriorityLevelConfiguration, replaced by the v1 version available since v1.29.

Read that as "nothing has been removed since 1.32" and you will be wrong. The guide tracks the GA and beta removal calendar, which has been quiet. The release notes have not. Kubernetes 1.33 removed support for the v1alpha1 versions of ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding, and dropped inactive serving code for authentication.k8s.io/v1alpha1. Kubernetes 1.35 stopped serving storage.k8s.io/v1alpha1 VolumeAttributesClass. Kubernetes 1.36 removed the scheduling.k8s.io/v1alpha1 Workload API in favour of a new v1alpha2, removed v1alpha1 SnapshotMetadataService support, and removed the v1alpha1 WebhookAdmissionConfiguration deprecated back in v1.17. None appear in the guide's removal chapters.

Sequential Upgrades, One Minor at a Time

On kubeadm the rung has a fixed shape, and it does not begin where most runbooks do. Every node first upgrades the kubeadm binary itself and confirms it with kubeadm version: plan and apply act through the tool present on that host, not the version you had in mind. Only then does kubeadm upgrade plan "check that your cluster can be upgraded" and list the versions available.

kubeadm's own documentation warns that requests "to the server will stall while the new etcd static pod is restarting," and suggests stopping kube-apiserver a few seconds beforehand, on control-plane nodes, to let in-flight requests complete. The documented form is three steps — SIGTERM, a twenty-second wait, then the upgrade — and the wait is the mechanism: firing the signal and the upgrade back to back implements none of it. A deliberate outage in exchange for not dropping calls on the floor: runbook, not memory.

Workers follow one at a time, and each repeats that same opening: upgrade kubeadm, then run kubeadm upgrade node, which for a worker "upgrades the local kubelet configuration." Skip it and the packages move while the config they read does not. Only then drain. On Talos the sequence collapses into talosctl upgrade-k8s, which will "automatically update the components needed to upgrade Kubernetes safely" and supports --dry-run. Whichever executor you use, take the etcd snapshot and rehearse a restore first — an untested snapshot is not a rollback plan.

#!/usr/bin/env bash
set -euo pipefail

NODE="${1:?usage: rung.sh <node> first-cp|other-cp|worker}"
ROLE="${2:?usage: rung.sh <node> first-cp|other-cp|worker}"
FROM="1.35"; TO="1.36"; PKG="1.36.3-*"   # apt-cache madison kubeadm

if [ "$(( ${TO#*.} - ${FROM#*.} ))" -ne 1 ]; then
  echo "refusing ${FROM} -> ${TO}: skipping a MINOR is unsupported" >&2
  exit 1
fi

# Step 1 — every node: upgrade kubeadm itself, then prove it.
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y "kubeadm=${PKG}"
sudo apt-mark hold kubeadm
kubeadm version   # must report ${TO} before going on

# Step 2 — role picks the verb; only first-cp applies.
case "$ROLE" in
  first-cp)
    sudo kubeadm upgrade plan
    sudo killall -s SIGTERM kube-apiserver   # control plane only
    sleep 20                                 # the wait IS the mitigation
    sudo kubeadm upgrade apply "v${PKG%-*}"
    # Required here by the kubeadm docs, and only here: upgrade your CNI
    # provider plugin per its own instructions, before the drain below.
    ;;
  other-cp | worker)
    sudo kubeadm upgrade node   # worker: rewrites kubelet config
    ;;
  *) echo "unknown role: ${ROLE}" >&2; exit 1 ;;
esac

# Step 3 — only now drain: a minor kubelet bump requires it.
kubectl drain "$NODE" --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y "kubelet=${PKG}" "kubectl=${PKG}"
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon "$NODE"

One step gets skipped almost universally, because nothing breaks when you do. Removing a served version does not strand the objects already persisted in etcd: for every removal the migration guide states that "All existing persisted objects are accessible via the new API," because the apiserver converts on read. What lingers is the storage encoding. Kubernetes "relies on API data being actively re-written" for that maintenance — stale storage versions and encryption at rest both — and rotating an at-rest key re-encrypts nothing until each object is written again. StorageVersionMigration forces the rewrite; its prerequisites are why most readers cannot run it.

apiVersion: storagemigration.k8s.io/v1beta1
kind: StorageVersionMigration
metadata:
  name: flowschemas-to-v1
spec:
  resource:
    group: flowcontrol.apiserver.k8s.io
    resource: flowschemas

Failure Modes: Where Upgrades Actually Break

Upgrades rarely fail on the apiVersion you scanned for. They fail in the ecosystem wrapped around the cluster, and five failures dominate.

• Helm-managed CRDs. Helm states plainly that "there is no support at this time for upgrading or deleting CRDs using Helm" — a deliberate decision, taken because of the danger of unintentional data loss. Every chart-installed operator therefore has CRD schemas frozen at whatever version first installed them. No apiVersion scanner reports it: the manifests are correct, the cluster is stale.
• Fail-closed admission webhooks. A ValidatingWebhookConfiguration with failurePolicy: Fail and a backend that is down, unmigrated or holding a stale CA bundle does not degrade the upgrade — it stops it, by rejecting the writes the upgrade must make. Enumerate webhooks before the rung.
• PodDisruptionBudgets that stall the drain. A PDB allowing zero disruptions, or a single-replica workload whose budget cannot be satisfied, turns kubectl drain from a step into an indefinite wait — the failure that makes a two-hour window a five-hour one.
• Node image skew. The kubelet window is three minors wide, so nothing complains while node images age. Then one more rung moves the apiserver and a pool at the edge falls out of the window in a single change.
• Storage left on a superseded version. The scan looked at manifests; the stale encoding sits in etcd, out of reach of an at-rest key rotation. Only one of those surfaces is in your repository.

Every one is discoverable before the change window, and none by the tool people run. Hence a gate rather than a checklist: something that must return a written verdict before the rung starts.

Click to zoom

The Audit Surface: End-of-Life Software in the Request Path

Upgrade debt stops being a preference the moment unmaintained software sits in the request path, and March 2026 supplied the canonical example. Ingress NGINX was retired: "there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered," repositories read-only. Critically, "existing deployments of Ingress NGINX will not be broken" — which is why the debt is easy to carry. Nothing breaks; the fix path ceases to exist.

For scale, CVE-2025-1974 in ingress-nginx carries a CVSS 3.1 base score of 9.8 CRITICAL as assessed by the Kubernetes CNA. That one was patched while the project still lived; the class is the point — after retirement, the next such finding has no upstream remediation, and an auditor need not understand ingress controllers to write that up. The Security Response Committee published the exposure check: kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx.

The inverse is the opportunity. A rehearsed cadence is among the cheapest auditable controls a platform team owns, because the rung emits its own evidence: a dated scan output with a zero-finding verdict, a restore-verified snapshot reference, the kubeadm upgrade plan transcript, and a post-upgrade metric read showing the gauge back at zero. Retain those four and "we upgrade on a defined cadence" becomes a sampleable artifact.

Exit Ramps: Don't Buy the Next Lock-In While Paying Off This One

Migrating off a retired controller is the moment to notice which manifests are portable. Ingress2Gateway translates Ingress objects to Gateway API resources, and got materially better at the annotation problem: before 1.0 it "only supported three Ingress-NGINX annotations," and for 1.0 it "supports over 30 common annotations." Its current release is v1.2.0, published 7 July 2026. It warns about what did not translate; those warnings are the work, and the data-path design that follows is a separate decision.

The general lesson: a vendor-scoped API deprecates on the vendor's clock, not upstream's. Red Hat deprecated the OpenShift-specific DeploymentConfig API in OpenShift Container Platform 4.14, and the lifecycle is its own too: Maintenance Support "ends at 18 months after GA." Not a criticism of the platform — an argument for knowing which manifests survive leaving it.

Price the stall on both sides of the fence. On EKS, as of 30 July 2026, standard support runs 14 months at $0.10 per cluster-hour, then 12 months of extended support at $0.60 — a sixfold control-plane price for not upgrading, per cluster, across a fleet. Self-hosting does not delete that cost; it converts it into engineering time you budget. Engineering time buys a capability you keep. Extended support buys a delay you rent.

#!/usr/bin/env bash
set -euo pipefail

# The Security Response Committee's own exposure check.
kubectl get pods --all-namespaces \
  --selector app.kubernetes.io/name=ingress-nginx

# Translate Ingress objects to Gateway API. Read every warning it prints.
ingress2gateway print --providers=ingress-nginx --all-namespaces > gwapi.yaml

# Rehearse the control-plane move without performing it (Talos).
talosctl --nodes "${CP_IP:?control-plane IP}" upgrade-k8s --to 1.36.3 --dry-run

The Long Game: A Cadence You Can Hold for a Decade

Every constraint here is knowable a year ahead, which makes upgrades one of the few operational risks you can retire by scheduling. A rung climbed on an ordinary Tuesday is a different exercise from one climbed because support expires Friday. What survives staff turnover is small: a named owner for the calendar, four evidence artifacts per rung, a rehearsed runbook rather than a remembered one, and a CI gate so manifests cannot drift backwards. The last makes the scan a repository-enforced rule — the same discipline as any other policy gate.

name: api-version-gate
on: pull_request

jobs:
  deprecated-apis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - name: Install Pluto
        uses: FairwindsOps/pluto/[email protected]
        with:
          # The ref above pins the action. THIS pins the scanner binary.
          IMAGE_TAG: v5.24.1
      - name: Fail the PR on a deprecated apiVersion
        run: pluto detect-files -d clusters/

The reckoning is optional. It arrives only for teams that treat upgrades as an incident to survive rather than a cadence to hold. Held properly, the ladder is boring — and boring is the entire objective for a system you intend to still be running in ten years.