Clusters as Cattle: Cluster API for Fleets on Owned Infra

Most teams reach their second Kubernetes cluster the way they reached their first: a runbook, a jump host, one person who remembers the order. That works to about four. Past that, the question is not how you build a cluster but how you describe one — precisely enough that a controller builds the next forty and, far harder, upgrades them.

A Cluster Is an Object, Not a Procedure

The gap is structural, and upstream names it from both sides. Kubernetes says of kubeadm that "by design, it cares only about bootstrapping, not about provisioning machines." Cluster API's introduction closes the other half: "while kubeadm and other bootstrap providers reduce installation complexity, they don't address how to manage a cluster day-to-day or a Kubernetes environment long term." Between those two sentences sits every hand-written provisioning script in production.

CAPI models the cluster itself as Kubernetes resources. A Cluster owns a control plane object and a set of worker pools. A MachineDeployment manages a MachineSet which manages Machine objects — deliberately the shape of Deployment, ReplicaSet and Pod, so the reconciliation semantics transfer. A KubeadmControlPlane owns replica count, version and rollout order. A Machine declares that one node should exist; something else makes it real.

That something else is a provider. Four kinds matter to a fleet operator: the core provider supplying the object model, a bootstrap provider rendering the cloud-init or Ignition that joins a blank machine, a control plane provider, and an infrastructure provider — the only component that knows your hardware. The glossary defines more; IPAM and runtime extension providers are first-class too.

The split that governs blast radius: management versus workload. Upstream defines a management cluster as "the cluster where one or more Infrastructure Providers run, and where resources (e.g. Machines) are stored," and a workload cluster as one "created by a ClusterAPI controller ... meant to be used by end-users." Every cluster is described in one place — the whole benefit, and the whole risk.

Click to zoom

The Management Cluster Is the Thing You Actually Have to Design

There is a chicken-and-egg problem: the controllers that create clusters must run on a cluster. The documented answer is an ephemeral bootstrap cluster — usually kind on a workstation — which creates the permanent one, after which clusterctl move pivots the CAPI objects across and the bootstrap cluster is destroyed.

#!/usr/bin/env bash
set -euo pipefail

# 1. Ephemeral bootstrap cluster.
kind create cluster --name capi-bootstrap

# 2. Alpha gate, OFF by default. Export before init or the
#    topology controllers never install.
export CLUSTER_TOPOLOGY=true

# 3. --infrastructure is the swappable one.
clusterctl init \
  --bootstrap kubeadm \
  --control-plane kubeadm \
  --infrastructure metal3

# 4. The PERMANENT management cluster (this file declares
#    Cluster/mgmt in namespace fleet). Wait: no kubeconfig
#    secret exists until the control plane initialises.
kubectl apply -f management-cluster.yaml
kubectl wait --for=condition=ControlPlaneInitialized \
  cluster/mgmt -n fleet --timeout=45m
clusterctl get kubeconfig mgmt -n fleet > /tmp/mgmt.kubeconfig

# 5. CNI before providers — Nodes stay NotReady without one.
kubectl --kubeconfig /tmp/mgmt.kubeconfig apply -f cni.yaml
clusterctl init --kubeconfig /tmp/mgmt.kubeconfig \
  --bootstrap kubeadm --control-plane kubeadm --infrastructure metal3

# 6. Pivot. One direction, once, while stable.
clusterctl move -n fleet --to-kubeconfig /tmp/mgmt.kubeconfig

kind delete cluster --name capi-bootstrap

The pivot is where teams get hurt, because move looks like an export. Upstream: "move has not been designed for being used as a backup/restore solution and it has several limitation for this scenario ..." and "every object's Status subresource ... is never restored during a move operation." A management cluster that lost its etcd is not recovered by re-running move; it needs the backup and restore discipline of any tier-0 datastore, designed first.

clusterctl is the imperative CLI the Quick Start uses; the Cluster API Operator is the declarative alternative, handling "the lifecycle of Cluster API providers within a management cluster using a declarative approach" and, per its README, extending clusterctl rather than replacing it. If your management cluster is itself GitOps-reconciled, that is the shape you want — the reason you would not hand-apply manifests across a fleet.

ClusterClass and Managed Topologies: Still Alpha, and What That Costs

ClusterClass makes a fleet a fleet: one template defines the shape, each Cluster references it and supplies variables. It is also the claim most overstated in 2026 write-ups. As of CAPI v1.14 the upstream page is still titled "Experimental Feature: ClusterClass (alpha)," and the feature-gate definitions register ClusterTopology: {Default: false, PreRelease: featuregate.Alpha} — identical at the v1.14.0 tag and on main. Off unless you turn it on, and no stability guarantee.

The second trap is a rename. The supported API is v1beta2; v1beta1 is deprecated since CAPI v1.11 and, per the versions reference, "in v1.16, April 2027 v1beta1 will stop to be served." v1beta2 renamed fields nearly every indexed tutorial still shows — templateRef for ref, classRef for class, and on machine-level references a bare apiGroup where ClusterClass templateRef kept apiVersion — and moved rollout strategy under spec.rollout. Copied YAML fails validation in ways the error does not explain.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: rack-standard
  namespace: fleet
spec:
  controlPlane:
    templateRef:
      apiVersion: controlplane.cluster.x-k8s.io/v1beta2
      kind: KubeadmControlPlaneTemplate
      name: rack-standard-control-plane
    machineInfrastructure:
      templateRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: Metal3MachineTemplate
        name: rack-standard-control-plane-machine
  infrastructure:
    templateRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: Metal3ClusterTemplate
      name: rack-standard-cluster
  workers:
    machineDeployments:
      - class: worker
        bootstrap:
          templateRef:
            apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
            kind: KubeadmConfigTemplate
            name: rack-standard-worker-bootstrap
        infrastructure:
          templateRef:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
            kind: Metal3MachineTemplate
            name: rack-standard-worker-machine

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: rack-07
  namespace: fleet
spec:
  topology:
    classRef:
      name: rack-standard
    version: v1.33.4
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: worker
          name: general
          replicas: 12

Upgrades Are Rollouts: Replace by Default, In Place by Exception

Two upgrade models, different rules. A standalone KubeadmControlPlane is upgraded by editing its Spec.Version field, which "will trigger a rolling upgrade of the control plane and, depending on the provider, also upgrade the underlying machine image." The same page is unambiguous: "you must always upgrade between Kubernetes minor versions in sequence." One minor at a time — the mechanics a kubeadm estate already lives with.

A managed topology is upgraded by editing the version field under spec.topology. Almost every write-up stops there; the omission is expensive. The v1.14 book's operate-cluster page documents the governing default: "a +2 minor Kubernetes version upgrade is not allowed in Cluster Topologies." Patch a stock ClusterClass two minors forward and you are refused. (That page states the ban flatly; the exception below comes from the chained-upgrade work and Runtime SDK pages, not the book.)

Chained upgrades are the opt-in. CAPI v1.12 introduced the ability to "upgrade by more than one Kubernetes minor version in a single operation": once the desired version changes, "Cluster API computes an upgrade plan, and then starts executing it." It cannot compute a plan out of nothing, and that prerequisite is rarely written down. Two sources: "setting the list of versions in the spec.kubernetesVersions field in the ClusterClass object," or "calling the runtime hook defined in the spec.upgrade field in the ClusterClass object." With neither, the +1 limit stands.

#!/usr/bin/env bash
set -euo pipefail

# STEP 1 — the prerequisite. Ordered oldest to newest, at least
# one version per minor between first and last entry. Alternative:
# spec.upgrade.external.generateUpgradePlanExtension. Set one or
# the other; with NEITHER, step 2 is rejected beyond +1 minor.
kubectl patch clusterclass rack-standard -n fleet --type=merge -p '{
  "spec": {
    "kubernetesVersions": ["v1.33.4", "v1.34.6", "v1.35.3"]
  }
}'

# STEP 2 — the multi-minor jump is now legal.
kubectl patch cluster rack-07 -n fleet --type=merge -p '{
  "spec": { "topology": { "version": "v1.35.3" } }
}'

# STEP 3 — read back the computed plan. Populated ONLY while a
# topology upgrade runs, and each version drops off once applied:
# an empty read mid-run is progress, not an error.
kubectl get cluster rack-07 -n fleet \
  -o jsonpath='{.status.controlPlane.upgradePlan}{"\n"}'
kubectl get cluster rack-07 -n fleet \
  -o jsonpath='{.status.workers.upgradePlan}{"\n"}'

Workers do not walk the same ladder. The glossary defines an efficient upgrade as one "where worker nodes skip some of the intermediate versions, when allowed by the Kubernetes version skew policy" — the worker plan is a subset of the control-plane plan. The Runtime SDK docs show a v1.30.0 to v1.33.0 jump: control plane walks v1.31.0, v1.32.3, v1.33.0; workers take only v1.32.3 and v1.33.0. CAPI v1.14 made that observable, listing "surface upgrade plan in cluster status".

Under both models is one mechanic: "Cluster API by default performs rollouts by creating a new machine and deleting the old one." On cloud VMs that is free. On a rack it is not, and the defaults show their origin — in the v1beta2 MachineDeployment maxUnavailable "defaults to 0" and maxSurge "defaults to 1", neither zero if the other is. That assumes one spare machine's worth of capacity you may not own.

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  name: rack-07-general
  namespace: fleet
spec:
  clusterName: rack-07
  replicas: 12
  # Required, and must match the template labels below.
  selector:
    matchLabels:
      nodepool: rack-07-general
  rollout:
    strategy:
      type: RollingUpdate
      rollingUpdate:
        # Inverted from the upstream default: free the host first,
        # rebuild onto it second. The pool runs one machine short per
        # rebuild — budget it, do not discover it.
        maxSurge: 0
        maxUnavailable: 1
  template:
    metadata:
      labels:
        nodepool: rack-07-general
    spec:
      clusterName: rack-07
      version: v1.33.4
      bootstrap:
        # Machine refs take apiGroup, not apiVersion.
        configRef:
          apiGroup: bootstrap.cluster.x-k8s.io
          kind: KubeadmConfigTemplate
          name: rack-standard-worker-bootstrap
      infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: Metal3MachineTemplate
        name: rack-standard-worker-machine

That is the standalone form. On a topology-managed cluster like rack-07 the same knobs live at spec.topology.workers.machineDeployments[].rollout.strategy.rollingUpdate on the Cluster, or on the MachineDeploymentClass — hand-edit the generated object and the topology controller reverts it. Upstream added the strategy explicitly for "making it easier to do immutable rollouts on bare metal / environments with constrained resources," answering a critique from our bare-metal provisioning comparison — that CAPI expects spare capacity nobody keeps idle — by removing that requirement, though not the re-bootstrap.

Removing the re-bootstrap is what in-place updates are for, and they are alpha: the InPlaceUpdates gate, enabled by EXP_IN_PLACE_UPDATES, "allows users to execute changes on existing machines without deleting the Machine and creating a new one." The gate is not an engine — the design is extension-driven, and "if the totality of the required changes cannot be covered by the defined extensions, Cluster API will fall back to the current behavior (rolling update)." Motivation is scale: "re-bootstrapping a bare metal machine takes ~10-15 mins on average." Until it lands, delete-first is the answer.

Click to zoom

Failure Modes That Only Appear at Fleet Scale

The remediation storm. MachineHealthCheck deletes unhealthy machines so they are rebuilt — right for one bad disk, wrong for one bad image. Upstream: "the default value for unhealthyLessThanOrEqualTo is 100%," meaning "the short circuiting mechanism is disabled by default and Machines will be remediated no matter the state of the cluster." Roll a broken node image and every machine failing its check is deleted and rebuilt from that same image, in parallel, until the pool is gone.

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineHealthCheck
metadata:
  name: rack-07-general
  namespace: fleet
spec:
  clusterName: rack-07
  # Must match labels on the Machines: the pool label set in
  # the MachineDeployment template above.
  selector:
    matchLabels:
      nodepool: rack-07-general
  checks:
    nodeStartupTimeoutSeconds: 1800
    unhealthyNodeConditions:
      - type: Ready
        status: Unknown
        timeoutSeconds: 300
      - type: Ready
        status: "False"
        timeoutSeconds: 300
  remediation:
    triggerIf:
      # The threshold counts UNHEALTHY, not healthy. Remediate while
      # 40% or fewer of the checked Machines are unhealthy; above
      # that, stop. Default is 100% — no short circuit at all.
      unhealthyLessThanOrEqualTo: 40%

Two details decide whether that valve exists. First, the threshold counts unhealthy Machines, not healthy ones. With 40% and 25 Machines checked, upstream states "if 10 or fewer nodes are unhealthy, remediation will be performed" and "if 11 or more nodes are unhealthy, remediation will not be performed." So it halts above 40% unhealthy, not below 40% healthy — a trip point 20 points of the pool away. Second, the selector must match labels actually on the Machines: cluster.x-k8s.io/deployment-name carries the MachineDeployment's object name, which a managed topology generates, so an explicit pool label matches more reliably. Under ClusterClass, do not hand-write the MHC — a healthCheck block on the control plane class and each machineDeployment class has the topology controller generate one with a correct selector.

The coverage gap. Upstream: "MachineHealthChecks currently only support Machines that are owned by a MachineSet or a KubeadmControlPlane." A standalone Machine — created by hand during an incident, or left over from an experiment — is remediated by nothing: a node the fleet believes is managed and is not, surfaced only by an inventory reconciliation you write yourself.

The stalled rollout that looks like a hang. Surge-first plus fixed capacity leaves a MachineDeployment sitting at one un-provisioned replacement indefinitely, with no error — the controller is correct, waiting for infrastructure that will never arrive. Alert on rollout duration, not only rollout failure.

Security and Audit Posture of the Lifecycle Layer

Classify the management cluster as tier 0 on day one. It stores the objects describing every cluster and the credentials to reach them, so compromise there is not lateral movement into one workload cluster — it is administrative access to the fleet, plus the ability to declare new machines into it. Treat its namespace boundaries as a control, and keep the node OS under it narrow.

Checked on 2026-08-16 across core cluster-api plus the AWS, Azure, vSphere, OpenStack, Hetzner and Metal3 providers: core had no published GitHub security advisories, and exactly one landed in that provider set. GHSA-rf84-wr5g-m3rp, published on the Cluster API Provider Metal3 repository on 18 May 2026, rated Moderate, no CVE assigned and patched in 1.11.8, 1.12.5 and 1.13.0, describes "multiple cross-namespace access control vulnerabilities in Cluster API Provider Metal3 allow[ing] users ... to reference, read, or claim resources belonging to other namespaces." One namespace boundary, crossed by the controller that owns your hardware.

The project is reducing its own surface: v1.14 split the API types into a separate Go module for a "considerably smaller and tightly controlled dependency tree, thus reducing exposure to CVEs from dependencies." For auditors, the declarative model is the gift — topology, version, replica counts and remediation thresholds are reviewable objects with git history, so "what shape was that cluster in March" is a query, not an interview: the posture that makes self-hosted SOC 2 tractable.

Exit Ramps: The Provider Contract as Optionality

The reason to run CAPI on owned infrastructure is not that it is easy: the provider contract is the seam. The book lists providers spanning Metal3, metal-stack, MAAS, Proxmox, OpenStack, Sidero, Tinkerbell, Harvester, KubeVirt and Oxide among others — and moving between them changes your infrastructure templates while leaving Cluster, MachineDeployment, KubeadmControlPlane and MachineHealthCheck intact. Topologies, upgrade procedures and muscle memory survive. Unusually clean, for an infrastructure layer.

The adoption boundary needs stating precisely. Managing "non-Cluster API provisioned Kubernetes-conformant clusters" is an explicit upstream non-goal: CAPI will not import or reconcile a control plane it did not create. Point it at a hand-built estate and nothing happens.

That is narrower than "CAPI is greenfield only," which is false. Incremental adoption is an explicit goal: "existing cluster lifecycle management tools should be able to adopt Cluster API in a staged manner, over the course of multiple releases, or even adopting a subset of Cluster API." An estate migrates by standing CAPI-managed clusters beside the hand-built ones and moving workloads across — slower than an import, but each migration is a rehearsal, on paths your platform already defines.

The Long Game

Proportionality is the honest test. Below a handful of clusters, a management cluster plus four providers plus an alpha gate is more moving parts than the problem has; a kubeadm runbook wins. In our engagements the crossover is not a cluster count but a change in the question: the moment somebody must know every cluster is the same shape, and prove it, the runbook has lost.

The smallest honest step is one management cluster owning one non-critical workload cluster, through one real upgrade. That answers what matters — whether your infrastructure provider is mature enough for your hardware, whether delete-first rollouts fit your maintenance windows, whether your team can debug a stuck reconcile at 3am — for the cost of two clusters, not forty.

A dated forcing function: v1beta1 stops being served in v1.16, April 2027, so anything written against the old field names has a deadline. Versions here were checked against CAPI v1.14 on 2026-08-16 and will move. The durable part is the shift in what a cluster is — from a procedure performed once and documented approximately, to an object with a desired state, a history, and a controller whose job is to make reality match. Worth owning for a decade; the provider under it, only as long as the hardware lasts.