K3s and Rancher Fleet: Kubernetes at the Disconnected Edge

A far-edge fleet breaks every assumption a datacenter platform team takes for granted. The clusters are not in a rack you can walk to; they are on oil platforms, ships, wind turbines, factory floors, and cell towers. The backhaul is cellular, satellite, or a rural line that drops for minutes or days, and there is often no inbound path to the site at all, by policy or by NAT. There can be thousands of sites, each a single-node or three-node K3s install nobody will ever log into interactively. The question is not "how do I deploy to Kubernetes" — it is "how does a system I cannot reach, on a network I cannot trust, keep converging to the state I declared, and tell me when it cannot?"

This article is the operational playbook for that fleet: K3s as the edge runtime, Rancher Fleet as a pull-based GitOps control plane built for intermittent connectivity, an air-gapped image-distribution scheme that survives sites with no registry access, per-site overlays that do not collapse into repository sprawl, and a reconnection-reconciliation model that makes an offline site a normal state rather than an incident. Gartner's much-quoted forecast put 75% of enterprise-generated data as created and processed outside a traditional centralized data center or cloud by 2025 — a shift that turns K3s from a homelab curiosity into a production fleet-management problem measured in site-count and staleness, not uptime.

The audience is the people who own that fleet: platform and infrastructure leads standing up edge estates, SREs who carry the pager for sites in three time zones, and security and compliance answering for machines running unattended in exposed locations. Sovereignty is the through-line — a fleet you can only operate while a vendor's cloud console is reachable is not a fleet you own.

Why the Far Edge Is Not Just Multi-Cluster GitOps With More Clusters

It is tempting to treat this as a scaling problem — the same datacenter multi-cluster GitOps you already run, pointed at more endpoints. It is not. Datacenter GitOps assumes clusters are continuously reachable, well-provisioned, and homogeneous, and it can lean on a control plane that reaches out to them. The far edge violates all three assumptions at once, and each violation changes the architecture rather than just the numbers.

• Connectivity is intermittent and often one-way. The control plane can never assume it can reach a site. Any design that pushes — SSH, an agent the server dials into, a reconcile that blocks on the target — stalls the instant a link drops, and links drop constantly.
• Sites are air-gapped or bandwidth-starved. Pulling a 400 MB image over a metered satellite link, times a thousand sites, is not a rollout — it is an outage and a bill. Images must be staged locally; the network carries manifests and digests, not layers.
• Configuration is heterogeneous by location. Site 12 has a different sensor, VLAN, or regulatory zone than site 900. You need per-site variation without a per-site pipeline, or the config surface explodes into thousands of near-duplicate files nobody can audit.
• Nobody is coming. There is no remote-hands SSH for most sites. Provisioning has to be zero-touch, recovery has to be automatic, and the running state has to survive the operator being completely absent for the machine's entire life.

Those four constraints point at one architecture: pull-based reconciliation, local image custody, label-driven overlays, and autonomy on disconnect. Fleet and K3s were built around exactly this shape — the default far-edge stack rather than a stretched datacenter tool.

Pull-Based by Construction: Rancher Fleet at the Edge

Fleet is a GitOps controller designed to manage clusters at a scale — its documentation cites up to a million — that no push model reaches. Its unit of work is the Bundle: a controller in a management cluster watches a GitRepo resource, renders the repository's manifests and Helm charts into Bundles, and each downstream cluster runs a fleet-agent that pulls the Bundles targeted at it, applies them, and reports status back best-effort. The direction of travel is the whole point: the agent initiates every connection outward, so the controller never needs a route into a site, and a cluster unreachable for six hours is not an error — just an agent that has not checked in yet. That outward connection often rides a mesh overlay back to the management cluster, and whoever runs the overlay's coordination plane decides which machines are on the network.

This inversion is what makes intermittent WAN survivable. In a push model, the reconcile loop is coupled to the reachability of every target, so one flapping satellite link can back up a queue or trip a timeout that touches unrelated sites. In Fleet's pull model, each site's fleet-agent owns its own reconcile cadence against the management cluster (pulling BundleDeployments, not cloning Git), so the failure domain is exactly one site — and the controller's job shrinks to rendering desired state and aggregating status, both of which it does whether or not any given agent is online.

Click to zoom

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: edge-baseline
  namespace: fleet-default
spec:
  repo: https://git.internal/edge/fleet-config
  branch: main
  # Poll git less often on the management cluster: trades sync latency for git/gitjob load.
  pollingInterval: 5m
  paths:
    - bundles/baseline      # everything every site runs
  targets:
    # Select clusters by label, not by name — onboarding is a label.
    - clusterSelector:
        matchLabels:
          fleet.cattle.io/tier: far-edge

# bundles/baseline/fleet.yaml
rolloutStrategy:
  maxUnavailable: 5%
  maxUnavailablePartitions: 0
  partitions:
    - name: canary
      # Unavailable must exceed MaxUnavailable to mark the partition NotReady.
      # maxUnavailable: 0 is the strict canary: one failure blocks later waves.
      maxUnavailable: 0
      clusterSelector:
        matchLabels:
          wave: canary
    - name: remaining
      maxUnavailable: 5%
      clusterSelector:
        matchLabels:
          # Mutually exclusive with wave: canary — do not also set
          # wave: remaining on canary sites (a cluster can match every
          # partition whose selector it satisfies).
          wave: remaining

The rolloutStrategy lives in the bundle's fleet.yaml, not on the GitRepo. A wave-based rollout needs an explicit canary partition with mutually exclusive cluster labels (for example wave: canary versus wave: remaining — canary sites must not also match the remaining selector) and maxUnavailable: 0 on the canary so a single failed canary site marks that partition NotReady. Fleet treats a partition as unavailable only when Unavailable exceeds MaxUnavailable, so maxUnavailable: 1 would still count a one-site failure as Ready. With maxUnavailablePartitions: 0, later partitions stay blocked until the canary partition is Ready — non-negotiable when a bad push could mean physically visiting a thousand locations. Sites still wear the GitRepo target label (fleet.cattle.io/tier: far-edge above) plus exactly one wave label; a targeted site with no wave match is discarded from the partitioned rollout. Without partitions, Fleet stages up to maxNew (default 50) sites before readiness can pause the rest — so for fleets under that batch size, a global maxUnavailable alone is not a canary.

Air-Gapped Image Distribution: The Part That Actually Breaks

Manifests are small; images are not. The single most common far-edge failure is a Bundle that applies cleanly and then sits in ImagePullBackOff because the site cannot reach a registry — and you cannot SSH in to diagnose it. Image distribution is therefore not an optimization; it is the difference between a fleet that self-heals and one that quietly rots. K3s gives you two composable mechanisms. First, a registries.yaml placed in /etc/rancher/k3s/ before install rewrites pulls for upstream registries to a mirror you control, and --disable-default-registry-endpoint stops any fallback to the internet — so a site that loses its link cannot silently try to reach docker.io and hang.

mirrors:
  docker.io:
    endpoint:
      - "https://registry.site.local:5000"
  ghcr.io:
    endpoint:
      - "https://registry.site.local:5000"
  quay.io:
    endpoint:
      - "https://registry.site.local:5000"
configs:
  "registry.site.local:5000":
    tls:
      # Site registry served under your own internal PKI, not a public CA.
      ca_file: /etc/rancher/k3s/site-registry-ca.crt
---
# Install flags that make the air-gap explicit and refuse silent fallback.
# Stage on a connected host first, then ship with the site image:
#   /usr/local/bin/k3s  and  ./install.sh
#   k3s-airgap-images-amd64.tar.zst under the agent images directory:
#     /var/lib/rancher/k3s/agent/images/   (required — not beside install.sh)
#   INSTALL_K3S_SKIP_DOWNLOAD=true \
#     INSTALL_K3S_EXEC='server --embedded-registry --disable-default-registry-endpoint' \
#     ./install.sh

Second, K3s ships an embedded distributed registry mirror. Enable it with --embedded-registry and every node hosts a local OCI registry on port 6443 and advertises images already in its containerd store over a peer-to-peer network on port 5001. On a multi-node site, an image staged onto one node becomes pullable by the others with no external registry. Combined with placing the air-gap image tarball under /var/lib/rancher/k3s/agent/images/ (so K3s imports it on start) and pre-loading application images into containerd at provisioning time, a site can bootstrap and run entirely from bytes that arrived before it needed the network. The WAN then only carries new image references; layers move on your schedule when a site can afford them. For the stateless, request-response slice of the mix, a 1–2 MB Wasm runtime footprint changes what is even deployable on the most bandwidth-starved hardware.

Per-Site Overlays Without a Repo per Site

Heterogeneity is where edge config either stays sane or degenerates into a directory per location. Fleet's answer is targetCustomizations inside fleet.yaml: one Bundle definition carries per-target overrides that are evaluated in order, first match wins, selected by cluster labels. Region, hardware class, regulatory zone, and site-specific values become dimensions expressed as labels on the cluster, and the repository holds one baseline plus a handful of overlays — not a thousand forks. A Kustomize overlay path or a block of Helm values is chosen by the labels a site already carries, so the config surface grows with the number of *kinds* of site, not the number of sites.

defaultNamespace: edge-workloads
helm:
  chart: ./chart
  values:
    replicas: 1
    telemetry:
      sampleInterval: 60s

# Evaluated top-down, first match wins. Labels live on the cluster object.
targetCustomizations:
  - name: eu-regulated
    clusterSelector:
      matchLabels:
        zone: eu
    helm:
      values:
        dataResidency: eu-only        # keep telemetry in-region
        telemetry: { sampleInterval: 30s }
  - name: low-bandwidth-sat
    clusterSelector:
      matchLabels:
        link: satellite
    helm:
      values:
        telemetry: { sampleInterval: 300s }  # spare the metered link
  - name: dual-node-sites
    clusterSelector:
      matchLabels:
        topology: ha-pair
    helm:
      values: { replicas: 2 }

The discipline that keeps this honest is to treat labels as the contract and forbid site-specific manifests outside the overlay mechanism. The moment someone hand-edits one site's resources out of band, they have created a snowflake that drift correction will fight and no reviewer can see in Git. Every meaningful difference between two sites should be legible as a difference in their labels — which makes the fleet auditable, because the reason a site behaves as it does is a declarative fact in the repository, not tribal knowledge.

Reconnection Reconciliation: Connectivity Is an Event, Not a Precondition

The defining metric of a far-edge fleet is not uptime — it is the staleness window: the time between a commit to desired state and a given site actually applying it. Because sites go dark, that window is a distribution, not a number, and managing the fleet means managing its tail. When a site is offline, K3s keeps running the last state it successfully applied. When the link returns, the fleet-agent re-establishes outward, pulls Bundle revisions accrued while it was gone, diffs desired against live, applies the delta, and reports status — whether the outage was short or planned.

Click to zoom

Two Fleet behaviors matter on reconnection. Drift correction, configured on the GitRepo, means any change made to a managed resource while the site was offline — a local hotfix, a manual kubectl edit, a half-finished intervention — is reverted to the declared state on the next reconcile, so Git remains the single truth even for machines tampered with in the field. And rollout gating still governs the estate: a returning site that reports NotReady surfaces that status and counts toward its partition's unavailability, but further partitions and batches pause only once Unavailable exceeds maxUnavailable (or NotReady partitions exceed maxUnavailablePartitions). With Fleet's default global maxUnavailable of 100%, a lone NotReady site does not halt the estate; the canary pattern above — maxUnavailable: 0 on the canary partition and maxUnavailablePartitions: 0 — is what makes a single failure actually block later waves. Holding a site at its prior good state on a failed upgrade is opt-in — set helm.atomic: true in fleet.yaml so a failed Helm upgrade is rolled back rather than left half-applied.

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: edge-baseline
  namespace: fleet-default
spec:
  repo: https://git.internal/edge/fleet-config
  correctDrift:
    enabled: true          # revert out-of-band field edits on reconcile
    force: false           # non-destructive where possible
  # During a known-bad window you pause the repo rather than fight it;
  # sites keep serving their last-applied state until you resume.
  paused: false

When You Need Device-Level Autonomy: KubeEdge as a Complement

Fleet plus K3s manages clusters. Some edge problems are really device problems — thousands of sensors and actuators behind a gateway, talking Modbus or MQTT, where the unit of autonomy is finer than a cluster. That is KubeEdge's territory. It splits into CloudCore, running beside your Kubernetes API server in a connected location, and EdgeCore, running on edge nodes, caching state in a local store (SQLite), and mediating devices through digital twins. Its edge-autonomy guarantee is explicit: when the cloud-edge link is unstable or the node reboots while offline, EdgeCore keeps already-scheduled workloads running from cached metadata, with no dependency on CloudCore being reachable.

The two are different altitudes, not rivals. Use K3s and Fleet when the site is a small cluster running general workloads and the problem is configuration and rollout across many locations. Reach for KubeEdge when the site is a constellation of devices needing twin-based modeling and message routing — the device tier has its own contract to read precisely. They compose: a Fleet-managed K3s cluster at a regional aggregation point can host CloudCore for the devices below it.

Security and Recovery at Sites You Cannot Reach

A machine sitting unattended in a physically exposed location is a threat model of its own. The node OS should be immutable and minimal, with no package manager, SSH shell, or writable system partition for an attacker with physical access to abuse — the case for an immutable Kubernetes OS like Talos is even stronger at the edge than in the datacenter, which at least has a locked door. Images must be signed and verified at admission, because an air-gapped mirror is still a mirror someone could poison. Secrets should be short-lived and scoped per site, never a fleet-wide credential whose theft from one stolen device compromises every other.

Recovery has to assume no human. When a disk fails or a node is replaced, provisioning must be zero-touch: the new node boots, joins with a bootstrap token, and the fleet-agent rebuilds the site from BundleDeployments on the management cluster plus the locally staged images, with no interactive step because there is no one to perform it. The disaster-recovery discipline of proving RPO and RTO with drills applies with a harder constraint: stateful edge data must be captured to storage you control and be restorable without a technician on site — the recovery you cannot automate is one that will not happen when a site is a day's travel away.

The Exit Ramp: A Fleet You Don't Get Locked Into

Optionality is easy to lose at the edge: a proprietary edge-management SaaS makes onboarding a thousand sites feel effortless — until pricing changes or the product is discontinued and your fleet is hostage to a control plane you do not own. The stack here is built from substitutable, open parts. The source of truth is a Git repository of plain manifests and Helm charts, portable to any GitOps engine. The runtime is upstream K3s, a CNCF-conformant distribution with no vendor dependency. Fleet is open source and speaks ordinary Kubernetes resources; if it stopped fitting, the same repository could be reconciled by Argo CD or Flux. Images live in your own OCI registries under your own PKI. Nothing in the critical path requires a specific vendor to stay in business or a specific cloud to stay reachable.

That substitutability is the exit ramp made concrete: swap the GitOps controller without touching the workloads, swap K3s for another conformant distribution without rewriting manifests, move registries without changing application code. Anti-lock-in at fleet scale lets you renegotiate, replace a failing component, or absorb an acquisition without a forced migration of hardware in the field.

The Long Game: Operating a Fleet a Decade From Now

Edge hardware outlives the software fashions current when it shipped. A turbine, a ship, a factory line is a fifteen-to-twenty-year asset, and the approach has to be legible to an engineer who joins long after the builders have moved on. Pull-based GitOps ages well because the desired state is a readable repository, not the accumulated side effects of a decade of manual changes — the same reason cloud repatriation onto owned infrastructure is a durable position rather than a fashion. The fleet's history is git log; its intent is the tree; its behavior at any site is derivable from labels and overlays.

The remaining long-game requirement is visibility, and most edge fleets neglect it until an outage. You cannot manage a staleness distribution you cannot see, so the fleet needs self-hosted observability that aggregates each site's last-check-in time, Bundle status, and reconciliation lag — with alerting keyed to the tail, the sites that have not reported in too long, not the healthy median. Declare intent in Git, let each site pull and converge, and know which sites have fallen behind and why. Build that, and a fleet you cannot physically reach becomes one you fully own.