Managing a Disconnected Fleet: K3s and Rancher Fleet at the Far Edge

A far-edge fleet breaks every assumption a datacenter platform team takes for granted. The clusters are not in a rack you can walk to; they are on oil platforms, ships, wind turbines, factory floors, and cell towers. The backhaul is cellular, satellite, or a rural line that drops for minutes or days, and there is often no inbound path to the site at all, by policy or by NAT. And there are not thirty clusters; there can be thousands, each a single-node or three-node K3s install nobody will ever log into interactively. The question is not "how do I deploy to Kubernetes" — it is "how does a system I cannot reach, on a network I cannot trust, keep converging to the state I declared, and tell me when it cannot?"

This article is the operational playbook for that fleet: K3s as the edge runtime, Rancher Fleet as a pull-based GitOps control plane built for intermittent connectivity, an air-gapped image-distribution scheme that survives sites with no registry access, per-site overlays that do not collapse into repository sprawl, and a reconnection-reconciliation model that makes an offline site a normal state rather than an incident. Forecasts have put roughly three-quarters of enterprise-generated data at the edge by the end of 2026 — a shift that turns K3s from a homelab curiosity into a production fleet-management problem measured in site-count and staleness, not uptime.

The audience is the people who own that fleet: platform and infrastructure leads standing up edge estates, SREs who carry the pager for sites in three time zones, and the security and compliance functions answering for machines running unattended in exposed locations. Sovereignty is the through-line — a fleet you can only operate while a vendor's cloud console is reachable is not a fleet you own.

Why the Far Edge Is Not Just Multi-Cluster GitOps With More Clusters

It is tempting to treat this as a scaling problem — the same datacenter multi-cluster GitOps you already run, pointed at more endpoints. It is not. Datacenter GitOps assumes clusters are continuously reachable, well-provisioned, and homogeneous, and it can lean on a control plane that reaches out to them. The far edge violates all three assumptions at once, and each violation changes the architecture rather than just the numbers.

• Connectivity is intermittent and often one-way. The control plane can never assume it can reach a site. Any design that pushes — SSH, an agent the server dials into, a reconcile that blocks on the target — stalls the instant a link drops, and links drop constantly.
• Sites are air-gapped or bandwidth-starved. Pulling a 400 MB image over a metered satellite link, times a thousand sites, is not a rollout — it is an outage and a bill. Images must be staged locally; the network carries manifests and digests, not layers.
• Configuration is heterogeneous by location. Site 12 has a different sensor, VLAN, or regulatory zone than site 900. You need per-site variation without a per-site pipeline, or the config surface explodes into thousands of near-duplicate files nobody can audit.
• Nobody is coming. There is no remote-hands SSH for most sites. Provisioning has to be zero-touch, recovery has to be automatic, and the running state has to survive the operator being completely absent for the machine's entire life.

Those four constraints point at one architecture: pull-based reconciliation, local image custody, label-driven overlays, and autonomy on disconnect. Fleet and K3s were built around exactly this shape, which is why the pairing has become the default far-edge stack rather than a stretched datacenter tool.

Pull-Based by Construction: Rancher Fleet at the Edge

Fleet is a GitOps controller designed to manage clusters at a scale — its documentation cites up to a million — that no push model reaches. Its unit of work is the Bundle: a controller in a management cluster watches a GitRepo resource, renders the repository's manifests and Helm charts into Bundles, and each downstream cluster runs a fleet-agent that pulls the Bundles targeted at it, applies them, and reports status back best-effort. The direction of travel is the whole point: the agent initiates every connection outward, so the controller never needs a route into a site, and a cluster unreachable for six hours is not an error — just an agent that has not checked in yet.

This inversion is what makes intermittent WAN survivable. In a push model, the reconcile loop is coupled to the reachability of every target, so one flapping satellite link can back up a queue or trip a timeout that touches unrelated sites. In Fleet's pull model, each site's fleet-agent owns its own reconcile cadence against Git, so the failure domain is exactly one site — and the controller's job shrinks to rendering desired state and aggregating status, both of which it does whether or not any given agent is online.

Click to zoom

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: edge-baseline
  namespace: fleet-default
spec:
  repo: https://git.internal/edge/fleet-config
  branch: main
  # Spare the WAN: poll less often than a datacenter would.
  pollingInterval: 5m
  paths:
    - bundles/baseline      # everything every site runs
  targets:
    # Select clusters by label, not by name — onboarding is a label.
    - clusterSelector:
        matchLabels:
          fleet.cattle.io/tier: far-edge
  # Roll changes out in waves so a bad release cannot hit all sites at once.
  rolloutStrategy:
    maxUnavailable: 5%
    maxUnavailablePartitions: 1

The rolloutStrategy is the safety valve people skip until a bad manifest ships. A wave-based rollout with a small maxUnavailable means a broken Bundle degrades a handful of sites and halts, rather than reaching the entire estate before anyone notices — the fleet equivalent of a canary, and non-negotiable when a bad push could mean physically visiting a thousand locations.

Air-Gapped Image Distribution: The Part That Actually Breaks

Manifests are small; images are not. The single most common far-edge failure is a Bundle that applies cleanly and then sits in ImagePullBackOff because the site cannot reach a registry — and you cannot SSH in to diagnose it. Image distribution is therefore not an optimization; it is the difference between a fleet that self-heals and one that quietly rots. K3s gives you two composable mechanisms. First, a registries.yaml placed in /etc/rancher/k3s/ before install rewrites pulls for upstream registries to a mirror you control, and --disable-default-registry-endpoint stops any fallback to the internet — so a site that loses its link cannot silently try to reach docker.io and hang.

mirrors:
  docker.io:
    endpoint:
      - "https://registry.site.local:5000"
  ghcr.io:
    endpoint:
      - "https://registry.site.local:5000"
  quay.io:
    endpoint:
      - "https://registry.site.local:5000"
configs:
  "registry.site.local:5000":
    tls:
      # Site registry served under your own internal PKI, not a public CA.
      ca_file: /etc/rancher/k3s/site-registry-ca.crt
---
# Install flags that make the air-gap explicit and refuse silent fallback:
#   curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true sh -s - \
#     --embedded-registry \
#     --disable-default-registry-endpoint

Second, K3s ships an embedded distributed registry mirror. Enable it with --embedded-registry and every node hosts a local OCI registry on port 6443 and advertises the images already in its containerd store over a peer-to-peer network on port 5001. On a multi-node site, an image staged onto one node becomes pullable by the others with no external registry at all. Combined with pre-loading k3s-airgap-images.tar and application images into the containerd store at provisioning time, a site can bootstrap and run entirely from bytes that arrived before it ever needed the network. The WAN then only carries new image references; the actual layers move on your schedule, over your bandwidth, when a site can afford them.

Per-Site Overlays Without a Repo per Site

Heterogeneity is where edge config either stays sane or degenerates into a directory per location. Fleet's answer is targetCustomizations inside fleet.yaml: one Bundle definition carries per-target overrides that are evaluated in order, first match wins, selected by cluster labels. Region, hardware class, regulatory zone, and site-specific values become dimensions expressed as labels on the cluster, and the repository holds one baseline plus a handful of overlays — not a thousand forks. A Kustomize overlay path or a block of Helm values is chosen by the labels a site already carries, so the config surface grows with the number of *kinds* of site, not the number of sites.

defaultNamespace: edge-workloads
helm:
  chart: ./chart
  values:
    replicas: 1
    telemetry:
      sampleInterval: 60s

# Evaluated top-down, first match wins. Labels live on the cluster object.
targetCustomizations:
  - name: eu-regulated
    clusterSelector:
      matchLabels:
        zone: eu
    helm:
      values:
        dataResidency: eu-only        # keep telemetry in-region
        telemetry: { sampleInterval: 30s }
  - name: low-bandwidth-sat
    clusterSelector:
      matchLabels:
        link: satellite
    helm:
      values:
        telemetry: { sampleInterval: 300s }  # spare the metered link
  - name: dual-node-sites
    clusterSelector:
      matchLabels:
        topology: ha-pair
    helm:
      values: { replicas: 2 }

The discipline that keeps this honest is to treat labels as the contract and forbid site-specific manifests outside the overlay mechanism. The moment someone hand-edits one site's resources out of band, they have created a snowflake that drift correction will fight and no reviewer can see in Git. Every meaningful difference between two sites should be legible as a difference in their labels — which makes the fleet auditable, because the reason a site behaves as it does is a declarative fact in the repository, not tribal knowledge.

Reconnection Reconciliation: Connectivity Is an Event, Not a Precondition

The defining metric of a far-edge fleet is not uptime — it is the staleness window: the time between a commit to desired state and a given site actually applying it. Because sites go dark, that window is a distribution, not a number, and managing the fleet means managing its tail. When a site is offline, K3s keeps running the last state it successfully applied, indifferent to the controller's existence. When the link returns, the fleet-agent re-establishes outward, pulls whatever Bundle revisions accrued while it was gone, diffs desired against live, applies the delta, and reports converged — regardless of whether the outage was short or planned.

Click to zoom

Two Fleet behaviors matter on reconnection. Drift correction, configured on the GitRepo, means any change made to a managed resource while the site was offline — a local hotfix, a manual kubectl edit, a half-finished intervention — is reverted to the declared state on the next reconcile, so Git remains the single truth even for machines tampered with in the field. And rollout gating still applies to a returning site: a Bundle that would fail health checks is not force-applied the moment the site reappears; the agent holds the prior good state and surfaces the failure instead of bricking a site you cannot reach.

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: edge-baseline
  namespace: fleet-default
spec:
  repo: https://git.internal/edge/fleet-config
  correctDrift:
    enabled: true          # revert out-of-band field edits on reconcile
    force: false           # non-destructive where possible
  # During a known-bad window you pause the repo rather than fight it;
  # sites keep serving their last-applied state until you resume.
  paused: false

When You Need Device-Level Autonomy: KubeEdge as a Complement

Fleet plus K3s manages clusters. Some edge problems are really device problems — thousands of sensors and actuators behind a gateway, talking Modbus or MQTT, where the unit of autonomy is finer than a cluster. That is KubeEdge's territory. It splits into CloudCore, running beside your Kubernetes API server in a connected location, and EdgeCore, running on edge nodes, caching state in a local store (SQLite), and mediating devices through digital twins. Its edge-autonomy guarantee is explicit: when the cloud-edge link is unstable or the node reboots while offline, EdgeCore keeps already-scheduled workloads running from cached metadata, with no dependency on CloudCore being reachable.

The two are different altitudes, not rivals. Use K3s and Fleet when the site is a small cluster running general workloads and the problem is configuration and rollout across many locations. Reach for KubeEdge when the site is a constellation of devices needing twin-based modeling and message routing, where cluster-per-site is the wrong granularity. They compose: a Fleet-managed K3s cluster at a regional aggregation point can host CloudCore for the devices below it, so the same pull-based, Git-declared discipline governs the cluster tier while KubeEdge governs the device tier beneath it.

Security and Recovery at Sites You Cannot Reach

A machine sitting unattended in a physically exposed location is a threat model of its own. The node OS should be immutable and minimal, with no package manager, SSH shell, or writable system partition for an attacker with physical access to abuse — the case for an immutable Kubernetes OS like Talos is even stronger at the edge than in the datacenter, which at least has a locked door. Images must be signed and verified at admission, because an air-gapped mirror is still a mirror someone could poison. Secrets should be short-lived and scoped per site, never a fleet-wide credential whose theft from one stolen device compromises every other.

Recovery has to assume no human. When a disk fails or a node is replaced, provisioning must be zero-touch: the new node boots, joins with a bootstrap token, and the fleet-agent rebuilds the site from Git plus the locally staged images, with no interactive step because there is no one to perform it. The disaster-recovery discipline of proving RPO and RTO with drills applies with a harder constraint: stateful edge data must be captured to storage you control and be restorable without a technician on site — the recovery you cannot automate is one that will not happen when a site is a day's travel away.

The Exit Ramp: A Fleet You Don't Get Locked Into

Optionality is easy to lose at the edge, because a proprietary edge-management SaaS makes onboarding a thousand sites feel effortless — right up until the pricing changes or the product is discontinued and your fleet is hostage to a control plane you do not own. The stack here is built from substitutable, open parts. The source of truth is a Git repository of plain manifests and Helm charts, portable to any GitOps engine. The runtime is upstream K3s, a CNCF-conformant distribution with no vendor dependency. Fleet is open source and speaks ordinary Kubernetes resources; if it stopped fitting, the same repository could be reconciled by Argo CD or Flux, the overlays reshaped rather than rewritten. Images live in your own OCI registries under your own PKI. Nothing in the critical path requires a specific vendor to stay in business or a specific cloud to stay reachable.

That substitutability is the exit ramp made concrete: swap the GitOps controller without touching the workloads, swap K3s for another conformant distribution without rewriting manifests, move registries without changing application code. Anti-lock-in at fleet scale is the property that lets you renegotiate, replace a failing component, or absorb an acquisition without a forced migration of hardware in the field — and the teams that lose it are the ones who bought the convenient bundle and found, a thousand sites later, that convenience was the collar.

The Long Game: Operating a Fleet a Decade From Now

Edge hardware outlives the software fashions current when it shipped. A turbine, a ship, a factory line is a fifteen-to-twenty-year asset, and the fleet-management approach has to be legible to an engineer who joins long after everyone who built it has moved on. Pull-based GitOps ages well precisely because the desired state is a readable repository, not the accumulated side effects of a decade of manual changes — the same reason cloud repatriation onto owned infrastructure is a durable position rather than a fashion. The fleet's history is git log; its intent is the tree; its behavior at any site is derivable from labels and overlays — a system a future team can operate, not just admire.

The remaining long-game requirement is visibility, and it is the piece most edge fleets neglect until an outage. You cannot manage a staleness distribution you cannot see, so the fleet needs self-hosted observability that aggregates each site's last-check-in time, Bundle status, and reconciliation lag into one view — with alerting keyed to the tail of the distribution, the sites that have not reported in too long, not the healthy median. A far-edge fleet is not managed by watching a thousand dashboards. It is managed by declaring intent in Git, letting each site pull and converge, and being told exactly which sites have fallen behind and why. Build that, and a fleet you cannot physically reach becomes one you fully own.