Kubernetes Disaster Recovery: Velero, etcd, RPO/RTO

Most Kubernetes "backup strategies" are an unfalsifiable claim. A Velero server is installed, a schedule exists, the dashboard is green, and everyone assumes the cluster can be recovered. Then a region goes dark — storage fails, a control-plane upgrade corrupts etcd, a misapplied GitOps change cascades, or a provider has a bad day — and the first real restore happens under maximum pressure. The question an auditor, board, or regulator asks is not "do you take backups?" It is "when did you last restore from one, how long did it take, and how much data did you lose?" If you cannot answer with a date and two numbers, you do not have disaster recovery. You have hope with a cron schedule.

This article lays out audit-grade Kubernetes DR: the three things you must back up (API resources, PV data, etcd), how to back each up with 2026 tooling, how to restore into a standby cluster in another region, and how to define RPO/RTO per workload and prove them with scheduled restore drills. Sovereignty runs through all of it: backups in your object store, under your keys, in your jurisdiction. A backup you cannot read without your provider is not a backup — it is a hostage.

The audience is who owns the consequences: CTOs and platform leads who sign the architecture, SREs who run the drills, and risk/compliance functions who must show evidence under NIS2 and DORA business-continuity obligations. DORA treats DR testing as a named, recurring control — not a one-time design review.

Three Things to Back Up — and Why Conflating Them Loses Clusters

Kubernetes disaster recovery fails most often because teams treat "the cluster" as one backup target. It is three, with three different failure modes, restore paths, and RPO characteristics. Conflating them produces a backup that restores some layers and silently drops others — the worst outcome, because it looks like success until someone checks the data.

1. Cluster and application resources — the desired state: namespaces, Deployments, StatefulSets, Services, CRDs, RBAC, ConfigMaps, and the operator objects that define what runs. This is what Velero captures from the Kubernetes API. If your manifests live in Git under GitOps, much of this is reproducible from source — but not the runtime-generated objects, admission-mutated fields, or out-of-band resources, which is why a Velero capture still matters.
2. Persistent-volume data — the bytes inside PVCs: database files, object data, message queues, uploaded artifacts. This is the part that is genuinely irreplaceable. It is backed up via CSI volume snapshots, and — critically — moved off the cluster's storage into an independent object store you operate so a storage-array failure does not take the backups with it. The same logic extends beyond the cluster: a self-hosted mesh coordinator's device registry and private keys are state that cannot be re-derived either, and losing them means re-enrolling every device by hand.
3. Control-plane state in etcd — every API object the cluster has ever been told about, including secrets, lease state, and resource versions. etcd is the cluster's source of truth. Lose etcd with no snapshot and you have lost the cluster's identity even if every node and volume survives.

Velero in 2026: CSI Snapshots, the Data Mover, and Schedules

Velero is the de facto standard for backing up Kubernetes resources and persistent volumes, and its 2026 architecture is materially better than the file-system-backup era many teams remember. Two changes matter. First, since Velero 1.14 the CSI snapshot capability is integrated into the core — you no longer install a separate CSI plugin. Second, the built-in data mover (using the Kopia uploader and a node-agent DaemonSet) takes a CSI VolumeSnapshot, reads the snapshot's data, and writes it to your object store — then removes that temporary CSI snapshot. This is the key sovereignty property: the durable backup is portable, deduplicated, encrypted data in a bucket you control, restorable into a different cluster on different storage in a different region.

The distinction is worth being precise about, because it is where RPO and portability are won or lost. A bare CSI snapshot is fast and cheap but lives in the same storage system as the volume — useful for quick rollback, useless if the array or region is gone. CSI snapshot data movement copies that snapshot's contents out to object storage. With Velero's --snapshot-move-data path the durable recovery point is the object-store copy; the CSI snapshot is only a temporary read source and is removed after upload. Velero orchestrates the snapshot, then the data mover launches a pod to transfer the data and marks a DataUpload resource Completed or Failed — a status you should alert on, because a Failed upload is the silent gap that surfaces only during a restore.

# Install Velero pointed at a self-hosted S3-compatible store (MinIO / Ceph RGW).
# The node-agent DaemonSet is required for the built-in data mover.
# Pin plugin to the Velero server major you install (matrix: v1.14.x↔Velero v1.18.x,
# v1.13.x↔v1.17.x, v1.12.x↔v1.16.x). Example assumes Velero v1.18.x.
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.0 \
  --bucket velero-prod \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --features=EnableCSI \
  --backup-location-config \
    region=minio,s3ForcePathStyle=true,s3Url=https://objstore.internal:9000 \
  --snapshot-location-config region=minio

# A backup that moves CSI snapshot data off-cluster into the bucket.
# --snapshot-move-data is what makes the backup portable and array-independent.
velero backup create app-now \
  --include-namespaces payments,ledger \
  --snapshot-move-data \
  --ttl 168h0m0s

Schedules are how you turn one-off backups into an RPO guarantee. A Velero Schedule is a cron expression that produces backups named <schedule>-<timestamp>, each with a TTL (default 30 days) after which Velero garbage-collects both the backup and its data. The schedule interval is, definitionally, the worst-case RPO for everything that backup covers: an hourly schedule means up to 59 minutes of changes are unprotected at the moment of failure. Choose the interval to meet the tier (covered below), and do not set it so aggressively that backups overlap and pile up — a backup that has not finished when the next one starts is its own kind of outage.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: hourly-payments
  namespace: velero
spec:
  # Top of every hour — worst-case RPO for this set is 60 minutes.
  schedule: "0 * * * *"
  useOwnerReferencesInBackup: false
  template:
    includedNamespaces:
      - payments
      - ledger
    snapshotMoveData: true        # move CSI snapshot data to the object store
    defaultVolumesToFsBackup: false
    ttl: 168h0m0s                 # retain 7 days, then GC backup + data
    hooks:
      resources:
        - name: checkpoint-postgres
          includedNamespaces: [payments]
          labelSelector:
            matchLabels: { app: postgres }
          pre:
            - exec:
                container: postgres
                command:
                  - /bin/sh
                  - -c
                  - "psql -c 'CHECKPOINT;'"  # flush buffers before snapshot
                onError: Fail

A pre hook that runs CHECKPOINT flushes dirty buffers before the volume snapshot, which can shorten crash-recovery after restore; it does not freeze writes, so the snapshot remains crash-consistent (PostgreSQL replays WAL). True application-consistent backup still needs a real quiesce/release pair or, better, the database's native backup path. For PostgreSQL specifically, running it under CloudNativePG moves this problem into the operator, which already treats base backups and continuous WAL archiving as first-class lifecycle operations rather than something you bolt onto a backup hook.

Click to zoom

etcd Snapshots: The Backup Velero Does Not Take for You

Velero backs up the API objects it can read through the Kubernetes API. It does not back up etcd as a datastore, and the two are not interchangeable. etcd holds the raw, consistent, point-in-time state of the entire cluster — including resourceVersions, lease and event state, and secrets in their stored form. For certain disasters, an etcd snapshot restore is the fastest and most faithful recovery: it brings back the cluster exactly as it was, in one operation, rather than replaying a resource-by-resource Velero restore. You want both backups for different scenarios — Velero for portable, selective, cross-version recovery; etcd snapshots for whole-cluster, same-topology recovery and for recovering from control-plane corruption that never made it into a Velero backup.

The mechanics depend on how your control plane runs etcd. On a kubeadm-style cluster you talk to etcd directly; on RKE2 and on Talos the datastore is managed for you and you use the platform's tooling. The commands below cover the three patterns you are most likely to meet in 2026.

# 1) kubeadm / self-managed etcd — snapshot save against a live member.
#    A saved snapshot carries an integrity hash that restore verifies.
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db

# 2) RKE2 — the platform manages etcd; use its snapshot tooling.
rke2 etcd-snapshot save --name etcd-$(date +%Y%m%d-%H%M%S)

# 3) Talos Linux — etcd sits behind the Talos API; talosctl handles it.
talosctl --nodes 10.0.1.10 etcd snapshot /backup/talos-etcd.db

# Upload off-node into the SAME sovereign object store as Velero.
# Paths differ per model — a single etcd-*.db glob only covers kubeadm.
# kubeadm: s3 cp cannot take multi-file globs; sync the timestamped set.
aws --endpoint-url https://objstore.internal:9000 \
  s3 sync /backup s3://etcd-snapshots/$(hostname)/ --exclude '*' --include 'etcd-*.db'
# RKE2: enable --etcd-s3* on the server, or sync the platform snapshot dir:
# aws --endpoint-url https://objstore.internal:9000 \
#   s3 sync /var/lib/rancher/rke2/server/db/snapshots s3://etcd-snapshots/$(hostname)/
# Talos: copy the path you just wrote (name does not match etcd-*.db).
# aws --endpoint-url https://objstore.internal:9000 \
#   s3 cp /backup/talos-etcd.db s3://etcd-snapshots/$(hostname)/

Restoring etcd is an offline, deliberate operation — never something you run casually against a live cluster. The snapshot is restored to a fresh data directory with etcdutl snapshot restore (the restore subcommand moved from etcdctl to the etcdutl utility in current etcd), the control plane is stopped, pointed at the restored data directory, and brought back up. On RKE2 you restore by stopping the service and running rke2 server --cluster-reset --cluster-reset-restore-path=<snapshot> on one server node (add --etcd-s3=false for a local file when S3 config is present), then rejoining the other servers — that flow stops the cluster for you; plan for the downtime explicitly. On Talos, restore is driven through the bootstrap API against the recovery snapshot. The common rule across all three: an etcd restore reverts the cluster to the snapshot's moment in time, so anything created after that snapshot is gone — which is exactly why the etcd snapshot interval is its own RPO, and why fifteen minutes is a common target for control-plane state.

# Restore the snapshot into a NEW data directory (does not touch the live one).
etcdutl snapshot restore /backup/etcd-20260624-031500.db \
  --data-dir /var/lib/etcd-restored

# Then: stop kube-apiserver + etcd, swap the data dir to /var/lib/etcd-restored,
# restart the static pods, and verify the API server serves the restored state.
# For a snapshot copied from a data dir (no hash), add --skip-hash-check.

CSI Volume Snapshots: Configuring the Layer That Holds Your Data

The volume layer is where DR design most often quietly breaks, because it depends on a chain of components — the CSI driver, the snapshot controller, a VolumeSnapshotClass, and Velero's plugin — all agreeing. Get the VolumeSnapshotClass wrong and Velero either skips your volumes or uses the wrong class, and you discover it at restore time. Two settings carry most of the weight: deletionPolicy and the Velero default-class label.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-prod-snapclass
  labels:
    # Velero uses the class carrying this label as the default for its driver.
    # Set it on exactly ONE class per CSI driver, or Velero cannot choose.
    velero.io/csi-volumesnapshot-class: "true"
driver: csi.example.storage.io
# Retain protects snapshots *you* create: kubectl-delete of the VolumeSnapshot leaves the
# storage snapshot. Velero still reaps *its* VolumeSnapshots with the backup (even under
# Retain) and, with data movement, discards the CSI snapshot as soon as upload completes.
deletionPolicy: Retain
parameters:
  # Driver-specific: e.g. incremental snapshots, snapshot location, etc.
  csi.storage.k8s.io/snapshotter-secret-name: csi-snap-creds
  csi.storage.k8s.io/snapshotter-secret-namespace: kube-system

deletionPolicy: Retain protects operator-owned snapshots: deleting a Kubernetes VolumeSnapshot you manage leaves the underlying storage snapshot, so a stray kubectl delete or namespace teardown cannot vaporize that recovery point. The trade-off is that you own lifecycle and must reap orphans deliberately. With --snapshot-move-data / snapshotMoveData: true, Velero uses a CSI snapshot only as a temporary consistent read source, then deletes it when the object-store upload finishes — so the durable recovery point is the data in the bucket, not an on-array snapshot. If you also need seconds-fast local rollback, schedule a separate storage-native or plain VolumeSnapshot path you own; do not expect a Velero data-mover backup to leave both.

Cross-Region Restore: Standing Up a Cluster That Was Never Yours

The restore that matters is the one to a cluster that did not exist when the disaster started: region A is gone, and you are bringing service up in region B from nothing but your object store. This is where the sovereignty of your backup store pays off directly — because all three backup streams live in a bucket you control, replicated off-site, you can install a fresh Velero in the standby cluster, point it at the same bucket, and restore. Nothing in the recovery path depends on region A being reachable or on a provider console you may not be able to log into.

The standby topology is a deliberate cost decision. A warm standby keeps a minimal cluster running in region B with Velero installed and the object store replicated, so recovery is a restore operation measured in minutes. A cold/rebuild approach provisions the standby cluster from GitOps and infrastructure-as-code only when disaster strikes, trading a longer RTO for near-zero standing cost. The right choice is per-tier, not per-organization.

# In the region-B cluster: install Velero pointed at the SAME object store
# (now reading the replicated bucket). Backups appear automatically.
velero backup-location create default \
  --provider aws --bucket velero-prod \
  --config region=minio,s3ForcePathStyle=true,s3Url=https://objstore-b.internal:9000 \
  --access-mode ReadOnly

# Confirm the standby cluster can see the backups taken in region A.
velero backup get

# Restore a specific backup. CSI data-mover rehydrates the volumes from
# object storage onto region-B's storage class.
velero restore create payments-dr \
  --from-backup hourly-payments-20260624-030000 \
  --include-namespaces payments,ledger \
  --restore-volumes=true

# Watch the restore and, crucially, the volume DataDownload completion.
velero restore describe payments-dr --details

A few cross-region restore realities catch teams out. StorageClass names and CSI drivers may differ between regions — Velero supports a StorageClass mapping config so PVCs bind to the right class in region B. LoadBalancer Services will provision new external addresses, so DNS or a global load balancer must be re-pointed as an explicit cutover step, not an afterthought. And volume rehydration is bandwidth-bound: moving terabytes of database state from object storage back onto block volumes is usually the single largest component of RTO, which is why the restore drill (next section) must measure it on representative data rather than on an empty test namespace.

Click to zoom

Defining RPO and RTO Per Workload — and the Trade-Off Nobody Names

RPO and RTO are the two numbers that make DR a contract instead of an aspiration. RPO — Recovery Point Objective — is the maximum data loss you accept, measured in time: an RPO of 15 minutes means a disaster may cost you up to the last 15 minutes of changes. It is governed by backup frequency and replication lag. RTO — Recovery Time Objective — is the maximum time to restore service: an RTO of one hour means you commit to being back within an hour of the disaster. It is governed by your restore mechanics, standby topology, and data volume. They are independent: you can have a tiny RPO (snapshot every minute) and a large RTO (slow restore), or vice versa, and most real systems are unbalanced in exactly that way.

The discipline is to set these per workload, not per cluster. A payment-authorization ledger and an internal Grafana instance do not deserve the same RPO, and pretending they do means you either overspend protecting dashboards or underspend protecting money. Tier your workloads explicitly:

Click to zoom

Here is the trade-off nobody names cleanly: synchronous replication buys near-zero RPO by coupling regions, and that coupling caps write throughput at inter-region latency. In a Tier-0 active-active design every committed write must round-trip to the second region before acknowledgment — 20 ms apart means 20 ms on every write path, bounded by distance rather than storage hardware. Asynchronous replication removes that penalty but reintroduces RPO equal to replication lag (seconds to low minutes). You cannot have zero RPO, zero latency cost, and regional independence at once; you always choose two of three. Naming that trade-off per workload is the job — the teams that get DR wrong discover which two they had only after an incident.

Restore Drills: Turning a Backup Into Evidence

A backup is a hypothesis. A restore is the experiment that tests it. Until you have restored — fully, into a real cluster, with the data validated — you do not know your RPO or RTO; you know the values you hoped for. The single highest-leverage thing most platform teams can do for resilience is to schedule restore drills as a recurring, calendared operation and treat a failed or slow drill as an incident with a postmortem, exactly like a production outage. This is not gold-plating. Under DORA's resilience-testing pillar, periodic testing of ICT continuity and recovery is a named obligation, and "we restore on a schedule and here are the timed results" is precisely the evidence an examiner expects. It is also precisely what SOC 2's Availability criterion (A1) asks for — a tested restore, not a documented plan.

A real drill restores into an isolated or ephemeral standby cluster, measures RTO from "disaster declared" to "verification passed," computes RPO from backup timestamp versus simulated failure, and — non-negotiably — validates data integrity rather than pod readiness alone. A Running pod against an empty or corrupt volume is a failed restore that looks successful. Bake integrity into the drill: row counts, checksums, known-record lookup, smoke test. Write the numbers with the date. That record is your DR program.

#!/usr/bin/env bash
set -euo pipefail

DRILL_START=$(date +%s)
BACKUP=$(velero backup get -l velero.io/schedule-name=hourly-payments -o json | jq -r 'if .items then (.items | max_by(.metadata.creationTimestamp).metadata.name) else .metadata.name end')
echo "Drill $(date -u +%FT%TZ): restoring ${BACKUP} into standby"

# 1) Restore into the isolated drill cluster/namespace.
velero restore create drill-$(date +%s) \
  --from-backup "${BACKUP}" \
  --namespace-mappings payments:payments-drill \
  --restore-volumes=true --wait

# 2) Validate DATA, not just pod status. Fail the drill on mismatch.
EXPECTED=$(cat /drill/expected-ledger-rows)
ACTUAL=$(kubectl -n payments-drill exec deploy/postgres -- \
  psql -tA -c 'SELECT count(*) FROM ledger;')
[ "${ACTUAL}" = "${EXPECTED}" ] || { echo "INTEGRITY FAIL: ${ACTUAL}!=${EXPECTED}"; exit 1; }

# 3) Record the measured RTO. This number — dated — is the audit evidence.
RTO=$(( $(date +%s) - DRILL_START ))
echo "DRILL PASS  rto_seconds=${RTO}  backup=${BACKUP}  at=$(date -u +%FT%TZ)" \
  | tee -a /drill/dr-evidence.log

Continuous validation closes the loop. Backups silently rot — a credential rotates and the data mover fails, a VolumeSnapshotClass label is removed, a storage migration changes the driver name, or object-store GC reaps a backup you assumed was retained. Alert on Velero backup phase, DataUpload/DataDownload completion, etcd-snapshot CronJob success, and replication lag between drills. Teams running KubeVigil and their self-hosted observability stack fold these into the same dashboards and routes, so a broken backup pages the day it breaks — not the day they need it.

Sovereign Backups and the Long Game

Where backups live is a sovereignty decision with the same weight as where workloads run. A managed backup service you cannot read without the vendor's console — under their keys, in their jurisdiction — is a recovery path that depends on a third party at your worst moment. The sovereign pattern is the opposite: your S3-compatible store (MinIO, Ceph RGW, or on-prem), keys you hold, object-lock/WORM and versioning so a compromised credential cannot erase recovery points, replicated to a second site you also control. The architecture that makes you independent of a single hyperscaler for compute makes you independent for recovery — when independence matters most.

This is where the long game meets audit-grade rigor. Immutable, versioned, dated restore evidence is the artifact that satisfies an auditor, survives leadership change, and still makes sense years and platform migrations later. Procedures last only if exercised, measured, and documented continuously — not designed once and trusted forever. An unrun DR plan is silent technical debt; a plan on a schedule with timed evidence in your own store is one of the few investments whose value grows with system age. The proactive counterpart is structured fault injection as audit evidence — a restore drill proves recovery; a chaos experiment proves survival.