Audit-Grade Kubernetes Disaster Recovery: Velero, etcd Snapshots, and Meeting RPO/RTO You Can Prove

Most Kubernetes "backup strategies" are an unfalsifiable claim. A Velero server is installed, a schedule exists, the dashboard is green, and everyone assumes the cluster can be recovered. Then a region goes dark — a storage array fails, a control-plane upgrade corrupts etcd, a misapplied GitOps change cascades, or a provider has a bad day — and the first real restore in the platform's history happens under maximum pressure, against an audience of executives. The question an auditor, a board, or a regulator actually asks is not "do you take backups?" It is "when did you last restore from one, how long did it take, and how much data did you lose?" If you cannot answer that with a date and two numbers, you do not have disaster recovery. You have hope with a cron schedule.

This article lays out an audit-grade DR architecture for Kubernetes: the three distinct things you must back up (cluster and application resources, persistent-volume data, and control-plane state in etcd), how to back each up correctly with current tooling in 2026, how to restore into a standby cluster in a different region, and — the part most teams skip — how to define Recovery Point Objective (RPO) and Recovery Time Objective (RTO) per workload and then prove you meet them with scheduled restore drills. The sovereignty thread runs through all of it: your backups belong in your object store, under your encryption keys, in your jurisdiction. A backup you cannot read without your provider's cooperation is not a backup — it is a hostage.

The audience here is the people who own the consequences: CTOs and platform leads who sign off on the architecture, SREs who run the drills, and the risk and compliance functions who have to show evidence under NIS2 and DORA business-continuity obligations. DORA in particular treats DR testing as a named, recurring control — not a one-time design review.

Three Things to Back Up — and Why Conflating Them Loses Clusters

Kubernetes disaster recovery fails most often because teams treat "the cluster" as one backup target. It is three, with three different failure modes, restore paths, and RPO characteristics. Conflating them produces a backup that restores some layers and silently drops others — the worst outcome, because it looks like success until someone checks the data.

1. Cluster and application resources — the desired state: namespaces, Deployments, StatefulSets, Services, CRDs, RBAC, ConfigMaps, and the operator objects that define what runs. This is what Velero captures from the Kubernetes API. If your manifests live in Git under GitOps, much of this is reproducible from source — but not the runtime-generated objects, admission-mutated fields, or out-of-band resources, which is why a Velero capture still matters.
2. Persistent-volume data — the bytes inside PVCs: database files, object data, message queues, uploaded artifacts. This is the part that is genuinely irreplaceable. It is backed up via CSI volume snapshots, and — critically — moved off the cluster's storage into an independent object store you operate so a storage-array failure does not take the backups with it.
3. Control-plane state in etcd — every API object the cluster has ever been told about, including secrets, lease state, and resource versions. etcd is the cluster's source of truth. Lose etcd with no snapshot and you have lost the cluster's identity even if every node and volume survives.

Velero in 2026: CSI Snapshots, the Data Mover, and Schedules

Velero is the de facto standard for backing up Kubernetes resources and persistent volumes, and its 2026 architecture is materially better than the file-system-backup era many teams remember. Two changes matter. First, since Velero 1.14 the CSI snapshot capability is integrated into the core — you no longer install a separate CSI plugin. Second, the built-in data mover (using the Kopia uploader and a node-agent DaemonSet) takes a CSI VolumeSnapshot, reads the snapshot's data, and writes it to your object store. This is the key sovereignty property: the backup no longer lives only as a storage-provider snapshot tied to one array or one cloud — it is portable, deduplicated, encrypted data sitting in a bucket you control, restorable into a different cluster on different storage in a different region.

The distinction is worth being precise about, because it is where RPO and portability are won or lost. A bare CSI snapshot is fast and cheap but lives in the same storage system as the volume — useful for quick rollback, useless if the array or region is gone. CSI snapshot data movement copies that snapshot's contents out to object storage. You want both: snapshots for speed, data movement for durability and portability. Velero orchestrates the snapshot, then the data mover launches a pod to transfer the data and marks a DataUpload resource Completed or Failed — a status you should alert on, because a Failed upload is the silent gap that surfaces only during a restore.

# Install Velero pointed at a self-hosted S3-compatible store (MinIO / Ceph RGW).
# The node-agent DaemonSet is required for the built-in data mover.
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.12.0 \
  --bucket velero-prod \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --features=EnableCSI \
  --backup-location-config \
    region=minio,s3ForcePathStyle=true,s3Url=https://objstore.internal:9000 \
  --snapshot-location-config region=minio

# A backup that moves CSI snapshot data off-cluster into the bucket.
# --snapshot-move-data is what makes the backup portable and array-independent.
velero backup create app-now \
  --include-namespaces payments,ledger \
  --snapshot-move-data \
  --ttl 168h0m0s

Schedules are how you turn one-off backups into an RPO guarantee. A Velero Schedule is a cron expression that produces backups named <schedule>-<timestamp>, each with a TTL (default 30 days) after which Velero garbage-collects both the backup and its data. The schedule interval is, definitionally, the worst-case RPO for everything that backup covers: an hourly schedule means up to 59 minutes of changes are unprotected at the moment of failure. Choose the interval to meet the tier (covered below), and do not set it so aggressively that backups overlap and pile up — a backup that has not finished when the next one starts is its own kind of outage.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: hourly-payments
  namespace: velero
spec:
  # Top of every hour — worst-case RPO for this set is 60 minutes.
  schedule: "0 * * * *"
  useOwnerReferencesInBackup: false
  template:
    includedNamespaces:
      - payments
      - ledger
    snapshotMoveData: true        # move CSI snapshot data to the object store
    defaultVolumesToFsBackup: false
    ttl: 168h0m0s                 # retain 7 days, then GC backup + data
    hooks:
      resources:
        - name: quiesce-postgres
          includedNamespaces: [payments]
          labelSelector:
            matchLabels: { app: postgres }
          pre:
            - exec:
                container: postgres
                command:
                  - /bin/sh
                  - -c
                  - "psql -c 'CHECKPOINT;'"  # flush buffers before snapshot
                onError: Fail

The backup hooks are not optional for stateful workloads. A volume snapshot is crash-consistent — equivalent to pulling the power cord — which most databases survive but none guarantee. A pre hook that issues a checkpoint or a quiesce, paired with a post hook that resumes, raises the backup from crash-consistent to application-consistent. For databases with a native backup tool, the stronger pattern is to run that tool's dump in the pre-hook and snapshot the result, so you are capturing a logically consistent export rather than a moment-in-time block image.

Click to zoom

etcd Snapshots: The Backup Velero Does Not Take for You

Velero backs up the API objects it can read through the Kubernetes API. It does not back up etcd as a datastore, and the two are not interchangeable. etcd holds the raw, consistent, point-in-time state of the entire cluster — including resourceVersions, lease and event state, and secrets in their stored form. For certain disasters, an etcd snapshot restore is the fastest and most faithful recovery: it brings back the cluster exactly as it was, in one operation, rather than replaying a resource-by-resource Velero restore. You want both backups for different scenarios — Velero for portable, selective, cross-version recovery; etcd snapshots for whole-cluster, same-topology recovery and for recovering from control-plane corruption that never made it into a Velero backup.

The mechanics depend on how your control plane runs etcd. On a kubeadm-style cluster you talk to etcd directly; on RKE2 and on Talos the datastore is managed for you and you use the platform's tooling. The commands below cover the three patterns you are most likely to meet in 2026.

# 1) kubeadm / self-managed etcd — snapshot save against a live member.
#    A saved snapshot carries an integrity hash that restore verifies.
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db

# 2) RKE2 — the platform manages etcd; use its snapshot tooling.
rke2 etcd-snapshot save --name etcd-$(date +%Y%m%d-%H%M%S)

# 3) Talos Linux — etcd sits behind the Talos API; talosctl handles it.
talosctl --nodes 10.0.1.10 etcd snapshot /backup/talos-etcd.db

# Whichever you use: copy the .db off the control-plane node immediately
# into the SAME sovereign object store as your Velero data.
aws --endpoint-url https://objstore.internal:9000 \
  s3 cp /backup/etcd-*.db s3://etcd-snapshots/$(hostname)/

Restoring etcd is an offline, deliberate operation — never something you run casually against a live cluster. The snapshot is restored to a fresh data directory with etcdutl snapshot restore (the restore subcommand moved from etcdctl to the etcdutl utility in current etcd), the control plane is stopped, pointed at the restored data directory, and brought back up. On RKE2 the rke2 etcd-snapshot restore flow stops the cluster for you; plan for that downtime explicitly. On Talos, restore is driven through the bootstrap API against the recovery snapshot. The common rule across all three: an etcd restore reverts the cluster to the snapshot's moment in time, so anything created after that snapshot is gone — which is exactly why the etcd snapshot interval is its own RPO, and why fifteen minutes is a common target for control-plane state.

# Restore the snapshot into a NEW data directory (does not touch the live one).
etcdutl snapshot restore /backup/etcd-20260624-031500.db \
  --data-dir /var/lib/etcd-restored

# Then: stop kube-apiserver + etcd, swap the data dir to /var/lib/etcd-restored,
# restart the static pods, and verify the API server serves the restored state.
# For a snapshot copied from a data dir (no hash), add --skip-hash-check.

CSI Volume Snapshots: Configuring the Layer That Holds Your Data

The volume layer is where DR design most often quietly breaks, because it depends on a chain of components — the CSI driver, the snapshot controller, a VolumeSnapshotClass, and Velero's plugin — all agreeing. Get the VolumeSnapshotClass wrong and Velero either skips your volumes or uses the wrong class, and you discover it at restore time. Two settings carry most of the weight: deletionPolicy and the Velero default-class label.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-prod-snapclass
  labels:
    # Velero uses the class carrying this label as the default for its driver.
    # Set it on exactly ONE class per CSI driver, or Velero cannot choose.
    velero.io/csi-volumesnapshot-class: "true"
driver: csi.example.storage.io
# Retain keeps the underlying snapshot if the VolumeSnapshot object is deleted —
# the safe default for DR. Delete would let object GC remove your restore point.
deletionPolicy: Retain
parameters:
  # Driver-specific: e.g. incremental snapshots, snapshot location, etc.
  csi.storage.k8s.io/snapshotter-secret-name: csi-snap-creds
  csi.storage.k8s.io/snapshotter-secret-namespace: kube-system

deletionPolicy: Retain is the audit-grade choice: deleting the Kubernetes VolumeSnapshot object leaves the actual snapshot in storage, so a stray kubectl delete or a namespace teardown cannot vaporize a recovery point. The trade-off is that you own snapshot lifecycle and must reap orphans deliberately — which is correct for data you are legally obliged to be able to recover. Pair the on-array snapshot with Velero's --snapshot-move-data so the durable copy also exists in object storage; the array snapshot gives you a seconds-fast local rollback, and the object-store copy gives you region-independent recovery. Different RPO/RTO tiers lean on different members of that pair.

Cross-Region Restore: Standing Up a Cluster That Was Never Yours

The restore that matters is the one to a cluster that did not exist when the disaster started: region A is gone, and you are bringing service up in region B from nothing but your object store. This is where the sovereignty of your backup store pays off directly — because all three backup streams live in a bucket you control, replicated off-site, you can install a fresh Velero in the standby cluster, point it at the same bucket, and restore. Nothing in the recovery path depends on region A being reachable or on a provider console you may not be able to log into.

The standby topology is a deliberate cost decision. A warm standby keeps a minimal cluster running in region B with Velero installed and the object store replicated, so recovery is a restore operation measured in minutes. A cold/rebuild approach provisions the standby cluster from GitOps and infrastructure-as-code only when disaster strikes, trading a longer RTO for near-zero standing cost. The right choice is per-tier, not per-organization.

# In the region-B cluster: install Velero pointed at the SAME object store
# (now reading the replicated bucket). Backups appear automatically.
velero backup-location create default \
  --provider aws --bucket velero-prod \
  --config region=minio,s3ForcePathStyle=true,s3Url=https://objstore-b.internal:9000 \
  --access-mode ReadOnly

# Confirm the standby cluster can see the backups taken in region A.
velero backup get

# Restore a specific backup. CSI data-mover rehydrates the volumes from
# object storage onto region-B's storage class.
velero restore create payments-dr \
  --from-backup hourly-payments-20260624-030000 \
  --include-namespaces payments,ledger \
  --restore-volumes=true

# Watch the restore and, crucially, the volume DataDownload completion.
velero restore describe payments-dr --details

A few cross-region restore realities catch teams out. StorageClass names and CSI drivers may differ between regions — Velero supports a StorageClass mapping config so PVCs bind to the right class in region B. LoadBalancer Services will provision new external addresses, so DNS or a global load balancer must be re-pointed as an explicit cutover step, not an afterthought. And volume rehydration is bandwidth-bound: moving terabytes of database state from object storage back onto block volumes is usually the single largest component of RTO, which is why the restore drill (next section) must measure it on representative data rather than on an empty test namespace.

Click to zoom

Defining RPO and RTO Per Workload — and the Trade-Off Nobody Names

RPO and RTO are the two numbers that make DR a contract instead of an aspiration. RPO — Recovery Point Objective — is the maximum data loss you accept, measured in time: an RPO of 15 minutes means a disaster may cost you up to the last 15 minutes of changes. It is governed by backup frequency and replication lag. RTO — Recovery Time Objective — is the maximum time to restore service: an RTO of one hour means you commit to being back within an hour of the disaster. It is governed by your restore mechanics, standby topology, and data volume. They are independent: you can have a tiny RPO (snapshot every minute) and a large RTO (slow restore), or vice versa, and most real systems are unbalanced in exactly that way.

The discipline is to set these per workload, not per cluster. A payment-authorization ledger and an internal Grafana instance do not deserve the same RPO, and pretending they do means you either overspend protecting dashboards or underspend protecting money. Tier your workloads explicitly:

Click to zoom

Here is the trade-off nobody names cleanly: synchronous replication buys you near-zero RPO by coupling your regions, and that coupling caps your write throughput at the speed of light between them. A Tier-0 active-active design with synchronous storage replication means every committed write must round-trip to the second region before it is acknowledged. If your regions are 20 ms apart, you have just added 20 ms of latency to the critical path of every write, and your maximum write rate is now bounded by inter-region latency, not by your storage hardware. Asynchronous replication removes that latency penalty but reintroduces RPO equal to the replication lag — typically seconds to low minutes. There is no configuration that gives you zero RPO, zero latency cost, and regional independence simultaneously; you are always choosing two of the three. Naming that trade-off explicitly, per workload, is the entire job. The teams that get DR wrong are the ones who never made the choice consciously and discovered which two they had only after an incident.

Restore Drills: Turning a Backup Into Evidence

A backup is a hypothesis. A restore is the experiment that tests it. Until you have restored — fully, into a real cluster, with the data validated — you do not know your RPO or RTO; you know the values you hoped for. The single highest-leverage thing most platform teams can do for resilience is to schedule restore drills as a recurring, calendared operation and treat a failed or slow drill as an incident with a postmortem, exactly like a production outage. This is not gold-plating. Under DORA, periodic testing of ICT continuity and recovery is a named obligation, and "we restore on a schedule and here are the timed results" is precisely the evidence an examiner expects.

A real drill restores into an isolated standby cluster (or an ephemeral one provisioned for the drill), measures the actual RTO with a stopwatch from "disaster declared" to "verification passed," computes the actual RPO from the backup timestamp versus the simulated failure time, and — non-negotiably — validates data integrity rather than just pod readiness. A pod that is Running against an empty or corrupt volume is a failed restore that looks like a success. Bake the integrity check into the drill: row counts, checksums, a known-record lookup, an application-level smoke test. Then write the numbers down with the date. That record is your DR program.

#!/usr/bin/env bash
set -euo pipefail

DRILL_START=$(date +%s)
BACKUP=$(velero backup get -o name | grep hourly-payments | sort | tail -1)
echo "Drill $(date -u +%FT%TZ): restoring ${BACKUP} into standby"

# 1) Restore into the isolated drill cluster/namespace.
velero restore create drill-$(date +%s) \
  --from-backup "${BACKUP##*/}" \
  --namespace-mappings payments:payments-drill \
  --restore-volumes=true --wait

# 2) Validate DATA, not just pod status. Fail the drill on mismatch.
EXPECTED=$(cat /drill/expected-ledger-rows)
ACTUAL=$(kubectl -n payments-drill exec deploy/postgres -- \
  psql -tA -c 'SELECT count(*) FROM ledger;')
[ "${ACTUAL}" = "${EXPECTED}" ] || { echo "INTEGRITY FAIL: ${ACTUAL}!=${EXPECTED}"; exit 1; }

# 3) Record the measured RTO. This number — dated — is the audit evidence.
RTO=$(( $(date +%s) - DRILL_START ))
echo "DRILL PASS  rto_seconds=${RTO}  backup=${BACKUP##*/}  at=$(date -u +%FT%TZ)" \
  | tee -a /drill/dr-evidence.log

Continuous validation closes the loop. Backups silently rot — a credential rotates and the data mover starts failing, a VolumeSnapshotClass label is removed, a storage migration changes the driver name, retention GC reaps a snapshot you assumed was kept. Alerting on Velero backup phase, DataUpload/DataDownload completion, etcd-snapshot CronJob success, and replication lag catches these between drills. Teams running KubeVigil and their self-hosted observability stack fold these signals into the same dashboards and alert routes as the rest of the platform, so a broken backup pages someone the day it breaks — not the day they need it.

Sovereign Backups and the Long Game

Where your backups live is a sovereignty decision with the same weight as where your workloads run. A backup sitting in a managed backup service you cannot read without the vendor's console — under their keys, in their jurisdiction, subject to their billing relationship — is a recovery path that depends on a third party being available, solvent, and cooperative at your worst moment. The sovereign pattern is the opposite: backups in your own S3-compatible object store (MinIO, Ceph RGW, or an on-prem array), encrypted under keys you hold, with object-lock/WORM immutability and versioning so a compromised credential cannot delete or overwrite your recovery points, replicated to a second site you also control. The same architecture that makes you independent of a single hyperscaler for compute makes you independent for recovery — and recovery is when independence matters most.

This is also the layer where the long game and audit-grade rigor meet. Immutable, versioned, dated restore evidence is not just operational hygiene — it is the artifact that satisfies an auditor, survives a leadership change, and still makes sense three years and several platform migrations later. Systems built to last decades need recovery procedures that last decades, and a procedure is only durable if it is exercised, measured, and documented continuously rather than designed once and trusted forever. A DR plan that has never been run is technical debt that compounds silently; a DR plan exercised on a schedule, with timed evidence in your own store, is one of the few infrastructure investments whose value grows with the age of the system it protects.