IT Disaster Recovery Testing Without a Maintenance Window

A maintenance window is a generous thing to give a test, and the one condition guaranteed to be absent during the event it rehearses. Its absence changes the code path the cluster takes, not just the stress. Tabletop, parallel and full-interruption say nothing about which branch of the operator's state machine you exercised — and that branch is the result.

The Drill You Can Schedule Is the Drill That Proves the Least

CloudNativePG's documentation is explicit. During a switchover its instance manager "first issues a CHECKPOINT, then initiates a fast shutdown of PostgreSQL before the designated new primary is promoted". Every committed byte lands before promotion, so a switchover cannot measure a loss it prevented. kubectl cnpg promote is documented as the way to "test a switch-over situation".

Failover is a different sequence. After unexpected errors longer than .spec.failoverDelay — default 0 — the cluster enters failover mode and the controller proceeds "in two steps": it marks the TargetPrimary as pending, and then "once all WAL receivers are stopped, there will be a leader election, and a new primary will be named."

When the old primary is reachable the operator still tries to shut it down, but shutdown is now a consequence of the failure, not a precondition of promotion. When it is genuinely gone, nothing checkpoints: "When synchronous replication is not enabled, some data loss is expected and accepted during failover, as a replica may lag behind the primary when promoted."

Click to zoom

What Removing the Maintenance Window Actually Changes

Dropping the window does not drop the controls; it moves them from the calendar into the cluster. Two things get written down before the first fault — an abort criterion invented at T+90 seconds is not one.

• Blast radius. One cluster, one namespace, one primary, replication verified healthy, not assumed — and the node branch is wider than that, since destroying a node takes every pod on it. A drill on a degraded cluster is an outage with paperwork.
• Abort criteria, each with a restore path — including the forgotten one: no primary appears at all. Quorum failover can legitimately refuse to promote and "wait for the situation to change": correct, not a hang. The restore path is returning the missing replicas. kubectl cnpg promote overrides the check, but the docs call that "a last resort" carrying explicit data loss.

Instrument First: The Four Numbers a Drill Has to Produce

A drill without instrumentation produces an anecdote. Four numbers make it evidence: RTO as write unavailability, RPO in acknowledged transactions, RPO in bytes, reconnect time. Three are unobtainable afterwards; instrument before T0. The probe commits a row against the read-write service in a loop, appending row id, commit LSN and timestamp to a local ledger and recording errors rather than discarding them. That error window is the RTO. An exact lost-transaction count exists only because the probe keeps that ledger; against real traffic you bound the loss.

-- Pre-drill baseline. Capture to the drill log.
SELECT
    now()                                   AS captured_at,
    pg_current_wal_lsn()                    AS primary_lsn,
    s.application_name,
    s.sync_state,
    s.flush_lsn,
    -- PREDICTED byte lag if the primary vanished at this instant. A bound
    -- to check the result against, never the result itself.
    pg_wal_lsn_diff(pg_current_wal_lsn(), s.flush_lsn) AS flush_lag_bytes,
    -- A commit DELAY, a time measure. Context, not the RPO.
    s.flush_lag AS flush_lag_interval
FROM pg_stat_replication s
ORDER BY s.application_name;

That last comment is load-bearing. PostgreSQL calls the lag columns "the commit delay that was (or would have been) introduced by each synchronous commit level" — useful, not a byte count. pg_wal_lsn_diff() "calculates the difference in bytes" — an RPO, but only once lsn1 is the right LSN, which the baseline is not.

Bound the Loss Before You Measure It

Measuring an unbounded RPO on a live cluster is not a drill; it is a scheduled incident. dataDurability "can be set to required or preferred, with the default being required". Under preferred, "write operations will continue even if fewer than the requested number of standbys are available" — availability bought with a wider worst case.

Quorum-based failover is the second lever: before promoting, "the operator performs a quorum check, following the principles of the Dynamo R + W > N consistency model", tracked in a FailoverQuorum resource. It became stable in 1.28.0, "graduating from the previous alpha.cnpg.io/failoverQuorum annotation" — which is deprecated but still takes precedence, so a stale one means the cluster is not configured the way its manifest reads.

CloudNativePG 1.30 adds a third: a Kubernetes Lease "that acts as a mutex serializing primary promotion". The release notes bound its scope — "The lease is a promotion gate, not a fence". After an abrupt loss a candidate "must observe the lease unchanged for a full leaseDurationSeconds before it may take over"; after a clean switchover there is no such wait. Budget it into the abrupt path only.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ledger
  namespace: data
spec:
  instances: 3

  postgresql:
    synchronous:
      # 'any' = quorum-based synchronous replication.
      method: any
      number: 1
      # required (the default) blocks commits when too few replicas remain.
      # preferred keeps writing, widening the worst-case RPO.
      dataDurability: required
      # Stable since 1.28.0. The alpha.cnpg.io/failoverQuorum annotation is
      # deprecated but still takes precedence -- remove it from the object.
      failoverQuorum: true

  # Failover mode entered after errors last longer than this.
  failoverDelay: 0

  # These bound the *graceful* paths. A drill that spends them
  # has not injected a crash.
  smartShutdownTimeout: 180
  stopDelay: 1800

  primaryLease:
    leaseDurationSeconds: 15
    renewDeadlineSeconds: 10
    retryPeriodSeconds: 2
    releasedLeaseDurationSeconds: 1

  storage:
    # size is required unless the PVC template already sets it.
    size: 500Gi              # match the existing volume
    storageClass: local-nvme # substitute yours -- node-local strands the PVC

Inducing Failover, Not Switchover, on a Cluster Still Taking Writes

This is where most drills fail. A plain kubectl delete pod looks like violence and is not: "the instance manager will take care of shutting down PostgreSQL in an appropriate way" — a CHECKPOINT, then a smart shutdown, escalating to fast only if PostgreSQL is still up. A messy switchover, and a flattering number.

Adding --grace-period=0 --force does not convert that into a crash. Kubernetes warns force deletion "does not wait for confirmation that the pod's processes have been terminated, which can leave those processes running". Two writable primaries is not a bolder drill; it is an API-desync hazard, and belongs in the abort criteria.

Two faults do reach the failover path. Node-level failure is documented end to end: "once the node stops reporting, the Kubernetes node lifecycle controller is the one that flips the condition to False", governed by --node-monitor-grace-period (40s on Kubernetes 1.29–1.31, 50s from 1.32). The operator sees it unready roughly 40 to 55 seconds later, and only then does failoverDelay start. A poller giving up inside that band calls a working drill broken.

The second is a process crash, sourced to the operator's lifecycle code rather than its documentation: when "postmaster has crashed", "we want to terminate the instance manager and let the Kubelet restart the Pod". Note what that promises — a Pod restart. Whether it becomes a promotion is guaranteed by neither source, so do not record it as one.

#!/usr/bin/env bash
set -euo pipefail

NS=data; CLUSTER=ledger; BRANCH="${1:?A (node) or B (postmaster)}"; DEADLINE=300
LOG="drill-$(date -u +%Y%m%dT%H%M%SZ).log"
# cnpg.io/instanceRole replaces the long-deprecated role= label.
PRIMARY="$(kubectl -n "$NS" get pods \
  -l "cnpg.io/cluster=$CLUSTER,cnpg.io/instanceRole=primary" \
  -o jsonpath='{.items[0].metadata.name}')"
NODE="$(kubectl -n "$NS" get pod "$PRIMARY" -o jsonpath='{.spec.nodeName}')"
# The last LSN this primary will ever report: the upper bound on lsn1.
T0_LSN="$(kubectl -n "$NS" exec "$PRIMARY" -c postgres -- \
  psql -qtAX -c 'SELECT pg_current_wal_lsn()')"
# The write probe is ALREADY running and never paused.
echo "T0 $(date -u +%Y-%m-%dT%H:%M:%SZ) $BRANCH primary=$PRIMARY lsn=$T0_LSN" \
  | tee -a "$LOG"

case "$BRANCH" in
  # A: an actually unreachable node. NOT a drain, NOT a pod delete.
  # Assumes the libvirt domain is the node name; map it if not.
  A) ssh "${HYPERVISOR:?set HYPERVISOR}" "virsh destroy $NODE" ;;
  # B: the postmaster. Not PID 1 (instance manager), not a backend.
  # Non-zero exit expected: the container dies under the exec.
  B) kubectl -n "$NS" exec "$PRIMARY" -c postgres -- \
       bash -c 'kill -9 "$(head -1 "$PGDATA/postmaster.pid")"' || true ;;
  *) echo "unknown branch: $BRANCH" >&2; exit 2 ;;
esac

# T1 = the operator NAMES A DIFFERENT PRIMARY. Not the cluster phase:
# "Cluster in healthy state" waits for the dead instance to rejoin (T3),
# or never, if its PVC is stranded on the dead node.
UNTIL=$(( $(date -u +%s) + DEADLINE ))
until NEW="$(kubectl -n "$NS" get cluster "$CLUSTER" \
      -o jsonpath='{.status.currentPrimary}')"; [ -n "$NEW" ] && [ "$NEW" != "$PRIMARY" ]; do
  # Quorum refusing to promote is a CORRECT outcome. Record it and restore.
  [ "$(date -u +%s)" -lt "$UNTIL" ] \
    || { echo "ABORT no promotion in ${DEADLINE}s" | tee -a "$LOG"; exit 1; }
  sleep 2
done
echo "T1 $(date -u +%Y-%m-%dT%H:%M:%SZ) promoted=$NEW" | tee -a "$LOG"
# T2 = the probe ledger's first commit after T0.

Click to zoom

Measuring the RPO That Actually Happened

The measurement runs against what is still alive. Once the old primary's postmaster is gone, so is pg_current_wal_lsn() and the exact crash LSN. What survives is the probe ledger and the standby's cut-off: pg_last_wal_receive_lsn() "returns the last write-ahead log location that has been received and synced to disk by streaming replication".

Which LSN goes on the left is the whole measurement. Not the T-5m baseline: by T0 the standby is minutes of WAL past it, so the subtraction goes negative. The honest lsn1 is the last commit the probe saw acknowledged before T0 — what the system promised a client was durable. T0_LSN, captured a heartbeat before the fault, bounds it above.

-- 0. The probe's ledger is a client-side file. Load it before joining.
CREATE TEMP TABLE probe_ledger (row_id bigint, commit_lsn pg_lsn,
                                acked_at timestamptz);
-- Paste the T0 field from the drill log here, once. Same format.
\set t0 '2026-09-09T06:14:03Z'
\copy probe_ledger FROM 'drill-probe-ledger.csv' WITH (FORMAT csv)

-- 1. Bytes lost. lsn1 = the last commit the CLIENT WAS ACKED before T0.
--    The crash LSN died with the postmaster; T0_LSN bounds it above.
--    Never the T-5m baseline: the standby is minutes of WAL past it,
--    so that subtraction returns a negative number, not an RPO.
SELECT pg_wal_lsn_diff(
         (SELECT commit_lsn FROM probe_ledger
           WHERE acked_at < TIMESTAMPTZ :'t0'
           ORDER BY commit_lsn DESC LIMIT 1),
         pg_last_wal_receive_lsn()) AS rpo_bytes;  -- <= 0: nothing acked lost

-- 2. Transactions lost: acked rows the promoted standby lacks. probe_rows
--    is the probe's target table; it survived promotion.
SELECT l.row_id, l.commit_lsn, l.acked_at
FROM   probe_ledger l
LEFT   JOIN probe_rows r USING (row_id)
WHERE  r.row_id IS NULL
  AND  l.acked_at < TIMESTAMPTZ :'t0'
ORDER  BY l.acked_at;

Rejoining the old instance is evidence too. pg_rewind "expects to find WAL in the target cluster's pg_wal directory reaching all the way back to the point of divergence". If that WAL is gone the rejoin becomes a full re-clone — a retention fact the drill surfaces cheaply.

The etcd Half: Leader Transfer Is Not Member Loss

The same distinction has an exact analogue a layer down. etcdctl move-leader "transfers leadership from the leader to another member in the cluster" — the switchover. Killing the leader is the failover, and the only one that exercises the election.

Killing has to mean SIGKILL. Under SIGTERM — systemctl stop, crictl stop, a static-pod delete — EtcdServer.Stop() runs TryTransferLeadershipOnShutdown() before HardStop(): "when stopping leader, Stop transfers its leadership to one of its peers before stopping the server". That is move-leader in a kill's clothing, and the sub-second gap it reports is an election that never ran.

State the arithmetic first: "For a cluster with n members, quorum is (n/2)+1" — three tolerate one failure, five tolerate two. Kill a second member of a three-node cluster and you have lost the control plane, not run a drill. Defaults give a "100ms heartbeat interval" and "a 1000ms election timeout" — the floor on noticing a dead leader.

#!/usr/bin/env bash
set -euo pipefail
EPS="https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379"
IFS=',' read -r -a EP_LIST <<<"$EPS"
LEADER_EP=""; TRANSFEREE_ID=""

# Derive both; never hardcode. An endpoint is the leader when the
# MemberID it reports equals the Leader it names. Both print decimal;
# --hex converts only MemberID, breaking the comparison.
derive() {
  local ep st mid lid; LEADER_EP=""; TRANSFEREE_ID=""
  for ep in "${EP_LIST[@]}"; do
    st="$(etcdctl --endpoints="$ep" -w fields endpoint status 2>/dev/null)" || continue
    read -r mid lid < <(awk -F' : ' '{gsub(/"/,"",$1)}
      $1=="MemberID"{m=$2} $1=="Leader"{l=$2} END{print m, l}' <<<"$st")
    if [ "$mid" = "$lid" ]; then LEADER_EP="$ep"
    else TRANSFEREE_ID="${TRANSFEREE_ID:-$mid}"; fi
  done
  : "${LEADER_EP:?no leader, so no quorum -- do not drill a cluster already down}"
}

etcdctl --endpoints="$EPS" -w table endpoint status  # baseline: leader, term

# 1. Switchover equivalent: graceful handover, sent TO THE LEADER. Aimed
#    at a follower, etcdctl refuses -- "no leader endpoint given".
# move-leader parses its argument as HEX; -w fields gave you decimal.
derive
etcdctl --endpoints="$LEADER_EP" move-leader \
  "$(printf '%x' "${TRANSFEREE_ID:?no follower}")"

# 2. Failover: leadership just moved -- re-derive before killing.
#    SIGKILL, never "systemctl stop": SIGTERM reaches EtcdServer.Stop(),
#    which transfers leadership before HardStop -- a second switchover.
#    Same trap in crictl stop or deleting a static-pod etcd.
derive
LEADER_HOST="$(awk -F'[/:]' '{print $4}' <<<"$LEADER_EP")"
REMAINING="$(printf '%s\n' "${EP_LIST[@]}" | grep -vxF "$LEADER_EP" | paste -sd, -)"
START=$(date -u +%s)
ssh "$LEADER_HOST" 'sudo systemctl kill -s SIGKILL etcd'
# Poll the SURVIVORS: health on a list still naming the killed member
# never comes back clean, so that loop hangs and times nothing.
until etcdctl --endpoints="$REMAINING" endpoint health >/dev/null 2>&1; do sleep 1; done
echo "election gap: $(( $(date -u +%s) - START ))s"
etcdctl --endpoints="$REMAINING" -w table endpoint status  # new leader, higher term

Clients, Reconnection, and the Failure Modes That Make a Drill Lie

A database back in eight seconds but noticed four minutes later is a four-minute outage. libpq accepts "a comma-separated list of host names… tried in order", and target_session_attrs=read-write requires "the server must not be in hot standby mode". Measure reconnect separately; it fails independently.

Four ways a drill returns a reassuring number that is wrong:

1. You exercised the shutdown path. Any zero RPO from promote, a pod delete or a drain measured a graceful shutdown, not a failover.
2. You measured the operator, not the application. Cluster in healthy state is an opinion; the probe's first commit is the fact. Take T2 from the probe.
3. You ran it at 3 a.m. on a quiet cluster. Lag scales with write volume, and lag is the gap. A trough-load drill reports a best case as a bound.
4. A pooler absorbed the failure. A pooler can mask a promotion so completely the probe never errors. Instrument both sides.

A chaos toolchain does this at scale; Stribog's note on turning LitmusChaos and Chaos Mesh results into audit evidence covers that pipeline, easier to reconstruct when traces and logs already land somewhere you control.

The Evidence a Drill Has to Leave Behind

An artifact worth keeping records the fault, T0 through T4 with wall-clock stamps, the cluster spec, the four numbers, and — the part most reports omit — what it does not evidence.

This is where reports overclaim. DORA Article 11(6)(a) requires financial entities to "test the ICT business continuity plans and the ICT response and recovery plans… at least yearly". The next subparagraph asks for something else: "switchovers between the primary ICT infrastructure and the redundant capacity, backups and redundant facilities" under Article 12. This drill never leaves the cluster and restores nothing, so it does not discharge that. NIS2 Article 21(2)(c) lists "business continuity, such as backup management and disaster recovery" among minimum measures — a duty this feeds, not one it certifies.

The restore side is a different drill — rebuilding a cluster that no longer exists — covered in the Velero and etcd-snapshot side of Kubernetes DR, and proving those backups survive an attacker with object-lock immutability. Run both; report them separately.

Exit Ramps and the Long Game

Nothing in the method belongs to CloudNativePG. The LSN arithmetic is PostgreSQL's, so it holds under Patroni or repmgr unchanged. The switchover-versus-failover split belongs to consensus systems, not to this operator. That is anti-lock-in in practical form: a vendor's runbook retires with the vendor, while four numbers and a ledger schema survive a migration still comparable.

Version drift is the other long-game tax, which is why these numbers are pinned. CloudNativePG 1.30.0 shipped on 29 June 2026 as the current stable minor series; the same day 1.28.4 shipped as "the final release in the 1.28.x series", with 1.28 "no longer supported". Defaults move between minors, so a result means something only alongside the version it was measured under — hence the manifest in the bundle.

Run it quarterly, on a live cluster, probe running. The first will be uncomfortable and will find something; that is the return. By the fourth the interesting number is not the RPO but its variance, and a measured range beats an aspiration.