Immutable Backups: Object Lock and Proving You Can Restore

Most backup architectures are designed against hardware failure, then relabelled as ransomware defence. That is not the same threat model: a failing disk does not hold your credentials.

The backup your attacker deletes first

Assume the position you will be in: the adversary has cluster-admin, has read every Secret your backup CronJob mounts, and is not in a hurry. Competent operators do not encrypt first: they remove the recovery points, because a victim who can restore does not pay.

Sophos measured it. Across 2,974 ransomware victims, “94% of organizations hit by ransomware in the past year said that the cybercriminals attempted to compromise their backups”, and 57% of those attempts succeeded. Median recovery cost was $3M where backups were compromised against $375K where they were not — March 2024 figures, not current-year data.

The question is narrow: given an attacker holding a valid backup credential, which recovery points survive? Our Kubernetes disaster-recovery article covers what to back up; this one assumes someone took it.

What Object Lock actually guarantees, and what it does not

S3 Object Lock is a per-object-version retention record. Versioning is a hard prerequisite — “Object Lock works only in buckets that have S3 Versioning enabled” — and protection is scoped to the version you locked. Retention periods and legal holds do not prevent new versions or delete markers on top.

That scoping produces the subject's most common misreading. A DELETE naming a version ID returns 403 Access Denied; one that does not returns 200 OK and inserts a delete marker, becoming the current version. The object vanishes from the listing; nothing was destroyed. If your test is “I deleted it and it disappeared”, you have tested nothing.

Two modes. GOVERNANCE has a documented override: the s3:BypassGovernanceRetention permission plus an explicit x-amz-bypass-governance-retention:true header — which AWS's console sends by default. COMPLIANCE has none: “a protected object version can't be overwritten or deleted by any user, including the root user in your AWS account”, and retention cannot be shortened.

Enforcement is per access path, not per bucket

Read the scope in a store's enforcement language. Ceph's December 2025 RGW deep dive is precise: once locked, an object “cannot be deleted or overwritten through the S3 endpoint, not even by an RGW admin account, until the lock expires”. Through the S3 endpoint: strong, narrow.

A February 2026 SeaweedFS thread carries the lesson. A reporter filed that COMPLIANCE failed to block deletes; the maintainer showed the tests were not comparable — “one with version id, one without” — and the reporter retracted. The residual finding was architectural: deletes still went through the filer UI, “a different component… guarded separately.” Not an Object Lock bug — a second data plane.

Generalise it. Any store with an admin API, UI, filer or POSIX gateway has a delete path the S3 layer never mediates, and the host filesystem — the OSD data directories, the drives — sits under all of them.

Lifecycle and replication are *not* on that list, and getting it backwards costs a design. AWS documents that lifecycle “configurations continue to function normally on protected objects, including placing delete markers. However, a locked version of an object cannot be deleted by a S3 Lifecycle expiration policy”; Ceph's deep dive agrees Object Lock takes precedence, and replication copies retention metadata to a destination that must itself have Object Lock enabled. The residual risk is subtler: a delete marker on top is “not WORM-protected”, so a restore resolving by key sees nothing while every locked version is intact. Resolve by version ID.

Click to zoom

Which store you run decides how much of this you get. Ceph RGW implements the full surface: bucket configuration, per-object retention in both modes, legal hold. Garage implements none — its matrix marks all six Object Lock endpoints Missing, and missing endpoints return 501 — a team that left MinIO for Garage has no immutability path. MinIO's object-locking documentation now sits under the commercial AIStor product, which requires versioning and, since RELEASE.2025-05-20T20-30-00Z, allows locking on an existing bucket. Our store comparison never carried this axis.

Even on Ceph, adding Object Lock to an existing bucket is a release question. The API reference documents PUT Bucket Object Lock returning 409 InvalidBucketState when the bucket was not created with it enabled; the deep dive says Tentacle enables it on existing versioned buckets, citing ceph/ceph#62063 — itself a backport, so it may not be Tentacle-only. Check your release, then test.

#!/usr/bin/env bash
set -euo pipefail

BUCKET="${BUCKET:-backups-worm}"
AWS="aws s3api --endpoint-url ${S3_ENDPOINT:?set S3_ENDPOINT}"
# Setup is not the writer's job; backup-writer may only PUT.
ADMIN="$AWS --profile retention-administrator"
WRITER="$AWS --profile backup-writer"

$ADMIN create-bucket --bucket "$BUCKET" --object-lock-enabled-for-bucket
$ADMIN put-object-lock-configuration --bucket "$BUCKET" \
  --object-lock-configuration \
  'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=35}}'

printf 'probe %s' "$(date -u +%FT%TZ)" > /tmp/probe
VID=$($WRITER put-object --bucket "$BUCKET" --key _lockcheck/probe \
        --body /tmp/probe --query VersionId --output text)

# THE TEST. A DELETE with no version ID returns 200 OK and a delete
# marker: correct, and proof of nothing.
if $ADMIN delete-object --bucket "$BUCKET" --key _lockcheck/probe \
     --version-id "$VID" 2>/tmp/lockcheck.err; then
  echo "FAIL: versioned DELETE succeeded - not WORM" >&2
  exit 1
fi
grep -q AccessDenied /tmp/lockcheck.err || {
  cat /tmp/lockcheck.err >&2; echo "FAIL: refused, but not AccessDenied" >&2
  exit 1
}

# The bytes must read back intact, and the mode must be what you set.
$ADMIN get-object --bucket "$BUCKET" --key _lockcheck/probe \
  --version-id "$VID" /tmp/readback >/dev/null
cmp -s /tmp/probe /tmp/readback || { echo "FAIL: readback differs" >&2; exit 1; }
$ADMIN get-object-retention --bucket "$BUCKET" --key _lockcheck/probe \
  --version-id "$VID" --query 'Retention.Mode' --output text | grep -qx COMPLIANCE

echo "PASS: versioned DELETE refused, object intact, mode COMPLIANCE"

Three identities, not one: the credential split

A lock is only as good as what the compromised credential can still do. An identity that can call PutObjectLockConfiguration weakens the default for everything written after; one holding s3:BypassGovernanceRetention makes GOVERNANCE decorative. Three identities:

• backup-writer — on the cluster, in a Secret, assumed compromised. s3:PutObject and s3:PutObjectRetention on one prefix, nothing else.
• retention-administrator — never on the cluster. Owns lock configuration, may lengthen retention, owns restic forget and prune.
• break-glass — offline, split-knowledge, audited on use. For a GOVERNANCE bypass; under COMPLIANCE it does nothing, which is the point.

Two details are easy to get backwards. Do not deny s3:PutObjectRetention unconditionally: AWS requires it “in order to place an Object Retention configuration on objects”, so denying it fails the writer's PUT with AccessDenied. That is loud; its mirror is silent — omit the lock headers into a bucket with no default retention and you get 200 OK and an unlocked object. And watch the operator — AWS's example caps s3:object-lock-remaining-retention-days with NumericGreaterThan. A floor is the opposite comparison; copying the example permits one-second locks.

{
  "Version": "2012-10-17",
  "Id": "worm-backup-writer",
  "Statement": [
    {
      "Sid": "WriterMayNotDeleteVersionsOrBypassOrReconfigure",
      "Effect": "Deny",
      "Principal": { "AWS": "arn:aws:iam::000000000000:user/backup-writer" },
      "Action": [
        "s3:DeleteObjectVersion",
        "s3:BypassGovernanceRetention",
        "s3:PutObjectLockConfiguration",
        "s3:PutBucketVersioning",
        "s3:PutLifecycleConfiguration",
        "s3:PutBucketReplication"
      ],
      "Resource": [
        "arn:aws:s3:::backups-worm",
        "arn:aws:s3:::backups-worm/*"
      ]
    },
    {
      "Sid": "RetentionFloorNotCeiling",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObjectRetention",
      "Resource": "arn:aws:s3:::backups-worm/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "35"
        }
      }
    }
  ]
}

Versioning is the substrate the lock is built on, so the writer may not touch it. Lifecycle and replication cannot expire a locked version, but the writer has no business reconfiguring either: a delete marker is a restore problem it should not be able to create. Keep the retention-administrator credential outside the cluster's secrets management path entirely.

Your backup tool probably cannot write to a locked bucket

A correct bucket now meets an incompatible client. Velero's v1.18 documentation — v1.18.2 is the latest release as of 13 August 2026 — states that “there was no explicit support in Velero to work with object storage that has "immutability" configuration”, because Velero modifies backup metadata after the initial write. On versioned S3-style stores that is not a crash, which is why it gets missed: writes succeed as new versions, “but when backups are deleted, old versions of the objects will not be deleted.” The bucket grows monotonically, and the upstream request is open since February 2025.

restic fails differently, and its own design reference has the mechanism: “restic processes are required to create a lock on the repository before doing anything”, and a lock is a file in locks/, created at the start of a run and removed at the end — so a default COMPLIANCE retention blocks restic's cleanup every run. An open request to relocate lock files has been filed since 4 June 2025, still labelled “state: need feedback”.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: worm-artifact-ship
  namespace: backup-worm
spec:
  schedule: "17 */6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: backup-writer
          containers:
            - name: ship
              image: registry.example.com/backup-tools:2026.08.1
              envFrom:
                - secretRef: { name: backup-writer-s3 } # AWS_* credentials
              env:
                - { name: S3_ENDPOINT, value: https://s3.example.com }
                # point this at however etcd is reachable on YOUR control
                # plane: static pods usually are not behind a Service.
                - { name: ETCD_ENDPOINTS, value: https://etcd.internal:2379 }
              command: ["/bin/bash", "-euo", "pipefail", "-c"]
              args:
                - |
                  # Opaque artifacts only. Velero and restic mutate after
                  # the first write, so they belong on tier 2.
                  ts=$(date -u +%Y%m%dT%H%M%SZ)
                  snap="/scratch/etcd-$ts.db"
                  etcdctl --endpoints="$ETCD_ENDPOINTS" \
                    --cacert=/tls/ca.crt --cert=/tls/tls.crt --key=/tls/tls.key \
                    snapshot save "$snap"
                  sha256sum "$snap" > "$snap.sha256"
                  until=$(date -u -d '+35 days' +%FT%TZ)
                  for f in "$snap" "$snap.sha256"; do
                    aws s3api put-object --endpoint-url "$S3_ENDPOINT" \
                      --bucket backups-worm --key "etcd/$ts/$(basename "$f")" \
                      --body "$f" --object-lock-mode COMPLIANCE \
                      --object-lock-retain-until-date "$until" \
                      --query VersionId --output text
                  done
              volumeMounts:
                - { name: scratch, mountPath: /scratch }
                - { name: etcd-tls, mountPath: /tls, readOnly: true }
          volumes:
            - { name: scratch, emptyDir: { sizeLimit: 8Gi } }
            - { name: etcd-tls, secret: { secretName: etcd-client-tls } }

restic gets a second tier with a weaker guarantee — say so rather than filing both under “immutable”. It describes append-only as a backend mode where “it can only be written to and read from, while delete and overwrite operations are denied”, and is candid: forget and prune need full delete access, so an attacker holding that voids the protection. An adversary appending garbage snapshots can also manoeuvre a retention policy into pruning the legitimate ones — hence --keep-within.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: restic-rest-server
  namespace: backup-appendonly # deliberately not the WORM namespace
spec:
  replicas: 1
  selector:
    matchLabels: { app: restic-rest-server }
  template:
    metadata:
      labels: { app: restic-rest-server }
    spec:
      containers:
        - name: rest-server
          image: restic/rest-server:0.14.0
          # CMD is /entrypoint.sh, which ignores argv: override command or
          # --append-only never reaches the binary. htpasswd -B -c .htpasswd backup
          command: ["rest-server"]
          args: ["--path=/data", "--append-only", "--htpasswd-file=/auth/.htpasswd"]
          ports: [{ containerPort: 8000 }]
          volumeMounts:
            - { name: repo, mountPath: /data }
            - { name: htpasswd, mountPath: /auth, readOnly: true }
      volumes:
        - { name: repo, persistentVolumeClaim: { claimName: restic-repo } }
        - { name: htpasswd, secret: { secretName: restic-htpasswd } }
---
apiVersion: v1
kind: Service # a Deployment name is not a DNS name; without this, NXDOMAIN
metadata: { name: restic-rest-server, namespace: backup-appendonly }
spec:
  selector: { app: restic-rest-server }
  ports: [{ port: 8000, targetPort: 8000 }]
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: restic-backup, namespace: backup-appendonly }
spec:
  schedule: "*/30 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: restic
              image: restic/restic:0.19.1
              env:
                # rest:http://<user>:<pass>@restic-rest-server:8000/ — creds
                # match .htpasswd; repo restic-init'd out of band. Plain
                # HTTP: no --tls here, so terminate at the mesh.
                - name: RESTIC_REPOSITORY
                  valueFrom:
                    secretKeyRef: { name: restic-rest-url, key: url }
                - { name: RESTIC_PASSWORD_FILE, value: /secrets/repo-password }
              # backup only. forget and prune need delete access, so they
              # run on the retention-admin host with --keep-within 35d.
              args: ["backup", "/data"]
              volumeMounts:
                - { name: data, mountPath: /data, readOnly: true }
                - { name: repo-password, mountPath: /secrets, readOnly: true }
          volumes:
            - { name: data, persistentVolumeClaim: { claimName: app-data } }
            - name: repo-password # Secret keys become filenames
              secret:
                secretName: restic-repo-password
                items: [{ key: repo-password, path: repo-password }]

Air gaps that are actually gapped

“Air gap” has no agreed definition — vendors apply it to network-reachable replicas — so arguing the word is unproductive. Replace it with a demonstration: from the production cluster, using the credential it holds, try to reach the second copy. Resolve the endpoint, open the port, list the bucket. If any of that succeeds, the copy shares a failure domain with the cluster.

Run it quarterly and keep the output — the answer changes without anyone deciding to change it: a peering route, a firewall rule, a DNS entry. Same discipline as chaos-engineering resilience verification. And the 3-2-1-1-0 rule you meet in procurement is Veeam's own extension of 3-2-1, a vendor framework rather than a standard — though the two digits it adds are this article's subject.

The restore rehearsal that produces evidence

Everything above is a claim about write paths. None of it says the backups restore. Sophos' State of Ransomware 2026 (2,158 leaders across 17 countries, fielded January–March 2026) reports backup-based recovery rose to 66% of attacks where data was encrypted, up from 54%, and recommends backups be tested regularly and stored offline or immutably.

Where DORA applies, the test is not optional: Article 12(2) requires that “testing of the backup procedures and restoration and recovery procedures and methods shall be undertaken periodically.” Article 12(3) governs where you restore to — entities restoring with their own systems must use ICT systems “physically and logically segregated from the source ICT system.” Neither mentions credentials or bucket policy: the identity split above is an IAM argument, not a regulatory one. The artifact below is our evidence format: “we test our backups” does not survive a DORA audit.

Click to zoom

Four properties separate a drill from a demo: address the recovery point by version ID, never by key; restore onto a plane with no route to production and no production secrets; assert usability at the application level, because a Running pod only proves the manifest applied; and measure both clocks.

#!/usr/bin/env bash
# Restore rehearsal. Runs on the restore plane, never the source cluster.
set -euo pipefail

BUCKET=backups-worm
KEY="${1:?usage: drill.sh <object key>}"
OUT="evidence/$(date -u +%Y%m%dT%H%M%SZ).json"
S3="aws s3api --endpoint-url ${S3_ENDPOINT:?} --profile restore-reader"
started=$(date -u +%FT%TZ)
t0=$(date +%s)
mkdir -p /scratch evidence

# Newest *version*, by VersionId: by key you would follow a delete marker,
# the one thing the writer credential can still add.
VID=$($S3 list-object-versions --bucket "$BUCKET" --prefix "$KEY" \
        --query 'sort_by(Versions,&LastModified)[-1].VersionId' --output text)
RET=$($S3 get-object-retention --bucket "$BUCKET" --key "$KEY" \
        --version-id "$VID" --output json)

$S3 get-object --bucket "$BUCKET" --key "$KEY" --version-id "$VID" \
  /scratch/artifact >/dev/null
digest=$(sha256sum /scratch/artifact | cut -d' ' -f1)

# Separate kube context, no route to production:
./restore-into.sh --context restore-plane --namespace drill \
  --artifact /scratch/artifact

# Usability, not liveness:
rows=$(kubectl --context restore-plane -n drill exec deploy/app -- \
         app-verify --count-rows)
newest=$(kubectl --context restore-plane -n drill exec deploy/app -- \
         app-verify --newest-record-utc)
[ "$rows" -gt 0 ] || { echo "FAIL: restored dataset is empty" >&2; exit 1; }

rto=$(( $(date +%s) - t0 ))
rpo=$(( $(date +%s) - $(date -u -d "$newest" +%s) ))

jq -n --arg key "$KEY" --arg vid "$VID" --argjson ret "$RET" \
      --arg digest "$digest" --argjson rto "$rto" --argjson rpo "$rpo" \
      --arg started "$started" --argjson rows "$rows" \
      --arg op "${DRILL_OPERATOR:?}" \
  '{object_key: $key, version_id: $vid, retention: $ret.Retention,
    artifact_sha256: $digest, measured_rto_seconds: $rto,
    observed_rpo_seconds: $rpo, assertion_result: $rows,
    started_utc: $started, operator: $op}' > "$OUT"

cosign sign-blob --key env://DRILL_SIGNING_KEY --yes "$OUT" > "$OUT.sig"
echo "evidence: $OUT"

Failure modes, cost, and the exit ramp

The dominant failure mode is not an attacker beating the lock; it is capacity. Under COMPLIANCE, retention length times write volume is a commitment you cannot revise downward, and the Velero version leak compounds it — model it against the replication factor your storage layer applies. The second is a policy that looks like success: a condition key your store ignores, a default retention never set, a writer that quietly stopped sending lock headers. Each yields unlocked objects behind green dashboards.

The exit ramp is unusual: immutability is designed to prevent movement. You cannot migrate a lock: it does not survive a copy onto another product. You write to the new store, re-apply retention at the destination, and let the old bucket age out on its original schedule, running both until the last locked version expires. Budget that overlap as the real cost of changing object stores.

The long game: a standing assertion

Immutability is not a project that completes. It is a small set of assertions that must stay true across a decade of upgrades, each with an expiry date set by somebody else's release notes:

1. A versioned DELETE against a locked object is refused, and the object still reads back.
2. The production credential cannot delete a version, bypass governance, reconfigure the lock, or reach the off-site copy.
3. Every non-S3 path to the bytes — admin API, filer, console, host filesystem — is enumerated and guarded.
4. Each backup tool sits on the tier its write pattern supports.
5. Lifecycle and replication are configured by the retention-administrator, never the writer.
6. A drill inside the last quarter produced an artifact with measured RTO, observed RPO and a passing assertion.

Re-run all six whenever the backend, its version, a backup tool or the credential model changes. Systems that last decades are not the ones nobody touched; they are the ones whose guarantees were re-proven every time something moved underneath.