Records Retention Policy: Delete the Data or Destroy the Key

Retention schedules think in record classes: invoices, payment audit trails, access logs. Infrastructure thinks in copies. The invoice row has a replica, a WAL archive, a bucket export, a log line, a Kafka event and a locked backup, each on a clock the schedule never mentions. When the period ends, the row goes. The copies stay.

What follows maps one record class onto its stores, sets floors against ceilings, compares two ways to end a period, and ends with what an auditor asks for: a record of what disposal reached.

A schedule is written against one copy

US federal records law has the cleanest vocabulary for this. 36 CFR 1220.18 defines disposition as "those actions taken regarding records no longer needed for the conduct of the regular current business of the agency", and disposition authority as "the legal authorization for the retention and disposal of records." Retention and disposal are one authority. The disposal half starts with an inventory, per record class, of every copy and its clock:

• System of record — the table the schedule was written against.
• Replicas and the WAL or PITR archive — replicas follow the primary; the archive follows its own window.
• Object storage — exports and attachments in a versioned bucket.
• Logs, events and metrics — Loki, Kafka and Prometheus data carrying identifiers from the record.
• Backups — restic or similar, often on locked media.
• Key management — if anything is encrypted, the key store and its snapshots are copies too.

Click to zoom

Draw this per record class. Every node without a clock silently contradicts the policy.

Floors and ceilings: two clocks that collide

Retention law comes in two directions. Floors say keep at least this long. SEC Rule 17a-4(a) requires broker-dealers to preserve certain records "for a period of not less than 6 years, the first two years in an easily accessible place" — certain records, not all of them. HIPAA's 45 CFR 164.316(b)(2)(i) sets a six-year floor too, but only on Security Rule documentation: policies, procedures and records of required security actions, kept "for 6 years from the date of its creation or the date when it last was in effect, whichever is later." A disposal record can itself fall in that class.

HIPAA does not set a retention period for medical records. HHS states it directly: "the HIPAA Privacy Rule does not include medical record retention requirements. Rather, State laws generally govern how long medical records are to be retained" (HHS FAQ 580). It does require safeguards for PHI "including through disposal."

Ceilings say keep no longer than necessary. GDPR Article 5(1)(e) requires personal data to be "kept in a form which permits identification of data subjects for no longer than is necessary for the purposes". When a floor and a ceiling cover the same record, the floor is itself a purpose: a legal retention duty makes identifiable storage necessary under Article 5(1)(e) until that duty ends. Once the longest applicable floor has ended and no other purpose applies, storage limitation makes disposal due. Article 17(3)(b) does a narrower job: it stops a data subject forcing erasure while processing is still necessary "for compliance with a legal obligation" under Union or Member State law.

A legal or litigation hold is a floor with no end date. A hold on ten records in a period blocks in-place deletion of those ten. It blocks key destruction for the entire period, because one key version covers every record written under it. None of these periods are advice: a schedule is set with counsel, per jurisdiction and sector (India's log-retention direction has its own analysis). Engineering makes the schedule happen.

Option A: delete in place, store by store

Every store has a native disposal mechanism, and nearly every one has a state that looks like deletion and is not.

Versioned object storage

On a versioned S3 bucket, a lifecycle Expiration on current versions removes nothing. Amazon's lifecycle examples say S3 acts "by adding a delete marker for each of the current object versions", which makes the data noncurrent. Only NoncurrentVersionExpiration "removes only the noncurrent versions." A rule cannot carry both Days and ExpiredObjectDeleteMarker; with Days set, S3 cleans up the expired markers itself, so the marker-only rule is for prefixes where the application deletes objects. The trap is the noncurrent clock: NoncurrentDays counts from when a version becomes noncurrent, not from creation, and an overwrite does that immediately, so a small value deletes an overwritten invoice early. On a prefix with a floor, set it to the full floor and record that a version can outlive the floor by up to one more period:

{
  "Rules": [
    {
      "ID": "invoices-scheduled-disposal",
      "Filter": { "Prefix": "invoices/" },
      "Status": "Enabled",
      "Expiration": { "Days": 2192 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 2192 }
    },
    {
      "ID": "exports-deleted-by-the-application",
      "Filter": { "Prefix": "exports/" },
      "Status": "Enabled",
      "Expiration": { "ExpiredObjectDeleteMarker": true },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 1 }
    }
  ]
}

On Ceph RGW, Garage or SeaweedFS, list versions after a test expiry and confirm noncurrent ones actually disappeared.

Loki, Kafka and Prometheus

Loki retention runs in the compactor only when retention_enabled is true; marked chunks are deleted after retention_delete_delay. retention_period applies to every stream, so set it no shorter than the strictest floor and use retention_stream to shorten the classes the schedule allows.

compactor:
  working_directory: /var/loki/compactor
  retention_enabled: true # without it the compactor only compacts
  retention_delete_delay: 2h # marked chunks are deleted after this delay
  delete_request_store: s3 # required once retention is enabled

limits_config:
  retention_period: 52596h # no shorter than the strictest floor (~six years)
  retention_stream:
    # Shorten classes the schedule allows; labelled at the shipper.
    - selector: '{record_class="debug"}'
      priority: 2
      period: 744h

schema_config:
  configs:
    - from: "2026-01-01"
      store: tsdb
      object_store: s3
      schema: v13
      index: { prefix: index_, period: 24h } # retention needs a 24h index period

With retention enabled, Loki's deletion API defaults to filter-and-delete, which removes lines; filter-only merely hides them. See the GDPR Article 32 walkthrough. Kafka's `retention.ms` discards "old log segments", and retention "is always done a file at a time", so segment.bytes sets the granularity: a record lives until its whole segment is eligible. retention.ms also rolls the active segment once the retention condition is met. With tiered storage, broker-local copies follow local.retention.* (default -2, meaning use retention.*) while retention.* governs the total. Prometheus retention defaults to 15d when neither time nor size retention is set, blocks "must be fully expired before they are removed", and cleanup "may take up to two hours." Compaction merges blocks, so disposal granularity is coarser than the flag suggests.

PostgreSQL

A bulk DELETE is the wrong tool. The routine vacuuming docs say a DELETE "does not immediately remove the old version of the row"; the space waits for VACUUM. Partition by the schedule's unit instead, and dispose by partition, which the partitioning docs call "far faster than a bulk operation" and which "entirely avoid the VACUUM overhead". On CloudNativePG the same SQL applies; the archive it ships to object storage has its own retention.

-- Monthly range partitions: one partition = one disposal unit.
CREATE TABLE payment_audit (
    recorded_at timestamptz NOT NULL,
    account_ref text        NOT NULL,
    payload     jsonb       NOT NULL
) PARTITION BY RANGE (recorded_at);

-- No DEFAULT partition: DETACH ... CONCURRENTLY is refused when one exists.
CREATE TABLE payment_audit_2020_09 PARTITION OF payment_audit
    FOR VALUES FROM ('2020-09-01') TO ('2020-10-01');
CREATE TABLE payment_audit_2026_10 PARTITION OF payment_audit
    FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');

-- Retention job, autocommit: CONCURRENTLY cannot run in a transaction block.
ALTER TABLE payment_audit DETACH PARTITION payment_audit_2020_09 CONCURRENTLY;
DROP TABLE payment_audit_2020_09;

Backups

restic forget removes snapshots, but "the data that was referenced by files in this snapshot is still stored in the repository" until prune runs. And --keep-within is measured from "the latest (most recent) snapshot", not from today: if backups stop, the window stays anchored to the last snapshot, so snapshots within it are kept indefinitely while older ones are still removed by the next forget. Run forget --prune, and alert on the newest snapshot's age.

Option B: destroy the key for the period

The alternative makes every copy unreadable at once. Encrypt each record at write time under its write window's key version, and destroy that version when the window's newest record passes its floor. Ciphertext in replicas, WAL and locked backups becomes unreadable. It is per-subject crypto-shredding (covered here) applied to time instead of a person.

Encryption must happen on the write path, before any copy exists. With OpenBao transit, the writer calls POST /transit/datakey/plaintext/:name, base64-decodes the returned data key and encrypts the record in-process with AES-256-GCM under those bytes — application code, not a shell step — then discards the plaintext key and stores the wrapped key (vault:vN:…) beside the ciphertext. The N is the key version, which is the write window. Store-native encryption — restic repository keys, server-side encryption, LUKS — is not the period key and is not reached by a trim.

Rotation happens at the start of each write window. Destruction of a version happens only when that window ended at least the retention period ago, so several versions coexist: with yearly windows and a six-year floor, seven are live. To destroy one, raise min_decryption_version, which stops older ciphertext from being decrypted, and min_encryption_version; then call trim. OpenBao's docs are explicit that min_available_version "can at most be equal to the lesser of" the two minimums, is "not allowed to be set when either" is zero, and that once trimmed, "previous versions of the key cannot be recovered."

#!/usr/bin/env bash
# One transit key per record class; one key version per write window.
# rotate runs at the START of each window. Record encryption is application code.
# Needs GNU date (date -d).
set -euo pipefail

: "${BAO_ADDR:?set BAO_ADDR to the OpenBao API address}"
KEY="records-payment-audit"               # never shared between record classes
WRITE_WINDOW="1 year"                     # how long one key version encrypts
RETENTION="6 years"                       # floor from the schedule, set with counsel
HOLD_DIR="/etc/retention/holds/${KEY}"    # v<N> file = legal hold on that window

cmd="${1:?usage: init | rotate | retire <version> <window-end>}"

case "$cmd" in
  init)
    # Irreversible once true; either permits key copies no trim reaches.
    bao write "transit/keys/${KEY}" type=aes256-gcm96 \
      exportable=false allow_plaintext_backup=false
    ;;

  rotate)
    bao write -f "transit/keys/${KEY}/rotate"
    echo "rotated ${KEY}: a new ${WRITE_WINDOW} write window starts now"
    ;;

  retire)
    version="${2:?key version whose write window has closed}"
    window_end="${3:?first day after that write window, YYYY-MM-DD}"
    if [[ -e "${HOLD_DIR}/v${version}" ]]; then
      echo "legal hold on ${KEY} v${version}: refusing to destroy it" >&2
      exit 1
    fi
    # The window's NEWEST record must have passed the floor, not its oldest.
    expires="$(date -u -d "${window_end} +${RETENTION}" +%s)"
    if (($(date -u +%s) < expires)); then
      echo "v${version} still holds records inside ${RETENTION}" >&2
      exit 1
    fi
    info="$(bao read -format=json "transit/keys/${KEY}")"
    min_dec="$(jq -r '.data.min_decryption_version' <<<"$info")"
    latest="$(jq -r '.data.keys | keys | map(tonumber) | max' <<<"$info")"
    # Oldest first: trim deletes every version below min_available_version.
    if ((version != min_dec)); then
      echo "retire v${min_dec} before v${version}" >&2
      exit 1
    fi
    if ((version >= latest)); then
      echo "v${version} is still the write key; rotate first" >&2
      exit 1
    fi
    next=$((version + 1))
    # Raise BOTH minimums first: trim may not exceed the lesser of the two.
    bao write "transit/keys/${KEY}/config" \
      min_decryption_version="$next" min_encryption_version="$latest"
    bao write "transit/keys/${KEY}/trim" min_available_version="$next"
    ;;

  *)
    echo "unknown command: ${cmd}" >&2
    exit 2
    ;;
esac

Classes with different schedules cannot share a version, hence one key per class. Versions end oldest-first, because trim deletes everything below min_available_version. Deleting a whole key is a separate act that requires deletion_allowed. Data written before the scheme existed is outside it. Still on Vault? The OpenBao migration gates come first.

NIST SP 800-88 Revision 2, published September 2025, describes cryptographic erase as "based on the sanitization of keys used to encrypt data". It is a media-sanitization technique, not a legal disposal standard; whether key destruction satisfies a given erasure duty is a question for counsel.

Where each option fails

Option A fails on copies it cannot touch. In S3 Object Lock compliance mode, a version's "retention period can't be shortened", and "the only way to delete an object under the compliance mode before its retention date expires is to delete the associated AWS account." Lifecycle still runs — it keeps "placing delete markers" — but "a locked version of an object cannot be deleted by a S3 Lifecycle expiration policy" (managing Object Lock). A dashboard counting markers reports success while every locked version is intact. The immutable-backups design that defends against ransomware is exactly the copy that outlives the schedule, so derive its lock period from the schedule.

The UK ICO accepts backup lag for erasure, provided the data is put "beyond use" and "held on your systems until it is replaced in line with an established schedule" (ICO, right to erasure). The page notes it is under review after the Data (Use and Access) Act. Either way, the backup clock must be written down.

Option B fails on copies it never covered. 800-88r2, citing ISO/IEC 27040, makes it a precondition that "no sensitive data has previously been stored on the ISM in plaintext form", because CE "can only sanitize keys related to encrypted data." A debug log or CSV export holding the record is outside the trim. The same standard requires that "all copies of the target cryptographic keys must be able to be sanitized". Key copies hide in four places:

• Exported or backed-up key material — once exportable or allow_plaintext_backup is set on a transit key it cannot be turned off, and any export or plaintext backup taken under it is a copy outside OpenBao that no trim reaches.
• OpenBao Raft snapshots — bao operator raft snapshot save produces a snapshot that can restore "OpenBao to the point in time when a snapshot was taken." It follows that a snapshot taken before a trim still holds the trimmed versions, so destruction is complete only when the last pre-trim snapshot expires — a date the disposal record should state.
• Unseal-key escrow and HSM backups — whatever restores the key store restores its keys.
• Legal holds — not a copy, but a hold on any record in the window blocks destruction.

Click to zoom

The verdict is layered. Option A for stores that delete precisely: the database by partition, Loki by stream, Kafka by segment, unlocked buckets. Option B for what Option A cannot reach: backups, locked media, archives. Write down where neither applies yet.

Proving disposal happened

800-88r2 says a certificate of sanitization "should be completed for each ISM that has been sanitized", and that validation results in "a decision to either approve the sanitization as being effective or reject it". That certificate is per storage medium; adapted to records, it becomes one entry per run: class, schedule reference, each store and mechanism, before-and-after state, what remains until when, and a validation decision.

{
  "record_class": "payment-audit",
  "schedule_reference": "retention-schedule.md section 4.2, rev 7",
  "executed_at": "2026-10-16T02:04:31Z",
  "operator": "retention-job, approved by records-owner@example.com",
  "legal_hold_checked": { "hold_present": false, "checked_at": "2026-10-16T02:00:02Z" },
  "actions": [
    {
      "store": "postgres://pg-primary/ledger.payment_audit",
      "mechanism": "detach-and-drop-partition",
      "unit": "payment_audit_2020_09",
      "period": { "from": "2020-09-01", "to": "2020-10-01" },
      "floor_ended": "2026-10-01",
      "before": { "partition_exists": true, "row_count": 418220 },
      "after": { "partition_exists": false }
    },
    {
      "store": "openbao://transit/keys/records-payment-audit",
      "mechanism": "key-version-trim",
      "unit": "key version 1",
      "period": { "from": "2019-01-01", "to": "2020-01-01" },
      "floor_ended": "2026-01-01",
      "before": { "min_available_version": 1 },
      "after": { "min_decryption_version": 2, "min_available_version": 2 }
    }
  ],
  "residual_copies": [
    {
      "store": "pg-wal-archive and base backups",
      "state": "rows for 2020-09 remain in archived WAL until archive expiry",
      "expires_by": "2026-11-15"
    },
    {
      "store": "openbao raft snapshots",
      "state": "pre-run snapshots still hold key version 1",
      "expires_by": "2026-11-15"
    }
  ],
  "validation": { "decision": "approve", "method": "post-run state checks" },
  "previous_record_sha256": "82496ebfaac12ad6a155e7f10d48b54ed2b41567cb6288fcebe88a72df0acf9e"
}

residual_copies turns WAL and snapshot lag from a hidden failure into a dated, stated fact. The timestamped hold check shows no hold existed at run time. Chaining each record to the previous hash makes a missing run visible.

SEC 17a-4(f)(2)(i) lets an electronic system qualify either with "a complete time-stamped audit trail" of "all modifications to and deletions of the record", sufficient to re-create it, or by preserving records "exclusively in a non-rewriteable, non-erasable format." Disposal records belong in an append-only store on their own retention, ideally as long as the longest floor they evidence; under HIPAA they may count as security documentation in their own right. An auditor asks for the schedule, the runs, and one sample traced end to end: a class and a period, gone from every node in the diagram or listed as residual.

Exit ramps and the long game

Disposal must survive a change of vendor. Lifecycle rules, stream rules and key hierarchies do not migrate themselves: they are configuration, not properties of the data. Keep the schedule as data in version control and generate each store's configuration from it, so a migration re-applies it. Owning the key store makes Option B your decision; with a provider's KMS, destruction is whatever their API and backups allow.

Object Lock compliance mode is a one-way door per object version, so set lock periods from the schedule on day one. Related deletion trails appear in CCPA deletion architecture and in what a data processing agreement promises at the end of processing.

Long-lived ciphertext carries one more clock. 800-88r2 warns that with CE "the data still resides in the ISM as ciphertext", and that if "cryptographic weaknesses" emerge or "computational capabilities (e.g., quantum computing) made it feasible to recover keys," it can become recoverable. Key destruction buys unreadability now; in-place deletion removes the ciphertext too. A policy built to last uses both, and records which ran where.