Cyber Security Incident Response Plan: The Evidence Trail

Search for an incident response plan and you get templates: a page of blank fields for the incident commander, the escalation path and the communications lead. Those fields answer who. They do not answer what happened, when, and how anyone knows — which is what an auditor, a regulator or opposing counsel asks six months later. This article treats the evidence trail as a design requirement of the response tooling, on infrastructure you operate, with Kubernetes containment commands that record themselves. The scenarios are illustrative; the commands and the standards are real.

The plan that fails the six-month test

Picture the review. An auditor picks an incident from last spring and asks four questions. Which pod was isolated, by whom, and at what time? What did the responder run inside it? Is the memory image in the case file the one that was taken from that pod, unaltered? And whose clock produced the timestamps in the timeline?

A filled-in template cannot answer any of them. The answers live in artefacts that either were produced during the response or do not exist: an API server audit entry, a terminal transcript, a SHA-256 written at collection time, an immutable object with a version ID, a clock offset noted when the session began. A post-incident report written a week later from chat scrollback and recollection is a narrative, not evidence — and SP 800-86 is blunt that electronic records can be altered, so organisations should be prepared to demonstrate their integrity.

So the useful question for the plan is not who is on the call. It is: which tool emits which record, where does that record land, and who can change it afterwards?

What an auditor reconstructs: the evidence model

NIST SP 800-61 Revision 3, published in April 2025, superseded the 2012 Rev. 2 and is structured as a CSF 2.0 Community Profile for incident response. Two of its subcategories are, in effect, the specification for an evidence trail:

• RS.AN-06: actions performed during an investigation are recorded, and the records' integrity and provenance are preserved. Its note lists the means — a paper logbook, audio/video recordings, or automatic session monitoring and logging, as permitted by the organisation's plan and policy.
• RS.AN-07: incident data and metadata are collected, and their integrity and provenance preserved. The publication concedes that formal chain-of-custody handling might not be performed for every incident, since most malware incidents will not result in prosecution — but collected incident data is still considered evidence, retained under the organisation's evidence preservation procedures and data retention policies.

RFC 3227 (BCP 55, February 2002) said most of this two decades earlier and is more concrete. Keep detailed notes with dates and times. Generate an automatic transcript where possible — it names the Unix script program — and do not write that transcript to media that is part of the evidence. Note the difference between the system clock and UTC, and say for every timestamp whether it is UTC or local. Collect first and analyse later, proceeding from volatile to less volatile: process tables and memory before disk.

Reduce those to properties a design can be checked against: actions recorded (who did what, attributable), integrity (a digest taken at collection, verifiable later by recomputing and comparing, which is how SP 800-86 describes verification), provenance and custody (RFC 3227 §4.1: where, when and by whom evidence was collected and handled, who held it, how it was stored, and every transfer), and a stated time base. Each property needs a component that produces it without a human remembering to.

An evidence pipeline you operate: IRIS, Velociraptor, the audit log and WORM storage

Every component below is open source and self-hostable, and each owns exactly one evidence property. That one-to-one mapping is the point: when the trail breaks, it is obvious which component did not do its job.

• Case record — [DFIR-IRIS](https://raw.githubusercontent.com/dfir-iris/iris-web/master/README.md). A web collaborative platform for incident responders to share technical details during investigations, LGPL-3.0. Its documentation describes evidence dropped into a case being hashed with SHA-256 for chain of custody, and case- and group-level RBAC with an audit trail behind every action. At the time of writing the latest stable release is v2.4.29 while the README advertises v3.0.0-beta.1, and the current docs appear to track v3 — verify each feature against the version you deploy.
• Endpoint collection — [Velociraptor](https://docs.velociraptor.app/docs/overview/). An open-source platform for endpoint monitoring, digital forensics and cyber response, AGPL-3.0. At the time of writing its docs ask users to update to 0.77.3 to address multiple CVEs; a forensic tool on every endpoint is a privileged agent and gets patched like one.
• Who touched the workload — the Kubernetes API server audit log. Every label, policy and debug action a responder takes through kubectl becomes an attributable event, if the audit policy asks for it. Shipping it to a SIEM is covered in Kubernetes audit logs into Wazuh.
• What was typed — script(1) transcripts. The audit log shows that an exec happened; it does not show what scrolled past in the session.
• Immutability — Ceph RGW with S3 Object Lock. The store that makes a hash meaningful, because nobody can quietly swap the object it describes.
• One clock — chrony on every host involved, offset recorded per session. The groundwork is in trusted time with chrony and PTP.

Click to zoom

Note what the case record is not: it is not the evidence store. IRIS holds the timeline, the indicators and references — hash, bucket key, version ID — while the bytes live in a store that refuses deletion. If the case tool is replaced, the evidence does not move.

Containment on Kubernetes that records itself

Start with the audit policy, because it decides whether the containment steps are attributable at all. This policy records exec, attach, port-forward, ephemeral containers and node proxy calls at RequestResponse, and pod and NetworkPolicy changes at Request:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: RequestResponse
    resources:
      - group: ""
        resources:
          - pods/exec
          - pods/attach
          - pods/portforward
          - pods/ephemeralcontainers
          - nodes/proxy
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["pods"]
      - group: "networking.k8s.io"
        resources: ["networkpolicies"]
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  - level: Metadata

This is not a cluster object, and kubectl apply will not accept it. It is the contents of the file the API server reads at startup through kube-apiserver --audit-policy-file, and the Kubernetes auditing docs are explicit that without that flag no events are logged at all. The policy also needs a backend to write to: --audit-log-path for the log backend, or a webhook configuration that ships events off the control plane. On a cluster that already has a policy, merge these rules into the live file rather than replacing it, because rules are matched in order and the existing file may carry deliberate None rules for noisy system traffic. Then confirm it took effect before an incident needs it: run a harmless kubectl label against a test pod and find the event in the log. If the flag or the backend is missing, the label, debug and NetworkPolicy calls below happen without being recorded, and the containment is no longer attributable.

Containment is then one label. A deny-all policy selects anything carrying the quarantine label, so isolating a pod is a single audited patch:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: incident-quarantine
  namespace: payments
spec:
  podSelector:
    matchLabels:
      incident.stribog.example/quarantine: "true"
  policyTypes:
    - Ingress
    - Egress

Three preconditions, stated plainly because each one silently turns containment into theatre. The CNI must enforce NetworkPolicy; some do not. The NetworkPolicy docs say policies are additive — the allowed traffic is the union of every policy selecting the pod — so if another policy already allows traffic to this pod by its app label, the quarantine policy cannot subtract it. Do not strip the pod's existing labels to escape them either: dropping a label the ReplicaSet selects on orphans the pod, and the controller starts a replacement from its template — without the quarantine label, back on the network. Narrow or remove the conflicting allow policy instead. Whether existing connections are cut is implementation-defined. And even as the only policy selecting the pod, it is not total isolation: the same docs say an ingress-isolated pod still accepts connections from its own node, so kubelet probes from the node still reach it, and NetworkPolicy cannot block localhost traffic in either direction.

Next, capture memory before anything kills the pod — RFC 3227's order of volatility. The kubelet Checkpoint API, beta since Kubernetes v1.30 and enabled by default, creates a stateful copy of a running container via POST /checkpoint/{namespace}/{pod}/{container}. It is CRIU-based and, per the alpha announcement in v1.25, the container does not know it was checkpointed. The archive is a tar written by default under /var/lib/kubelet/checkpoints on the node, its contents depend on the container runtime, and a runtime without checkpointing returns the same 500 as any other runtime failure, distinguishable only by the response body — test on your nodes before an incident does. Checkpoint time tracks memory footprint, so a large container can hit the default CRI timeout; the API takes an optional timeout query parameter.

There is no kubectl subcommand for it, and the API server's node proxy cannot make the call. The kubelet authorization docs map /checkpoint/* to the nodes/checkpoint subresource, and the API server's own kubelet client role, system:kubelet-api-admin, does not include it — so cluster-admin through the API server is refused. POST directly to the kubelet on port 10250 with a client certificate whose user holds create on nodes/checkpoint. That subresource is cluster-scoped, so it takes a ClusterRole and ClusterRoleBinding; a namespaced Role will not authorize it. Status codes diagnose it: 401, credentials rejected; 403, webhook authorization denied; 404, feature gate off or no such pod or container; 500, runtime failure. A direct kubelet call is not an API server audit event, so attribution for this step comes from the ir-session transcript the script runs inside.

#!/usr/bin/env bash
# Contain, checkpoint, hash, verify, lock. Run from the evidence host inside ir-session.
set -euo pipefail

case_id="IR-2026-0142"
ns="payments" pod="api-7c9d8" ctr="api"
bucket="ir-evidence"
rgw="https://rgw.evidence.example"
retain_until="2033-11-09T00:00:00Z"   # from your evidence retention policy

# --overwrite: a rerun after a failed checkpoint must not stop here.
kubectl -n "$ns" label pod "$pod" --overwrite \
  incident.stribog.example/quarantine=true

# Assumes the Kubernetes node name is also an SSH destination from this host.
node="$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}')"
# One address per line; dual-stack nodes report more than one InternalIP.
addrs="$(kubectl get node "$node" -o jsonpath=\
'{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}')"
node_ip="$(awk '!/:/ { print; exit }' <<<"$addrs")"   # prefer IPv4
if [ -z "$node_ip" ]; then
  node_ip="$(awk 'NF { print; exit }' <<<"$addrs")"
  [ -n "$node_ip" ] || { echo "no InternalIP on ${node}" >&2; exit 1; }
  node_ip="[${node_ip}]"   # an IPv6 literal in a URL needs brackets
fi

# Straight to the kubelet: the API server node proxy cannot do this call.
# No curl -f: on failure the response body is the diagnosis.
# TLS needs node_ip as a SAN on the kubelet serving cert; else use a DNS name.
resp="${case_id}-checkpoint-response.json"
status="$(curl -sS -X POST -o "$resp" -w '%{http_code}' \
  --cacert kubelet-ca.crt --cert ir-checkpoint.crt --key ir-checkpoint.key \
  "https://${node_ip}:10250/checkpoint/${ns}/${pod}/${ctr}")"
if [ "$status" != "200" ]; then
  echo "checkpoint failed: HTTP ${status}" >&2
  cat "$resp" >&2
  exit 1
fi

# 200 body: {"items":["<absolute path of the tar>"]}. The runtime writes it
# root-only: read it as root over ssh. Never chmod it or scp it.
archive="$(jq -er '.items[0]' "$resp")"
file="$(basename "$archive")"
node_sum="$(ssh "$node" sudo -n sha256sum -- "$archive" | cut -d ' ' -f 1)"
ssh "$node" sudo -n cat -- "$archive" > "$file"
echo "${node_sum}  ${file}" | sha256sum -c -   # transfer verified
sha256sum "$file" > "${file}.sha256"

# AWS CLI 2.23+ sends flexible checksums some Ceph RGW versions reject.
export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
for f in "$file" "${file}.sha256"; do
  aws --endpoint-url "$rgw" s3api put-object \
    --bucket "$bucket" --key "${case_id}/${f}" --body "$f" \
    --content-md5 "$(openssl md5 -binary "$f" | base64)" \
    --object-lock-mode COMPLIANCE \
    --object-lock-retain-until-date "$retain_until" \
    --query VersionId --output text
done

# No "kubectl delete pod": its controller would recreate it unlabelled.
# Scaling the owner to zero is a separate, recorded decision.

Two details in that script exist because the obvious version fails exactly when it matters. First, the retry. kubectl label refuses to change a label that is already present unless --overwrite is passed. A checkpoint that returns 500 or runs past the runtime timeout leaves the quarantine label in place, and the responder's natural next move is to run the script again. Without --overwrite, that rerun stops at the label under set -e and never sends the checkpoint request, which is the one step that preserves memory. Second, the address. A JSONPath filter returns every match, joined with spaces, and a dual-stack node reports more than one InternalIP. Interpolating that straight into a URL gives curl something that is not a host. The script lists the addresses one per line, prefers IPv4, and wraps an IPv6 literal in brackets. Both failures would be recorded faithfully by the transcript, but a faithful record of a lost memory image is still a lost memory image.

Debugging follows the same rule: do it through the API so it is logged. kubectl debug adds an ephemeral container, which at the API level is a patch/update on the pods/ephemeralcontainers subresource. The debugging docs point out that this subresource is deliberately not part of the default admin and edit ClusterRoles, so grant responders both verbs on it in a dedicated, audited role — a role granting only update fails when kubectl debug patches. Where the original must stay untouched, kubectl debug --copy-to debugs a copy of the pod instead.

Chain of custody as files, not a form

The responder's shell is itself an evidence source. Run every session through a wrapper that pins the time base, records the clock offset, and writes the transcript to the evidence host — never the affected node, per RFC 3227. `script` records the raw terminal stream and, with a timing log, lets scriptreplay replay the session later:

#!/usr/bin/env bash
# ir-session: record a responder shell on the evidence host, then hash it.
set -euo pipefail

case_id="${1:?usage: ir-session CASE-ID}"
dir="/srv/evidence/${case_id}/sessions"
export TZ=UTC
base="${dir}/$(date -u +%Y%m%dT%H%M%SZ)-${USER}"
mkdir -p "$dir"

{
  echo "case=${case_id} responder=${USER} host=$(hostname) start=$(date -u +%FT%TZ)"
  chronyc -n tracking
} > "${base}.clock.txt"

script --quiet --log-out "${base}.typescript" --log-timing "${base}.timing"

sha256sum "${base}.clock.txt" "${base}.typescript" "${base}.timing" > "${base}.sha256"
echo "session sealed: ${base}.sha256"

Then one custody record per artefact, created at collection, carrying the RFC 3227 §4.1 fields and appended to — never rewritten — on every transfer. It is stored next to the artefact under the same lock:

{
  "artefact_id": "IR-2026-0142-A003",
  "description": "kubelet checkpoint of container api, pod payments/api-7c9d8",
  "sha256": "9f2c4e1ab07d53c8e6f1a2b4c9d0e7f3a5b6c8d9e0f1a2b3c4d5e6f7a8b9c0d1",
  "source": { "cluster": "prod-eu-1", "node": "worker-07", "namespace": "payments", "pod": "api-7c9d8" },
  "collected_by": "responder-a",
  "collected_at": "2026-11-09T09:41:07Z",
  "time_base": "UTC; chrony offset in sessions/20261109T093812Z-responder-a.clock.txt",
  "method": "kubelet POST /checkpoint on :10250, client cert ir-checkpoint",
  "storage": {
    "bucket": "ir-evidence",
    "key": "IR-2026-0142/checkpoint-api-7c9d8_payments-api.tar",
    "version_id": "example-version-id",
    "object_lock_mode": "COMPLIANCE",
    "retain_until": "2033-11-09T00:00:00Z"
  },
  "transfers": [
    {
      "at": "2026-11-09T09:43:55Z",
      "from": "worker-07:/var/lib/kubelet/checkpoints",
      "to": "evidence-host:/srv/evidence/IR-2026-0142",
      "by": "responder-a",
      "sha256_verified": true
    }
  ]
}

On the storage side, Ceph RADOS Gateway implements the S3 Object Lock API: a bucket-level configuration applied by default to every new object, plus per-object retention in GOVERNANCE or COMPLIANCE mode. The bucket must be created with Object Lock enabled (--object-lock-enabled-for-bucket); Ceph refuses a lock configuration on a bucket created without it. In the S3 model as AWS defines it, compliance mode means no user, including root, can overwrite or delete a protected version or shorten its retention, while governance mode can be bypassed with special permissions. The same pattern backs immutable backups.

Failure modes: where evidence trails break

• Analysis on the only copy. Opening a memory image in a tool that writes caches beside it changes the evidence. Collect, hash, lock, then analyse a copy whose digest you verified.
• Unsynced clocks. The audit log, the SIEM and a node's journal disagree by seconds, and the timeline cannot order events. Record the offset per session; don't infer it afterwards.
• The pod was deleted first. A controller restart or an eager responder destroys the memory state. Checkpoint before delete is a plan step, not a habit.
• Transcripts on the compromised host. Anything written to the affected node is as trustworthy as the node. Sessions run from the evidence host.
• Exec that bypasses the audit log. SSH straight to a node and crictl exec leave no API server event. Route responder access through the API, and alert on node logins — runtime detection from Falco or Tetragon catches what the audit log cannot see.
• IR records readable by everyone. A shared channel full of exploited vulnerabilities and suspect accounts is its own incident, which the next section deals with.

Security and compliance: who can read the record, and whose clock is running

RS.AN-06 is explicit that incident response records are sensitive: they can contain data on exploited vulnerabilities, recent breaches and users who may have acted inappropriately, so their confidentiality and integrity should be safeguarded and access limited to authorised personnel. Checkpoints raise the stakes. The Checkpoint API page warns that a checkpoint typically contains all memory pages of all processes in the container — including private data and possibly encryption keys — readable by whoever holds the archive. Treat the evidence bucket as a secrets store: separate credentials, read access by case, access itself audited.

The trail also decides whether you can meet a disclosure clock with facts. For US public companies, the SEC's 2023 adopting release explains that investors are best served knowing within four business days after the materiality determination that the incident occurred and why management concluded it was material; under Item 1.05(c), the registrant may delay that disclosure only if the Attorney General determines that it poses a substantial risk to national security or public safety and notifies the Commission of that determination in writing. India's six-hour window is set by the CERT-In directions. The EU runs a different, staged reporting clock under NIS2, covered in NIS2 on infrastructure you run. Each clock is easier to meet, and to defend afterwards, when the timeline is already assembled from records. For SOC 2, the same records double as operating-effectiveness evidence, as in SOC 2 on self-hosted Kubernetes.

Exit ramp: evidence in formats that outlive the tools

Case-management tools change; evidence retention outlasts them. Keep everything that matters in formats readable without the tool that produced it: raw-text typescripts and timing files, tar checkpoints, .sha256 files any sha256sum -c can check, JSON custody records, and audit events as JSON lines. Store them behind the S3 API, which Ceph, other object stores and the hyperscalers all speak. IRIS then holds references, not the only copy, and migrating case tools leaves past incidents intact. Lock-in in incident response is worse than inconvenience: an orphaned case is one you can no longer prove.

The long game: testifying years later

RFC 3227's warning about testifying years later is the real acceptance test for the plan. Run it as a drill: pick a closed incident, hand the stored trail to someone who did not work it, and ask them to reconstruct the timeline, verify every digest, and name who held each artefact when. Every point where they have to ask a person instead of reading a record is the next change to the plan.

Repeat it yearly against older incidents, because that is where formats rot, keys expire and buckets get reorganised. A plan built this way does not depend on anyone's memory surviving — only on records you own, in formats you can still read, on storage that refused to forget.