Self-Hosted Chaos Engineering as Audit-Grade Evidence

Most chaos engineering programmes die the same way. A team runs an enthusiastic first game day, kills some pods, discovers something alarming about a retry loop, fixes it, and writes a wiki page. Six months later nobody can say whether the fix still holds, whether the experiment was re-run, or what the system did while the fault was live. The practice was real. The record is a memory.

That gap has stopped being merely hygiene. Operational-resilience regimes now ask regulated organisations to demonstrate — on a schedule, with documentation — that they have tested failure of systems supporting critical functions. The question is no longer 'do you do chaos engineering?' but 'show me the test, the scope, the result, and the remediation.' A wiki page is not an answer.

This article is about closing that gap on infrastructure you own: how to structure fault-injection experiments with LitmusChaos and Chaos Mesh so that the by-product of running them is admissible evidence rather than an anecdote. Where disaster recovery proves you can restore after a failure, this proves you survive one — proactively, repeatably, and with a chain of custody.

The Game Day Produces a Memory, Not an Artifact

The classic game day is a scheduled, human-driven exercise: engineers gather, someone injects a fault, everyone watches dashboards, and the group discusses what happened. Valuable as a learning ritual and as practice for the human side of incident response, it is still structurally incapable of producing evidence, for three reasons.

First, the pass/fail criterion lives in the heads of the participants. If 'the system stayed healthy' was never written down as a machine-checkable assertion before the fault, then afterwards it is a judgement call, and judgement calls do not survive an auditor asking how the conclusion was reached. Second, the observability record is unlabelled: the traces and metrics from the fault window are mixed into the same streams as ordinary production traffic, with nothing marking which requests were affected by a deliberate experiment. Third, nothing is retained in a tamper-evident form — the wiki page can be edited, and the Grafana dashboard has long since rolled past its retention window.

The fix is not to stop running game days. It is to demote the human exercise to one output of a pipeline whose primary output is a signed record. The experiment definition becomes a reviewed artifact in Git. The steady-state hypothesis becomes probes. The verdict becomes a signed document with a retention policy. The humans still gather and still learn — but the evidence no longer depends on anyone remembering to write it down.

What the Regulation Actually Asks For

It is worth being precise here, because vendors routinely overstate what the rules say. Under the EU's Digital Operational Resilience Act (Regulation (EU) 2022/2554), Article 24 requires financial entities to establish and maintain a digital operational resilience testing programme, to ensure tests are undertaken by independent parties, whether internal or external (Article 24(4)), and to test all ICT systems and applications supporting critical or important functions at least yearly (Article 24(6)). Article 25(1) enumerates the test types the programme must provide for — among them vulnerability assessments, gap analyses, source-code reviews, performance testing, end-to-end testing, penetration testing, and, most relevant here, scenario-based tests.

Note what this is not. The advanced threat-led penetration testing regime in Articles 26 and 27 — the intelligence-driven red-team exercise conducted at least every three years — applies only to designated entities whose failure would have systemic effects, and chaos engineering is not a substitute for it. Fault injection is a scenario-based resilience test, not an adversary simulation. Conflating the two is the fastest way to fail a review.

The useful insight is that a well-structured chaos experiment maps almost exactly onto what a scenario-based test is supposed to be: a defined scenario, a defined scope, a predicted outcome, an observed outcome, and a remediation path where the two diverge. The regulation does not name any tool, and it certainly does not require a SaaS platform. It requires that the test happened, that it covered the right systems, and that you can show the result. Everything in this article is about making that showable. The same discipline satisfies the equivalent expectations under NIS2 and the wider European compliance surface. A SOC 2 Type 2 examination asks the same question of a different audience — operating effectiveness over a period, not design intent at a point in time — and a signed, retained chaos verdict is exactly the kind of evidence that satisfies it.

Choosing the Engine: Litmus and Chaos Mesh

Both leading open-source options are CNCF Incubating projects — neither has graduated, which is worth knowing before a maturity argument to a risk committee. Chaos Mesh originated at PingCAP, entered the CNCF in July 2020, and moved to the Incubator in February 2022. LitmusChaos originated at MayaData, was donated to the CNCF in 2020, and reached Incubating status in 2022 as well.

Click to zoom

The practical difference is one of centre of gravity rather than capability. Chaos Mesh is the richer fault-injection engine: a broad set of granular CRDs covering pod, network, IO, stress, time, kernel, DNS, and HTTP faults, injected by a privileged per-node chaos-daemon, with a clean built-in dashboard. If your experiments need precise, low-level failure modes — packet corruption at a specific rate, filesystem latency on a particular mount, clock skew inside one container — Chaos Mesh gets there with less effort.

Litmus centres on the experiment lifecycle instead: a ChaosHub of versioned, reusable experiment definitions, a ChaosCenter control plane that schedules across clusters, Argo-based workflows for sequencing multi-step scenarios, and a first-class probe model that produces a ChaosResult with a resilience score. For an evidence programme running the same catalogue of scenarios quarterly across many clusters, that lifecycle model is the more valuable half.

• Pick Chaos Mesh when fault fidelity dominates — you need specific kernel, IO, or network failure semantics, and you are running a small number of clusters.
• Pick Litmus when programme mechanics dominate — a reusable catalogue, multi-cluster scheduling, and GitOps-driven chaos pipelines matter more than exotic fault types.
• Both are CNCF Incubating, both are Apache-2.0, and both run entirely on infrastructure you own with no telemetry egress. Neither decision is a lock-in event.
• Neither produces audit evidence on its own. Both record results in Kubernetes objects with ordinary etcd lifetimes — useful operationally, insufficient as a retained record.

The Steady-State Hypothesis Is the Control

The single most important structural move is to write the pass criterion down as executable probes before the fault runs. In Litmus this is native: probes attach to the experiment, run in defined modes relative to the chaos window — before it (SOT), during it (Continuous), after it (EOT), or at both edges — and their aggregate result determines the verdict. This is what converts 'we think it held up' into a computed outcome.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payments-api-pod-delete-q3
  namespace: payments
  labels:
    experiment_id: "exp-2026-q3-0142"   # catalogue ID; stamp Deployment pod template for OTel
spec:
  engineState: active
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: payments
    applabel: "app=payments-api"
    appkind: deployment
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "180"
            - name: PODS_AFFECTED_PERC
              value: "34"        # blast radius: one third, never all
        probe:
          - name: slo-success-rate-holds
            type: promProbe
            mode: Continuous
            runProperties:
              probeTimeout: 5s
              interval: 10s
              retry: 2
              probePollingInterval: 10s
            promProbe/inputs:
              endpoint: http://prometheus.monitoring:9090
              query: |
                sum(rate(http_requests_total{job="payments-api",code!~"5.."}[1m]))
                / sum(rate(http_requests_total{job="payments-api"}[1m]))
              comparator:
                criteria: ">="
                value: "0.995"
          - name: checkout-path-reachable
            type: httpProbe
            mode: Edge
            runProperties:
              probeTimeout: 3s
              interval: 5s
              retry: 3
            httpProbe/inputs:
              url: http://payments-api.payments/healthz/deep
              method:
                get:
                  criteria: "=="
                  responseCode: "200"

Two details in that manifest carry disproportionate weight. PODS_AFFECTED_PERC is the blast-radius control — an experiment that kills every replica proves nothing except that zero replicas cannot serve traffic, and it converts a test into an incident. And the promProbe asserts a service-level objective rather than a liveness check, which is the difference between proving the pods came back and proving that users were still served while they were gone.

Precision Faults and Scoped Blast Radius

Where Chaos Mesh earns its place is in the fault primitives. A pod kill is the least interesting failure a distributed system can experience; the failures that actually break systems are partial and ambiguous — the dependency that is slow rather than down, the disk that returns errors on one in twenty writes, the DNS resolution that intermittently fails. Chaos Mesh expresses these directly, and its selector model scopes them tightly.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: ledger-dependency-latency
  namespace: payments
  annotations:
    experiment_id: "exp-2026-q3-0143"  # catalogue ID; stamp Deployment pod template for OTel
spec:
  action: delay
  mode: fixed-percent
  value: "40"                      # 40% of matched pods, not all
  duration: "5m"
  selector:
    namespaces: [payments]
    labelSelectors:
      app: payments-api
  direction: to
  target:
    mode: all
    selector:
      namespaces: [ledger]
      labelSelectors:
        app: ledger-svc
  delay:
    latency: "400ms"
    jitter: "120ms"
    correlation: "50"

That experiment answers a question a pod kill cannot: when the ledger service degrades but does not fail, does the payments API shed load gracefully, or does it exhaust its connection pool and take down callers that had nothing to do with the ledger? That is the failure mode that produces real outages, and it is precisely the scenario a regulator means by a scenario-based test. Run it behind the same progressive-delivery gates you use for releases and the analysis machinery is already in place.

The Experiment ID Is the Join Key

This is the step almost everyone skips, and it is the one that makes the difference between a result and evidence. Assign every experiment a stable identifier at design time, and ensure it lands on every signal the system emits while the fault is live. Without it, you have a verdict that asserts something and an observability stack that cannot corroborate it. With it, an auditor — or a post-incident reviewer two years later — can pull the exact traces recorded during the fault window and see the behaviour for themselves.

Click to zoom

Mechanically, this is a small amount of plumbing on a stack you already run. Neither Litmus nor Chaos Mesh copies the experiment identifier onto the application pods that emit telemetry, so the runbook must stamp experiment_id onto the target Deployment's pod template — annotations or labels under spec.template.metadata — before the fault window, then strip that key on teardown. Annotating only currently running Pods is not enough for pod-delete or pod-kill: deleted Pods are recreated from the template without the runtime annotation, and with the fail-closed resource/chaos_default processor those replacements stamp chaos.experiment_id as none for the rest of the window. Changing the pod template starts a Deployment rollout, so do this work outside the fault window: patch the template first, then either annotate already-running Pods with the same key (covers the transition) or wait until rollout status is complete so every replica carries the key before inject; on teardown remove the template key and any live-Pod copy so post-window traffic is not mis-tagged — knowing the strip rolls again. A collector k8sattributes processor still extracts from: pod and promotes that pod annotation to a resource attribute on spans, metrics, and log records for the duration of the run — wire the same processors into the traces, metrics, and logs pipelines — queryable like any other dimension in your self-hosted observability stack.

# Before inject (outside the fault window): template first, then live Pods.
# Patching spec.template starts a Deployment rollout — dual-write covers the gap.
kubectl -n payments patch deployment payments-api --type=merge \
  -p '{"spec":{"template":{"metadata":{"annotations":{"experiment_id":"exp-2026-q3-0142"}}}}}'
kubectl -n payments annotate pods -l app=payments-api \
  experiment_id=exp-2026-q3-0142 --overwrite
# Or wait until every replica is on the new template before inject:
# kubectl -n payments rollout status deployment/payments-api

# After the window: strip template + live copies (template edit rolls again).
kubectl -n payments patch deployment payments-api --type=json \
  -p '[{"op":"remove","path":"/spec/template/metadata/annotations/experiment_id"}]'
kubectl -n payments annotate pods -l app=payments-api experiment_id-

processors:
  k8sattributes:
    extract:
      annotations:
        - key: experiment_id
          from: pod
          tag_name: chaos.experiment_id
      labels:
        - key: experiment_id
          from: pod
          tag_name: chaos.experiment_id

  # Fail closed at resource scope: insert only if k8sattributes left the key absent.
  resource/chaos_default:
    attributes:
      - key: chaos.experiment_id
        value: "none"
        action: insert

service:
  pipelines:
    traces:
      processors: [k8sattributes, resource/chaos_default, batch]
    metrics:
      processors: [k8sattributes, resource/chaos_default, batch]
    logs:
      processors: [k8sattributes, resource/chaos_default, batch]

Signing and Retaining the Verdict

A ChaosResult living in etcd is an operational signal, not a record. It shares the cluster's lifecycle, it is mutable by anyone with the right RBAC, and it will be gone in the next cluster rebuild. The final step is to extract the verdict, bind it to the inputs that produced it, sign it, and put it somewhere with a retention policy — the same supply-chain machinery you already use for signing and attesting build artifacts applies unchanged.

#!/usr/bin/env bash
set -euo pipefail

EXP_ID="exp-2026-q3-0142"
NS="payments"

# Real fault-window bounds — not ChaosEngine object birth.
# creationTimestamp diverges when the engine is created early, left standing
# and re-activated, or delayed by scheduling. Litmus chaos-exporter derives
# start/end from ExperimentDependencyCheck / Summary engine events
# (metrics litmuschaos_experiment_start_time / litmuschaos_experiment_end_time).
# Convert those epoch seconds (or event lastTimestamp) to RFC3339 before export.
: "${GIT_SHA:?GIT_SHA must be set to the reviewed experiment-manifest commit}"
: "${WINDOW_START:?WINDOW_START must be experiment start (RFC3339) from exporter/events}"
: "${WINDOW_END:?WINDOW_END must be experiment end (RFC3339) from exporter/events}"
[[ -n "$WINDOW_START" && -n "$WINDOW_END" ]] || {
  echo "window bounds must be non-empty RFC3339 timestamps" >&2
  exit 1
}

kubectl -n "$NS" get chaosresult \
  payments-api-pod-delete-q3-pod-delete -o json \
  | jq --arg id "$EXP_ID" --arg start "$WINDOW_START" --arg end "$WINDOW_END" --arg sha "$GIT_SHA" '{
      experiment_id:   $id,
      verdict:         .status.experimentStatus.verdict,
      probes:          .status.probeStatuses,
      manifest_commit: $sha,
      window: { start: $start, end: $end }
    }' > "verdict-${EXP_ID}.json"

# Sign keylessly (OIDC identity, transparency-log backed).
cosign sign-blob --yes \
  --bundle "verdict-${EXP_ID}.bundle" \
  "verdict-${EXP_ID}.json"

# Retain in object storage with an immutability policy set.
mc cp "verdict-${EXP_ID}.json" "verdict-${EXP_ID}.bundle" \
  "evidence/resilience-testing/2026/q3/"

What that artifact gives you is a closed loop: the signature proves the verdict was not edited after the fact, the commit SHA proves which reviewed hypothesis was tested, the window bounds pin which telemetry interval is in scope (sourced from the run, not from object creation time), and the experiment ID lets anyone reconstruct that telemetry — four claims, independently checkable, none dependent on trusting the team that ran the test.

The Game Day Runbook, Rewritten Around Evidence

Once the pipeline exists, the human exercise gets simpler rather than more elaborate, because most of what a traditional runbook covers — who watches which dashboard, who decides whether it passed — is now handled by the probes. What remains is a short, repeatable sequence any team can run each quarter that leaves the same artifacts behind every time.

1. Select the scenario from the catalogue, not from the imagination. The catalogue is version-controlled and covers the systems supporting critical functions; picking from it is what makes coverage auditable.
2. Fix the hypothesis by review. The probes and blast radius merge through a pull request with an independent reviewer before the experiment is scheduled. This is the step that supplies independence.
3. Announce the window and confirm the abort path — post the experiment ID where responders will see it, so a genuine unrelated incident during the window is not misattributed to the test, and confirm the kill switch reverses the fault.
4. Stamp experiment_id on the workload pod template before inject; strip on teardown. Neither engine copies the CR identifier onto application pods. Outside the fault window, patch the Deployment spec.template.metadata annotations or labels so replacements inherit the key under pod-delete/pod-kill; dual-write live Pods or wait for the template-driven rollout so already-running replicas match. Live-Pod-only annotate is not enough. Remove the template key (and live copies) after the window so later traffic is not mis-tagged — the strip rolls again.
5. Run it and leave it alone. Resist the urge to intervene while probes are still evaluating; a rescued experiment produces a verdict about the operators, not the system.
6. Sign, retain, and file findings. Export WINDOW_START / WINDOW_END from chaos-exporter start/end metrics or the matching ChaosEngine events (ExperimentDependencyCheck / Summary) — never from .metadata.creationTimestamp — require GIT_SHA, sign the verdict, and retain it. A failed hypothesis becomes a remediation ticket linked to the experiment ID, and the re-test that closes it is scheduled at the same time — not left to be remembered.

Failure Modes That Make the Evidence Worthless

• Testing in a non-representative environment. An experiment in a staging cluster with one tenth the traffic and none of the data volume proves very little about the production system. Where full production testing is not yet safe, say so explicitly in the record and scope the claim accordingly — an honest limitation is defensible; an unstated one is a finding.
• Probes that assert liveness instead of the SLO. kubectl get pods returning Running is not a steady state. If the probe cannot fail while users are suffering, it is decorative.
• Join key lost on recreation. Annotating only currently running Pods leaves pod-delete/pod-kill replacements without experiment_id, so k8sattributes cannot promote it and resource/chaos_default stamps none for the rest of the fault window. Stamp the Deployment pod template before inject, then dual-write or roll live replicas.
• Unbounded blast radius. An experiment that takes out every replica is an outage you scheduled. Bound it with PODS_AFFECTED_PERC or mode: fixed-percent, and require an abort path.
• No abort path. Every experiment needs a documented, tested kill switch — deleting the chaos resource must reliably reverse the fault. Verify that this works before you need it, not during.
• Evidence with no retention policy. A verdict in a namespace that gets torn down next quarter is not retained. Bind retention to the same schedule as the rest of your compliance record.
• Telemetry window from object birth. ChaosEngine .metadata.creationTimestamp is when the CR was created, not when the experiment ran. Signing that as window.start yields a complete-looking artifact with the wrong interval — worse for audit join than an explicit null. Bind bounds from chaos-exporter litmuschaos_experiment_start_time / litmuschaos_experiment_end_time (or ExperimentDependencyCheck / Summary events), convert to RFC3339, and fail closed if either bound is unset or empty.
• Re-running only the experiments that pass. A quietly retired failing scenario is the single most damaging pattern here. Track the catalogue, not the results.

Exit Ramps and the Long Game

The optionality argument here is unusually clean. Both engines are Apache-2.0 and run entirely on infrastructure you control, so the fault-injection layer is replaceable by design. The durable asset is not the engine — it is the experiment catalogue and the evidence archive. Keep hypotheses as portable probe definitions and verdicts as plain signed JSON in object storage, and swapping Litmus for Chaos Mesh becomes a translation across the injection layer, not a loss of history.

That is the inversion worth internalising. Commercial chaos platforms sell the injection — the commoditised part — and hold the history, which compounds. Building the evidence layer yourself — a few hundred lines of pipeline around two mature open-source controllers — keeps that compounding asset on your side of the boundary, without a vendor renewal date attached to the record.

Run this for three years and the archive becomes something no dashboard can give you: a longitudinal record of how your system's failure behaviour changed as it grew. You can show that a scenario which failed in one quarter passed in the next and has passed every quarter since — which is a far stronger claim than any point-in-time test, and exactly the kind of durable, verifiable record that survives an audit, a re-architecture, and the departure of everyone who was in the room for the first game day. If you are building a resilience-testing programme that has to hold up under that kind of scrutiny, that is the sort of work we take on.