Self-Hosted Chaos Engineering as an Audit-Grade Compliance Control

Most chaos engineering programmes die the same way. A team runs an enthusiastic first game day, kills some pods, discovers something genuinely alarming about a retry loop, fixes it, and writes a page in a wiki. Six months later nobody can say whether the fix still holds, whether the experiment was ever re-run, or what the system actually did while the fault was live. The practice was real. The record is a memory.

That gap has stopped being merely an engineering hygiene problem. Operational-resilience regimes now ask regulated organisations to demonstrate — on a schedule, with documentation — that they have tested the failure of the systems supporting critical functions. The question is no longer 'do you do chaos engineering?' but 'show me the test, the scope, the result, and the remediation.' A wiki page is not an answer to that question.

This article is about closing that gap on infrastructure you own: how to structure fault-injection experiments with LitmusChaos and Chaos Mesh so that the by-product of running them is admissible evidence rather than an anecdote. Where disaster recovery proves you can restore after a failure, this proves you survive one — proactively, repeatably, and with a chain of custody.

The Game Day Produces a Memory, Not an Artifact

The classic game day is a scheduled, human-driven exercise: engineers gather, someone injects a fault, everyone watches dashboards, and the group discusses what happened. It is genuinely valuable as a learning ritual and as a way to exercise the human side of incident response. It is also structurally incapable of producing evidence, for three reasons.

First, the pass/fail criterion lives in the heads of the participants. If 'the system stayed healthy' was never written down as a machine-checkable assertion before the fault, then afterwards it is a judgement call, and judgement calls do not survive an auditor asking how the conclusion was reached. Second, the observability record is unlabelled: the traces and metrics from the fault window are mixed into the same streams as ordinary production traffic, with nothing marking which requests were affected by a deliberate experiment. Third, nothing is retained in a tamper-evident form — the wiki page can be edited, and the Grafana dashboard has long since rolled past its retention window.

The fix is not to stop running game days. It is to demote the human exercise to one output of a pipeline whose primary output is a signed record. The experiment definition becomes a reviewed artifact in Git. The steady-state hypothesis becomes probes. The verdict becomes a signed document with a retention policy. The humans still gather and still learn — but the evidence no longer depends on anyone remembering to write it down.

What the Regulation Actually Asks For

It is worth being precise here, because vendors routinely overstate what the rules say. Under the EU's Digital Operational Resilience Act (Regulation (EU) 2022/2554), Article 25 requires financial entities to maintain a testing programme that includes an enumerated range of test types — among them vulnerability assessments, gap analyses, source-code reviews, performance testing, end-to-end testing, penetration testing, and, most relevant here, scenario-based tests. Systems supporting critical or important functions must be tested at least annually, and tests must be carried out by independent parties, whether internal or external.

Note what this is not. The advanced threat-led penetration testing regime in Articles 26 and 27 — the intelligence-driven red-team exercise conducted at least every three years — applies only to designated entities whose failure would have systemic effects, and chaos engineering is not a substitute for it. Fault injection is a scenario-based resilience test, not an adversary simulation. Conflating the two is the fastest way to fail a review.

The useful insight is that a well-structured chaos experiment maps almost exactly onto what a scenario-based test is supposed to be: a defined scenario, a defined scope, a predicted outcome, an observed outcome, and a remediation path where the two diverge. The regulation does not name any tool, and it certainly does not require a SaaS platform. It requires that the test happened, that it covered the right systems, and that you can show the result. Everything in this article is about making that showable. The same discipline satisfies the equivalent expectations under NIS2 and the wider European compliance surface.

Choosing the Engine: Litmus and Chaos Mesh

Both leading open-source options are CNCF Incubating projects — neither has graduated, which is worth knowing before you present a maturity argument to a risk committee. Chaos Mesh originated at PingCAP, where it was built to test the TiDB distributed database, was accepted into the CNCF in July 2020, and moved to the Incubator in February 2022. LitmusChaos originated at MayaData, was donated to the CNCF in 2020, and reached Incubating status in 2022 as well.

Click to zoom

The practical difference is one of centre of gravity rather than capability. Chaos Mesh is the richer fault-injection engine: a broad set of granular CRDs covering pod, network, IO, stress, time, kernel, DNS, and HTTP faults, injected by a privileged per-node chaos-daemon, with a clean built-in dashboard. If your experiments need precise, low-level failure modes — packet corruption at a specific rate, filesystem latency on a particular mount, clock skew inside one container — Chaos Mesh gets there with less effort.

Litmus centres on the experiment lifecycle instead: a ChaosHub of versioned, reusable experiment definitions, a ChaosCenter control plane that schedules across clusters, Argo-based workflows for sequencing multi-step scenarios, and a first-class probe model that produces a ChaosResult with a resilience score. For an evidence programme running the same catalogue of scenarios quarterly across many clusters, that lifecycle model is the more valuable half.

• Pick Chaos Mesh when fault fidelity dominates — you need specific kernel, IO, or network failure semantics, and you are running a small number of clusters.
• Pick Litmus when programme mechanics dominate — a reusable catalogue, multi-cluster scheduling, and GitOps-driven chaos pipelines matter more than exotic fault types.
• Both are CNCF Incubating, both are Apache-2.0, and both run entirely on infrastructure you own with no telemetry egress. Neither decision is a lock-in event.
• Neither produces audit evidence on its own. Both record results in Kubernetes objects with ordinary etcd lifetimes — useful operationally, insufficient as a retained record.

The Steady-State Hypothesis Is the Control

The single most important structural move is to write the pass criterion down as executable probes before the fault runs. In Litmus this is native: probes attach to the experiment, run in defined modes relative to the chaos window — before it (SOT), during it (Continuous), after it (EOT), or at both edges — and their aggregate result determines the verdict. This is what converts 'we think it held up' into a computed outcome.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payments-api-pod-delete-q3
  namespace: payments
  labels:
    experiment_id: "exp-2026-q3-0142"   # join key — see below
spec:
  engineState: active
  chaosServiceAccount: litmus-admin
  appinfo:
    appns: payments
    applabel: "app=payments-api"
    appkind: deployment
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "180"
            - name: PODS_AFFECTED_PERC
              value: "34"        # blast radius: one third, never all
        probe:
          - name: slo-success-rate-holds
            type: promProbe
            mode: Continuous
            runProperties:
              probeTimeout: 5s
              interval: 10s
              retry: 2
              probePollingInterval: 10s
            promProbe/inputs:
              endpoint: http://prometheus.monitoring:9090
              query: |
                sum(rate(http_requests_total{job="payments-api",code!~"5.."}[1m]))
                / sum(rate(http_requests_total{job="payments-api"}[1m]))
              comparator:
                criteria: ">="
                value: "0.995"
          - name: checkout-path-reachable
            type: httpProbe
            mode: Edge
            runProperties:
              probeTimeout: 3s
              interval: 5s
              retry: 3
            httpProbe/inputs:
              url: http://payments-api.payments/healthz/deep
              method:
                get:
                  criteria: "=="
                  responseCode: "200"

Two details in that manifest carry disproportionate weight. PODS_AFFECTED_PERC is the blast-radius control — an experiment that kills every replica proves nothing except that zero replicas cannot serve traffic, and it converts a test into an incident. And the promProbe asserts a service-level objective rather than a liveness check, which is the difference between proving the pods came back and proving that users were still served while they were gone.

Precision Faults and Scoped Blast Radius

Where Chaos Mesh earns its place is in the fault primitives. A pod kill is the least interesting failure a distributed system can experience; the failures that actually break systems are partial and ambiguous — the dependency that is slow rather than down, the disk that returns errors on one in twenty writes, the DNS resolution that intermittently fails. Chaos Mesh expresses these directly, and its selector model scopes them tightly.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: ledger-dependency-latency
  namespace: payments
  annotations:
    experiment_id: "exp-2026-q3-0143"
spec:
  action: delay
  mode: fixed-percent
  value: "40"                      # 40% of matched pods, not all
  duration: "5m"
  selector:
    namespaces: [payments]
    labelSelectors:
      app: payments-api
  direction: to
  target:
    mode: all
    selector:
      namespaces: [ledger]
      labelSelectors:
        app: ledger-svc
  delay:
    latency: "400ms"
    jitter: "120ms"
    correlation: "50"

That experiment answers a question a pod kill cannot: when the ledger service degrades but does not fail, does the payments API shed load gracefully, or does it exhaust its connection pool and take down callers that had nothing to do with the ledger? That is the failure mode that produces real outages, and it is precisely the scenario a regulator means by a scenario-based test. Run it behind the same progressive-delivery gates you use for releases and the analysis machinery is already in place.

The Experiment ID Is the Join Key

This is the step almost everyone skips, and it is the one that makes the difference between a result and evidence. Assign every experiment a stable identifier at design time, and propagate it into every signal the system emits while the fault is live. Without it, you have a verdict that asserts something and an observability stack that cannot corroborate it. With it, an auditor — or a post-incident reviewer two years later — can pull the exact traces recorded during the fault window and see the behaviour for themselves.

Click to zoom

Mechanically, this is a small amount of plumbing on a stack you already run. The chaos controller sets the identifier as a label or annotation; a collector processor stamps it onto spans and metrics for the duration of the run; and because it lands as a resource attribute, it is queryable in exactly the same way as any other dimension in your self-hosted observability stack.

processors:
  k8sattributes:
    extract:
      annotations:
        - key: experiment_id
          from: pod
          tag_name: chaos.experiment_id
      labels:
        - key: experiment_id
          from: pod
          tag_name: chaos.experiment_id

  # Fail closed: an unlabelled fault window is not evidence.
  attributes/chaos_default:
    actions:
      - key: chaos.experiment_id
        value: "none"
        action: insert

service:
  pipelines:
    traces:
      processors: [k8sattributes, attributes/chaos_default, batch]
    metrics:
      processors: [k8sattributes, attributes/chaos_default, batch]

Signing and Retaining the Verdict

A ChaosResult living in etcd is an operational signal, not a record. It shares the cluster's lifecycle, it is mutable by anyone with the right RBAC, and it will be gone in the next cluster rebuild. The final step is to extract the verdict, bind it to the inputs that produced it, sign it, and put it somewhere with a retention policy — the same supply-chain machinery you already use for signing and attesting build artifacts applies unchanged.

#!/usr/bin/env bash
set -euo pipefail

EXP_ID="exp-2026-q3-0142"
NS="payments"

# 1. Capture the computed verdict and probe values from the chaos engine.
kubectl -n "$NS" get chaosresult \
  payments-api-pod-delete-q3-pod-delete -o json \
  | jq --arg id "$EXP_ID" '{
      experiment_id:  $id,
      verdict:        .status.experimentStatus.verdict,
      probes:         .status.probeStatuses,
      manifest_commit: env.GIT_SHA,
      window: { start: .status.experimentStatus.startTime, end: now|todate }
    }' > "verdict-${EXP_ID}.json"

# 2. Sign it keylessly (OIDC identity, transparency-log backed).
cosign sign-blob --yes \
  --bundle "verdict-${EXP_ID}.bundle" \
  "verdict-${EXP_ID}.json"

# 3. Retain in object storage with an immutability policy set.
mc cp "verdict-${EXP_ID}.json" "verdict-${EXP_ID}.bundle" \
  "evidence/resilience-testing/2026/q3/"

What that artifact gives you is a closed loop: the signature proves the verdict was not edited after the fact, the commit SHA proves which reviewed hypothesis was tested, and the experiment ID lets anyone reconstruct the telemetry. Three separate claims, independently checkable, none of them dependent on trusting the team that ran the test.

The Game Day Runbook, Rewritten Around Evidence

Once the pipeline exists, the human exercise gets simpler rather than more elaborate, because most of what a traditional runbook spends its pages on — who watches which dashboard, who decides whether it passed — is now handled by the probes. What remains is a short, repeatable sequence that any team can run each quarter and that leaves the same artifacts behind every time.

1. Select the scenario from the catalogue, not from the imagination. The catalogue is version-controlled and covers the systems supporting critical functions; picking from it is what makes coverage auditable.
2. Fix the hypothesis by review. The probes and blast radius merge through a pull request with an independent reviewer before the experiment is scheduled. This is the step that supplies independence.
3. Announce the window and confirm the abort path — post the experiment ID where responders will see it, so a genuine unrelated incident during the window is not misattributed to the test, and confirm the kill switch reverses the fault.
4. Run it and leave it alone. Resist the urge to intervene while probes are still evaluating; a rescued experiment produces a verdict about the operators, not the system.
5. Sign, retain, and file findings. A failed hypothesis becomes a remediation ticket linked to the experiment ID, and the re-test that closes it is scheduled at the same time — not left to be remembered.

Failure Modes That Make the Evidence Worthless

• Testing in a non-representative environment. An experiment in a staging cluster with one tenth the traffic and none of the data volume proves very little about the production system. Where full production testing is not yet safe, say so explicitly in the record and scope the claim accordingly — an honest limitation is defensible; an unstated one is a finding.
• Probes that assert liveness instead of the SLO. kubectl get pods returning Running is not a steady state. If the probe cannot fail while users are suffering, it is decorative.
• Unbounded blast radius. An experiment that takes out every replica is an outage you scheduled. Bound it with PODS_AFFECTED_PERC or mode: fixed-percent, and require an abort path.
• No abort path. Every experiment needs a documented, tested kill switch — deleting the chaos resource must reliably reverse the fault. Verify that this works before you need it, not during.
• Evidence with no retention policy. A verdict in a namespace that gets torn down next quarter is not retained. Bind retention to the same schedule as the rest of your compliance record.
• Re-running only the experiments that pass. A quietly retired failing scenario is the single most damaging pattern here. Track the catalogue, not the results.

Exit Ramps and the Long Game

The optionality argument here is unusually clean. Both engines are Apache-2.0 and run entirely on infrastructure you control, which means the fault-injection layer is replaceable by design. But the durable asset is not the engine — it is the experiment catalogue and the evidence archive. Keep the hypotheses expressed as portable probe definitions and the verdicts as plain signed JSON in object storage, and swapping Litmus for Chaos Mesh becomes a translation exercise across the injection layer, not a loss of history.

That is the inversion worth internalising. The commercial chaos platforms sell the injection, which is the commoditised part, and hold the history, which is the part that compounds. Building the evidence layer yourself — a few hundred lines of pipeline around two mature open-source controllers — keeps the compounding asset on your side of the boundary. It also means your resilience record does not have a vendor's renewal date attached to it.

Run this for three years and the archive becomes something no dashboard can give you: a longitudinal record of how your system's failure behaviour changed as it grew. You can show that a scenario which failed in one quarter passed in the next and has passed every quarter since — which is a far stronger claim than any point-in-time test, and exactly the kind of durable, verifiable record that survives an audit, a re-architecture, and the departure of everyone who was in the room for the first game day. If you are building a resilience-testing programme that has to hold up under that kind of scrutiny, that is the sort of work we take on.