Catching the Breach in Kernel Time: Falco and Tetragon for Runtime Defense

There is a category of Kubernetes security spending that buys a great deal of comfort and very little protection: the posture scan. A nightly job enumerates your RBAC bindings, flags the pods running as root, notes the missing network policies, and emails a tidy compliance score. None of it watches what your workloads actually do at runtime. When an attacker exploits a deserialization bug in a Java service, drops a shell, and reaches for /etc/shadow at 03:00, the posture scan has nothing to say — it audited the configuration, not the behavior. The breach happens in the gap between what your cluster is permitted to do and what it is doing right now, and closing that gap is the job of runtime threat detection.

For most of the last decade, runtime detection carried a tax that kept it off serious production clusters: it meant a kernel module you were nervous to load, or a sampling probe that missed events under churn, or an agent whose CPU overhead made the platform team quietly disable it. eBPF dissolved that objection. By 2026 the two leading open-source tools — Falco, which graduated from the CNCF in February 2024, and Tetragon, the runtime security project that grew out of the Cilium ecosystem — are both production-grade, both eBPF-native, and both deployable on a cluster you own end to end. They solve overlapping but genuinely distinct problems, and the right architecture for a serious environment usually runs both.

This article is a production-grade walkthrough for platform and security engineers deciding how to instrument a cluster for runtime threats. It covers how eBPF made detection cheap, how Falco maps detections to the MITRE ATT&CK framework, why detection alone leaves a time-of-check/time-of-use window that Tetragon closes with synchronous in-kernel enforcement, and how to pipe the whole signal into a self-hosted observability stack instead of a vendor's cloud. The goal is a detection-and-response layer you control, tuned to a defensible overhead, with an audit trail that lives inside your boundary.

Posture Scanning Tells You What's Allowed; Runtime Detection Tells You What's Happening

Kubernetes Security Posture Management (KSPM) and runtime threat detection are frequently conflated in procurement conversations, which leads teams to buy one and believe they have the other. KSPM tools — Kubescape, kube-bench, Trivy's misconfiguration scanner — evaluate the declared state of the cluster against benchmarks like the CIS Kubernetes Benchmark and the NSA/CISA hardening guidance. They answer the question "is this cluster configured in a way that could be exploited?" That is a valuable question, and it composes naturally with admission control through policy-as-code, which refuses non-conforming workloads before they ever schedule.

Runtime detection answers a different question: "is something being exploited right now?" It watches the live stream of kernel events — process executions, file opens, network connections, privilege changes, kernel-module loads — and matches them against behavioral rules. A workload can pass every posture check and still be compromised the moment an attacker finds a code-execution path inside it. Posture management hardens the box; runtime detection watches what climbs out of it. The two are complementary layers of the same defense, not substitutes, and a mature program runs both with the boundary between them drawn deliberately rather than by accident of which vendor was bought first.

How eBPF Made Runtime Detection Cheap Enough to Run Everywhere

eBPF (extended Berkeley Packet Filter) lets verified programs run inside the Linux kernel in response to events — syscalls, function entry/exit, tracepoints — without patching kernel source or loading an out-of-tree module. Every eBPF program passes through the kernel's verifier at load time, which proves the program cannot crash the kernel, dereference invalid memory, or loop forever before it is allowed to attach. This is the property that makes eBPF acceptable on production nodes where a buggy kernel module would be an outage waiting to happen: the kernel itself refuses to run an unsafe program.

The second enabling advance is CO-RE — Compile Once, Run Everywhere. A CO-RE eBPF object, compiled once against the libbpf library, relocates its references to kernel struct fields at load time using BTF (BPF Type Format) metadata embedded in the running kernel. The practical consequence is enormous: you ship one probe binary that runs across kernel versions without recompiling per node. Falco's modern eBPF probe is libbpf/CO-RE based and is the recommended driver as of recent releases, requiring a kernel of roughly 5.8 or newer with BTF exposed. It replaces both the legacy eBPF probe and the loadable kernel module for most deployments — the same shift toward auditable, in-tree mechanisms that runs through Cilium's eBPF dataplane on the networking side.

It is worth being precise — and honest — about overhead, because the marketing claims run ahead of the measurements. eBPF runtime monitoring is not free. A 2025 peer-reviewed comparison (RITECH 2025, SCITEPRESS) measured Falco's CPU overhead in the low-to-mid twenties of a percent under a deliberately heavy syscall load, with the figure rising as the active rule set grew. Tetragon, in the same study and in community benchmarks, came out as the most CPU-efficient of the eBPF tools, in part because it filters events in the kernel before they ever reach userspace. The honest framing for a platform team is this: overhead is real and proportional to syscall volume and rule complexity, it is dominated by event copy-out to userspace, and it is tunable — which is most of what the rest of this article is about.

Falco: Broad Detection Mapped to MITRE ATT&CK

Falco's strength is breadth. It ships a maintained corpus of community rules covering the common container attack behaviors, and its rule language is simple enough that a security engineer can read, audit, and extend it without learning a programming language. A Falco rule has four required fields — rule (a name), desc, condition (a boolean filter expression over the event stream), and output (the alert template) — plus a priority and optional tags. The tags field is where the MITRE ATT&CK mapping lives, and the Falco rule style guide expects security rules to carry their relevant technique as a tag.

# A representative Falco rule: a shell with a controlling terminal,
# spawned inside a container, is a classic post-exploitation signal.
# The tags carry the MITRE ATT&CK technique so the alert is coverage-mappable.
- rule: Terminal shell in container
  desc: A shell was spawned with an attached terminal inside a container
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name image=%container.image.repository
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: NOTICE
  tags: [container, shell, mitre_execution, T1059]

# A higher-severity rule: a non-trusted process reading credential stores.
- rule: Read sensitive credential file
  desc: An unexpected program opened /etc/shadow or /etc/sudoers for reading
  condition: >
    open_read and container
    and (fd.name=/etc/shadow or fd.name=/etc/sudoers)
    and not proc.name in (trusted_credential_readers)
  output: >
    Sensitive credential file read
    (user=%user.name file=%fd.name proc=%proc.name container=%container.name)
  priority: WARNING
  tags: [filesystem, mitre_credential_access, T1003]

Around the detection core, Falco has grown an ecosystem that matters for production. Falcosidekick is a fan-out router that forwards alerts to sixty-plus destinations — Slack, Elasticsearch, Loki, S3, webhooks — so you can land Falco events wherever your incident workflow lives. The plugins system extends Falco beyond syscalls to other event sources, letting the same engine evaluate, for example, cloud audit logs alongside kernel events. And Falco Talon, the project's response engine introduced in 2024, consumes alerts and runs declarative "actionners" — terminate the pod, apply a NetworkPolicy, label the workload for quarantine. That last piece is important, and it is also where the architectural limit of a detection-first tool becomes visible.

The TOCTOU Gap: Why Detection Alone Leaves a Window

Falco is, by design, an observer. Its eBPF probe copies kernel events into a ring buffer; the Falco userspace process reads them, evaluates the rule set, and emits an alert. Every step after the kernel hook happens in userspace, after the syscall has already completed. When Falco tells you a process opened /etc/shadow, the read has already returned the file's contents to the attacker. When Talon fires a response to kill the offending pod, it is closing the barn door — the response is asynchronous, separated from the malicious action by a real interval of milliseconds to seconds.

This is the classic time-of-check/time-of-use (TOCTOU) gap, and it is not a Falco defect — it is intrinsic to any detect-then-respond architecture. For a great many threats, fast detection plus automated response is exactly the right trade: you accept a small window in exchange for broad, cheap, auditable coverage. But for a specific, high-value class of actions — reading a credential store, loading a kernel module, writing to a container-escape path — a window of any size is a window too many. Those are the actions you want prevented, synchronously, before the syscall body runs. That requires moving the decision out of userspace and into the kernel itself.

Click to zoom

Tetragon: Enforcement in Kernel Time

Tetragon attaches eBPF programs to kprobes, tracepoints, and — critically — LSM (Linux Security Module) BPF hooks, the kernel's designated access-control decision points. Its unit of configuration is the TracingPolicy custom resource, which declares which kernel functions to hook, which arguments to extract, and — through selectors — exactly which events to act on, all evaluated inside the kernel. Because the filtering happens in-kernel, the events that never match are dropped before any userspace copy-out, which is the structural reason Tetragon's overhead stays low even on busy nodes.

What sets Tetragon apart is that a selector can carry a matchActions clause that enforces. Two enforcement mechanisms matter. Sigkill sends a SIGKILL to the offending process directly from kernel context — the process is terminated and never returns from the syscall. Override uses the kernel's error-injection framework to replace the syscall's return value with an error code such as EPERM, so the call fails as though permission were denied; the function body is never executed. Override requires the kernel to be built with CONFIG_BPF_KPROBE_OVERRIDE, a constraint worth verifying on your nodes before you design around it.

# Tetragon TracingPolicy: block reads of credential stores in the kernel.
# The LSM file-permission hook fires BEFORE the read is served; Override
# returns EPERM so the contents are never disclosed. The decision is
# synchronous and in-kernel — no userspace round-trip, no TOCTOU window.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "block-credential-theft"
spec:
  kprobes:
    - call: "security_file_permission"   # LSM hook, evaluated pre-access
      syscall: false
      return: true
      args:
        - index: 0
          type: "file"                   # struct file *
        - index: 1
          type: "int"                    # requested access mask
      returnArg:
        index: 0
        type: "int"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "/etc/shadow"
                - "/etc/sudoers"
          matchActions:
            # Override returns an error from the kernel; the read never succeeds.
            # Requires the node kernel built with CONFIG_BPF_KPROBE_OVERRIDE.
            # Swap for "action: Sigkill" to terminate the process outright.
            - action: Override
              argError: -1

Tetragon also records full process lineage — parent, grandparent, the binary that executed, the capabilities held at event time — which makes its event stream far more useful for reconstructing an attack chain than a flat syscall log. When a shell appears in a container, Tetragon can show you the entire ancestry that led to it, which is the difference between an alert and an investigation you can actually close. You can stream those events with the tetra CLI, in compact form scoped to the pods you care about:

# Stream Tetragon events with process lineage, scoped to one app's pods.
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
  tetra getevents -o compact --pods my-app

# Run Falco with the modern eBPF probe (CO-RE, no kernel module to load).
falco --modern-bpf

# Confirm the node kernel supports Override enforcement before relying on it.
grep -E 'CONFIG_BPF_KPROBE_OVERRIDE' /boot/config-$(uname -r)

Running Both: Falco for Breadth, Tetragon for the Kill Switch

The Falco-versus-Tetragon framing that dominates blog comparisons is the wrong frame, in the same way the Cilium-versus-Istio debate was. They are not competing for the same slot. Falco gives you broad, ATT&CK-mapped detection across a large rule corpus, with a mature output ecosystem and a community that keeps the rules current. Tetragon gives you low-overhead in-kernel observation and the ability to synchronously prevent a short list of unambiguous, high-severity actions. The defensible architecture runs Falco as the wide detection net and Tetragon as the targeted enforcement layer for the handful of behaviors where the TOCTOU window is unacceptable.

Concretely: let Falco watch for the broad behavioral signals — unexpected shells, suspicious outbound connections, package-manager execution in production containers, sensitive mounts — and route those to your SIEM with ATT&CK tags for triage. Reserve Tetragon enforcement for the assertions you are willing to make absolute: no process reads /etc/shadow, no container loads a kernel module, nothing writes to the paths associated with container escape. Keep the Tetragon enforcing policy set small and high-confidence — every enforcing rule is a potential way to break a legitimate workload, so the bar for promoting a detection into an in-kernel block should be high and the policy should be reviewed like the production change it is.

Click to zoom

Piping Alerts to a Self-Hosted SIEM

Where the alerts land is not an afterthought — it is the difference between a detection capability you own and one you rent. The default posture of most commercial runtime-security products is to ship your security telemetry to the vendor's cloud, where it is correlated, retained, and — should you ever leave — held hostage to the renewal conversation. For an engineering practice built on sovereignty, that is the wrong default. Falcosidekick routes the entire alert stream to sinks you operate: Loki or Elasticsearch for search, S3-compatible object storage you control for retention, and a webhook to Talon for response.

# Falcosidekick Helm values — every sink is inside the cluster boundary.
# No alert leaves infrastructure you operate.
config:
  loki:
    hostport: "http://loki.monitoring.svc:3100"
    minimumpriority: "notice"
  elasticsearch:
    hostport: "https://elastic.monitoring.svc:9200"
    index: "falco"
    minimumpriority: "notice"
  # Route WARNING+ to the response engine for automated containment.
  webhook:
    address: "http://falco-talon.falco.svc:2803"
    minimumpriority: "warning"

Keeping the telemetry inside your boundary is also what makes it usable as audit evidence. When a NIS2 or DORA assessor asks how you detect and respond to runtime intrusions, the answer is a set of version-controlled Falco rules and Tetragon TracingPolicies, an event stream with verdicts and timestamps in a store you operate, and a retention policy you can prove. That is a far stronger position than a screenshot from a vendor console and a hope that the SaaS configuration has not silently drifted. The flow records belong in the same self-hosted observability pipeline as the rest of your operational signal, correlatable against application traces and infrastructure metrics in one place.

Tuning Under Load Without Going Blind

The recurring objection to runtime detection — "it's too expensive to run in production" — is now an engineering problem with known levers, not a reason to skip the control. The dominant cost is event copy-out from kernel to userspace, so the highest-leverage move is to filter as early and as deeply in the kernel as possible. Tetragon's in-kernel selectors do this natively: a TracingPolicy that scopes to specific binaries, namespaces, or argument values discards non-matching events before they cost a userspace round-trip. For Falco, the equivalent discipline is pruning the rule set to what your threat model actually requires rather than running the entire community corpus, and avoiding rules whose conditions match extremely high-frequency syscalls unless you genuinely need them.

Treat overhead as something you measure, not something you assume. Establish a baseline of node CPU and syscall rate without the agents, enable detection in observation-only mode, and measure the delta under representative load — including your noisiest batch workloads, which are where syscall-heavy rules hurt most. Set priority thresholds so low-value INFORMATIONAL events are not paying the full pipeline cost to reach storage. Promote a rule from observe to enforce only after you have watched it in production long enough to trust that it will not page someone at 03:00 for a legitimate cron job. The aim is a configuration where the security signal is high and the overhead sits in a single-digit-percent band you have verified on your own hardware, rather than a number copied from a vendor's benchmark deck.

The Exit Ramp: No Lock-In in Your Detection Layer

The optionality argument for this stack is unusually strong because the detection contract is portable in a way most security tooling is not. Falco rules and Tetragon TracingPolicies are plain YAML in your repository. The detections are anchored to MITRE ATT&CK technique IDs, which are a vendor-neutral standard — your coverage map is expressed in a vocabulary that survives a change of tool. The telemetry lands in open stores (Loki, Elasticsearch, OTel-compatible pipelines) you already operate. If you decide tomorrow to swap Falco for Tracee, Aqua's eBPF detection-and-forensics tool, or to add a different engine alongside, you are changing the producer of events, not rebuilding your entire detection and response apparatus.

Contrast that with the commercial runtime-security model, where the rules are proprietary, the ATT&CK mapping is buried in a console, the telemetry lives in the vendor's cloud, and the response automation is wired to their orchestration. Leaving means losing your detection history, re-authoring your rules in a new dialect, and re-integrating your response workflow. The open-source eBPF stack is the same capability with the exit ramp designed in — which is precisely the property that lets you adopt it without betting the security posture of the next decade on a single vendor's roadmap.

Click to zoom

Putting It Together: A Runtime Defense You Operate

The minimum viable runtime defense for a cluster you take seriously looks like this. Deploy Falco with the modern eBPF probe and a pruned, ATT&CK-tagged rule set covering the behaviors in your threat model. Wire Falcosidekick to a self-hosted SIEM and to Falco Talon for automated containment of the broad-detection cases. Deploy Tetragon and write a small, high-confidence set of enforcing TracingPolicies for the actions you will not tolerate even for a millisecond — credential-store reads, kernel-module loads, container-escape writes. Run everything in observation mode first, measure the overhead on your own hardware, and promote to enforcement deliberately. Pair the whole thing with policy-as-code admission control and workload identity so that detection sits on top of a hardened, authenticated substrate rather than carrying the entire burden alone.

The long game here is a security posture that does not depend on a single engineer's knowledge or a single vendor's continued goodwill. The rules are YAML in version control. The detections speak MITRE ATT&CK. The enforcement runs in the kernel, on a host you can harden down to the immutable OS layer. The evidence lives in your boundary. When the person who built this leaves, or the vendor pivots, or the auditor arrives, the capability is legible and it is yours. That is what runtime defense looks like when it is engineered to last rather than bought to check a box — and in 2026, with Falco graduated and Tetragon production-ready, there is no longer a credible overhead excuse for not building it.