Default Deny, Actually: Auditing Kubernetes Network Policy

Ask a platform team whether the cluster is segmented and you are shown a manifest. Ask when it was last tested and the room goes quiet. In our engagements that is the most reliable gap in an otherwise disciplined estate: intent written down, reviewed, version-controlled, never exercised.

The Policy That Protects Nothing

Kubernetes promises less here than most readers assume. Network policies are implemented by the network plugin, and "creating a NetworkPolicy resource without a controller that implements it will have no effect." The API server validates a schema, not enforcement, and kubectl get networkpolicy returns a row either way. Four failure modes follow, none visible in the manifest that caused it:

1. No implementing plugin. The object is accepted, stored and reviewed; the datapath never hears about it. Common where somebody assumed the default plugin enforced policy.
2. The selector matches no pod. A pod is non-isolated until a policy selects it, so a podSelector matching nothing changes nothing. Kubescape ships C-0299 for exactly this: a policy that "gives a false sense of isolation — the policy exists but protects nothing."
3. The direction is not covered. Isolation is per-direction: "by default, a pod is non-isolated for egress" until a policy both selects it and names Egress in policyTypes. Omit it and outbound stays open while the object is called default-deny in review.
4. An additive allow already opened the path. Policies "do not conflict; they are additive", the allowed set is their union, and "order of evaluation does not affect the policy result." There is no explicit deny to override a permissive rule, so one broad allow added to unblock a migration survives every review after it.

Click to zoom

What Your Scanners Are Actually Checking

Start with the sharpest. CIS check 5.3.2, "Ensure that all Namespaces have NetworkPolicies defined", ships in kube-bench as type: "manual" with scored: false and no audit command. Its remediation is addressed to a person: "Follow the documentation and create NetworkPolicy objects as you need them." The tool does not test the control because it cannot, and its own YAML says so.

The other three stop at the same wall. Kubescape's C-0206 asks whether a namespace has a NetworkPolicy object; Trivy's KSV-0038 is Rego over manifest shape, its hasSelector helper accepting any of several selector shapes as present; Checkov's CKV2_K8S_6 is a graph check with cond_type: connection and operator: exists, asserting an edge between a workload and a NetworkPolicy. All four are correct about YAML, and none emits a packet.

Two are still worth running. Kubescape's C-0299 catches failure mode two, invisible without a cluster walk. And Kyverno closes the gap the existence checks complain about: its reference policy generates a NetworkPolicy named default-deny, empty podSelector, both policyTypes, whenever a Namespace appears — which guarantees the object exists everywhere, not that it does anything.

None of this criticises the tools; it is a boundary in the API they inspect. Upstream's limitations section — the current docs say Kubernetes 1.36 — lists "advanced policy querying and reachability tooling" and "the ability to log network security events" among what the API does not provide.

Rolling Out Default-Deny Without an Outage

The manifest is the easy part, and it is two objects, not one. A default deny-all egress policy also blocks DNS; upstream is explicit that workloads needing resolution require a separate policy allowing egress to the cluster DNS service. Ship both, or the namespace stops resolving names.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  # Empty selector: every pod in the namespace, now and later.
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

What makes a rollout survivable is the rung below the manifest: intent applied where it is observed, not enforced. Cilium's Policy Audit Mode "configures Cilium to allow all traffic while logging all connections that would otherwise be dropped by network policies", and with it enabled "no network policy is enforced so this setting is not recommended for production deployment." A staging step, never a resting state — L3/L4 only, and per-endpoint it resets to the daemon default on agent restart.

#!/usr/bin/env bash
set -euo pipefail

NS=payments
POD=checkout-7d9c8b5f4-2xk9p
CILIUM_NS=kube-system

NODE="$(kubectl get pod -n "$NS" "$POD" -o jsonpath='{.spec.nodeName}')"
AGENT="$(kubectl get pod -n "$CILIUM_NS" -l k8s-app=cilium \
  --field-selector "spec.nodeName=$NODE" \
  -o jsonpath='{.items[0].metadata.name}')"

# The CiliumEndpoint is named after the pod and holds the numeric ID.
EP="$(kubectl get cep -n "$NS" "$POD" -o jsonpath='{.status.id}')"
[ -n "$EP" ] || { echo "no CiliumEndpoint for $NS/$POD" >&2; exit 1; }

audit() {   # audit Enabled | audit Disabled
  kubectl exec -n "$CILIUM_NS" "$AGENT" -c cilium-agent -- \
    cilium-dbg endpoint config "$EP" PolicyAuditMode="$1"
}

# Allows traffic, logs what policy WOULD have dropped. Per-endpoint,
# reset to the daemon default if the agent restarts.
audit Enabled
trap 'audit Disabled' EXIT   # nothing is enforced while audit is on

# Hubble runs inside the agent owning this endpoint; the CLI otherwise
# talks to localhost:4245, where a Relay error looks like a quiet
# cluster. --verdict takes AUDIT; compact output PRINTS "AUDITED", so
# grepping AUDITED matches nothing and reads as "nothing was denied".
rc=0
timeout 900 kubectl exec -n "$CILIUM_NS" "$AGENT" -c cilium-agent -- \
  hubble observe --follow --type policy-verdict --verdict AUDIT \
  --namespace "$NS" --output compact | tee audit-flows.txt || rc=$?

# 124 is timeout doing its job on --follow. Anything else is a broken
# command, not an empty result set.
[ "$rc" -eq 124 ] || { echo "hubble observe failed: exit $rc" >&2; exit 1; }

Calico expresses the same idea as a resource: a StagedNetworkPolicy's rules "are used to preview network behavior and do not enforce network traffic."

apiVersion: projectcalico.org/v3
kind: StagedNetworkPolicy
metadata:
  name: default.default-deny-all
  namespace: payments
spec:
  tier: default
  order: 2000
  selector: all()
  # No ingress or egress rules + both types = a previewed deny-all.
  types:
    - Ingress
    - Egress

Antrea takes a third route: enableLogging: true logs "the first packet of any traffic flow that matches this rule" to /var/log/antrea/networkpolicy/np.log, extended to standard NetworkPolicies by annotating the *Namespace* — not the policy — with networkpolicy.antrea.io/enable-logging: "true". A rollout aid, not evidence: since v1.13 it is best-effort and "therefore not meant to be used for compliance purposes", capped by default at 5000 packets per second per node.

Click to zoom

Proving Enforcement With Traffic, Not Manifests

Upstream's own verification has the right instinct and the wrong cadence: kubectl run busybox --rm -ti --image=busybox -- /bin/sh, then wget --spider --timeout=1 nginx. One packet, one verdict — by a human, when nobody doubted the policy.

NetAssert v2 turns that into a suite: a Go rewrite that "utilises the ephemeral container support in Kubernetes to verify network connectivity" between live workloads, driven by YAML, TCP and UDP only. Privilege tracks protocol: a TCP test injects only a scanner, which "requires no privileges nor any Linux capabilities"; a UDP test also injects a sniffer requiring "cap_raw capabilities", so restrictive Pod Security admission can block UDP tests. Maintained: v2.1.6 shipped 2026-08-04.

---
- name: prober-must-reach-its-own-echo
  type: k8s
  protocol: tcp
  targetPort: 8080
  timeoutSeconds: 30
  attempts: 3
  exitCode: 0
  src:
    k8sResource:
      kind: deployment
      name: netprobe
      namespace: netprobe
  dst:
    k8sResource:
      kind: deployment
      name: probe-echo
      namespace: netprobe

- name: prober-must-not-reach-payments
  type: k8s
  protocol: tcp
  targetPort: 8080
  timeoutSeconds: 30
  attempts: 3
  exitCode: 1
  src:
    k8sResource:
      kind: deployment
      name: netprobe
      namespace: netprobe
  dst:
    k8sResource:
      kind: deployment
      name: checkout
      namespace: payments

Static analysis belongs beside this, not instead of it. Policy Assistant, under kubernetes-sigs/network-policy-api, "is a static analysis tool which simulates the action of network policies for the given traffic" and needs no cluster — useful in a pull request. Its README scopes it to NetworkPolicy v1 plus AdminNetworkPolicy and BaselineAdminNetworkPolicy, and a simulator agreeing with your intent says nothing about the plugin on the node. Cyclonus now points readers there.

One shortcut deserves a warning. Cilium's drop_count_total is labelled by reason and direction and enabled in the metric catalogue; flows_to_world_total is disabled by default and counts dropped flows "if and only if the drop reason is Policy denied" unless any-drop is set. And no Cilium metrics exist until the agent runs with prometheus.enabled=true. Zero on a dashboard is equally consistent with perfect segmentation and with a metric nobody turned on.

A Continuous Segmentation Prober

The artefact worth building is small and unusual: a scheduled job that fails on an unexpected *success*. Every other harness a platform team owns alerts when something stops working. Segmentation regressions do the opposite: nothing breaks, no alert fires, and the cluster is quietly less isolated than the diagram says. The pattern is ours; upstream offers a manual probe, nothing scheduled.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: segmentation-prober
  namespace: netprobe
spec:
  schedule: "17 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 24
  failedJobsHistoryLimit: 48
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          securityContext:
            runAsNonRoot: true
            runAsUser: 65532
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: prober
              image: registry.internal/netprobe/busybox:1.37
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: [ALL]
              command: [/bin/sh, -c]
              args:
                - |
                  set -u
                  TS=$(date -u +%Y-%m-%dT%H:%M:%SZ)
                  RC=0

                  probe() {   # probe <open|closed> <host> <port>
                    if nc -w 3 "$2" "$3" </dev/null >/dev/null 2>&1
                    then got=open; else got=closed; fi
                    if [ "$got" = "$1" ]; then s=ok; else s=FAILED; RC=1; fi
                    echo "$TS $s expect=$1 got=$got $2:$3"
                  }

                  # Control paths. MUST be open, or everything below
                  # passes for the wrong reason.
                  probe open kube-dns.kube-system 53
                  probe open probe-echo.netprobe 8080

                  # Segmentation assertions. A success here IS the
                  # finding — but assert closed ONLY where something
                  # denies it: line one is the payments default-deny,
                  # the next two need a default-deny in data and an
                  # egress policy here.
                  probe closed checkout.payments 8080
                  probe closed postgres.data 5432
                  probe closed kubernetes.default 443

                  exit $RC

Three things carry the weight. Run it from a namespace out of scope for the segment you assert. Keep backoffLimit: 0 — a retry that eventually succeeds against a forbidden path is the finding, not a flake. And emit one timestamped line per path per run into append-only storage: the record is the deliverable, not the exit code, as with resilience testing as evidence.

The prober is valuable on the correct side of that boundary: a year of hourly pass/fail records per path is the continuously-generated artefact SOC 2 evidence collection is starved of, and what a segmentation tester should be handed on day one.

ClusterNetworkPolicy Changes the Rollout

The upstream API is moving. network-policy-api v0.2.0, released 2026-04-21, replaces the v1alpha1 AdminNetworkPolicy and BaselineAdminNetworkPolicy types "with a single API, an explicit Admin or Baseline tier, and rule actions Accept, Deny, and Pass". The Baseline tier is a better home for a cluster-wide floor than an object generated into every namespace. One inversion comes with it: rules "should be read as-is, i.e. there will not be any implicit isolation effects", and "CNPs with no egress rules do not affect egress traffic." An object carrying only an ingress Deny is not a default-deny.

# Requires the CRD from sigs.k8s.io/network-policy-api (release-0.2) AND
# per-CNI enablement:
#   Cilium >= 1.20:  --enable-k8s-cluster-network-policy=true
#                    (Helm k8sClusterNetworkPolicy.enabled=true)
#   Antrea >= 2.7.0: ClusterNetworkPolicy feature gate in
#                    antrea-controller.conf — disabled by default.
# Never apply without the two DNS carve-outs below.
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: default-baseline-deny
spec:
  tier: Baseline
  priority: 10
  subject:
    # Carve-out 1: kube-system out of scope, so CoreDNS does not
    # inherit the ingress Deny.
    namespaces:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: [kube-system]
  # Read as-is: no implicit isolation. Drop the egress block and egress
  # stays open, however the rules are named. namespaces: {} means pods
  # in every namespace — not host-networked pods, not the world.
  # Rule order within one object is precedence: Accept above Deny.
  ingress:
    - action: Deny
      name: deny-ingress-from-pods
      from:
        - namespaces: {}
  egress:
    # Carve-out 2: without it, every pod's egress to CoreDNS is
    # denied and name resolution dies cluster-wide.
    - action: Accept
      name: allow-dns-egress
      to:
        - pods:
            namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: kube-system
            podSelector:
              matchLabels:
                k8s-app: kube-dns
      protocols:
        - udp:
            destinationPort:
              number: 53
        - tcp:
            destinationPort:
              number: 53
    - action: Deny
      name: deny-egress-to-pods
      to:
        - namespaces: {}

The DNS carve-outs are not optional. On the one install where this object does anything — CRD present, per-CNI flag on — applying it without them denies every pod egress to CoreDNS and kills name resolution cluster-wide; the namespaced allow-dns-egress above only helps where it already exists. The enablement comment is the rest of the manifest. Cilium documents support "starting with Cilium 1.20", with --enable-k8s-cluster-network-policy set and the sigs.k8s.io/network-policy-api CRD installed. Antrea introduced it "as an Antrea Controller alpha feature in v2.7 and is disabled by default".

Three cautions. The Admin tier is not a stronger suggestion — under Cilium its rules "take precedence over all policies at the NetworkPolicy tier level ... and cannot be overridden by them", which is both the point and the outage. And the upstream implementations tracker, updated 21 April 2026, still lists ClusterNetworkPolicy only for Kube-network-policies, Kube-OVN and Calico v3.32 — it lags the vendors' own docs. And namespaces: {} peers are pods — north-south traffic, node IPs and host-networked pods stay outside this floor.

Pilot it; do not migrate to it. The ladder above works unchanged — the Baseline object simply becomes the thing you stage and probe.

The Failure Modes Worth Rehearsing

A probe suite is only as good as its cases, and the useful ones are documented ambiguities rather than obvious paths. Each deserves a named test:

• Node-local traffic. Upstream's ipBlock text carries an exception: "traffic to and from the node where a Pod is running is always allowed, regardless of the IP address of the Pod or the node." Probe from the *same* node.
• hostNetwork pods. Behaviour "is undefined", and the most common implementation "ignores hostNetwork pods when matching podSelector and namespaceSelector." Anything on the host network sits outside your model.
• Protocols beyond TCP, UDP and SCTP. A deny-all "is only guaranteed to deny TCP, UDP and SCTP connections. For other protocols, such as ARP or ICMP, the behaviour is undefined." A ping across default-deny is compliant, not a bug.
• ipBlock against intra-cluster addresses on Cilium. "By default, ipBlock rules in NetworkPolicy do not match intra-cluster IPs"; matching them requires policy-cidr-match-mode set to pods or nodes. A CIDR deny that looks cluster-wide may not be.
• Established connections across a policy change. Implementation defined. Rehearse it: open a long-lived connection, apply the deny, record what your plugin does.

It is also the honest limit of packet-level segmentation as a tenancy boundary: it constrains who talks to whom and says nothing about the API server, shared CRDs, or resource contention. Where hard tenant isolation is the requirement, the boundary belongs a layer up — at virtual control planes.

Exit Ramps: Keeping Segmentation Portable

Segmentation is unusually easy to write in a form you cannot leave. Sort what you have into three buckets early; it only gets harder as policies get better.

1. Travels intact. Standard NetworkPolicy objects, and the probe suite. The probes are the most portable asset you own: intent in workloads and ports, surviving a CNI swap unchanged and doubling as the migration acceptance test.
2. Travels with work. ClusterNetworkPolicy once it stabilises. Budget a rewrite, not a re-apply.
3. Does not travel. CiliumNetworkPolicy, Calico tiers and StagedNetworkPolicy, Antrea's logging annotations, dashboards on one vendor's metric names. All defensible, but price them as vendor-specific first.

Knowing which bucket each object sits in makes changing data plane a number rather than a discovery — the payoff for working the eBPF versus iptables decision deliberately.

The Long Game: Segmentation as a Tested Property

All of it collapses into a cadence: probes in CI on every policy change, hourly against production, and again after every CNI upgrade. That last especially: these are plugin behaviours, and a minor bump can change one without a release note you would have read.

NIST SP 800-207's sixth tenet holds that authentication and authorization are "dynamic and strictly enforced before access is allowed ... continually reevaluating trust in ongoing communication." That tenet concerns resource access generally; reading it onto L3/L4 policy is our analogy, not a NIST requirement. A control asserted once and re-read annually is not a zero-trust property — the reasoning behind workload identity verified per connection.

Versions here were checked on 2026-08-15 — Cilium v1.20.0, Calico v3.32.1, Antrea v2.7.0, NetAssert v2.1.6, network-policy-api v0.2.0 — and will drift. The cadence will not. A team that can answer "when did you last prove that namespace cannot reach the database, and what did the run say" owns its segmentation. One that answers by opening a manifest owns a document.