Policy as Code at Cluster Scale: Building a Governance Layer with Kyverno That Outlives Any One Engineer

Every Kubernetes platform has a policy problem. In the early days it looks manageable: a handful of webhook configurations, a naming convention document someone wrote, a Slack reminder before production deploys. Then the principal engineer who built the original admission webhook leaves. The document goes stale. A team deploys a pod without resource limits, a scheduler overcommit cascade takes down a node group, and the post-mortem reveals that the control everyone assumed was enforced was enforced nowhere except in one person's memory.

This is the policy debt trap. It is not a Kubernetes problem and it is not a tooling problem. It is an organizational problem that tooling can solve — but only if you treat policies the way engineers treat code: version-controlled, tested in CI, reviewed before merge, deployed through a pipeline, and monitored in production. The Kyverno vs. OPA debate has been relitigated in every conference talk since 2021. This article skips it entirely. The question worth answering in 2026 is not which engine to choose. It is how to build a policy corpus that scales organizationally — that developers can reason about, that teams can contribute to without breaking production, and that produces audit-ready evidence automatically. The engine is 20% of the problem. The lifecycle is 80%.

What follows is the architecture of a Kyverno deployment built for the long game: one that a regulated platform team can hand to an auditor and say, with confidence, that every control is tested, versioned, and automatically enforced. Recent Kyverno releases have made this architecture cleaner than ever, with CEL policies reaching stable and native alignment with Kubernetes ValidatingAdmissionPolicies. But the architecture works across Kyverno versions — the lifecycle patterns matter more than any single release.

The Policy Debt Trap: Why Most Kubernetes Security Controls Are One Engineer's Retirement Away from Gone

The failure mode is consistent across organizations of every size. A motivated senior engineer implements a set of admission controls — ValidatingWebhookConfigurations registered by hand, or a first-generation OPA Gatekeeper deployment with a handful of ConstraintTemplates. The controls work. The engineer knows why each rule exists, knows which teams have informal exceptions, and carries the mental model that connects each policy to a specific incident or compliance requirement. Then the engineer leaves.

What remains is a set of webhook registrations that the new team is afraid to modify, a collection of ConstraintTemplates with no test suite, and a README that has not been updated since 2023. The worst outcome is not that controls break — it is that the team cannot tell whether they are working. Admission webhooks fail open by default unless you configure failurePolicy: Fail. Background scans surface violations in PolicyReport CRDs that no one reads. The controls persist as configuration archaeology: present, inherited, and no longer understood.

The solution is not better documentation. It is making the policies themselves the source of truth, in a form that a CI system can test and a GitOps controller can deploy. A policy that exists as a YAML file in a Git repository, with a kyverno test suite that runs on every pull request, is durable in a way that a Slack thread or a README never is. The institutional memory is encoded in the tests, in the PR history, and in the policy metadata that links each rule to the compliance requirement it satisfies.

Policy as Code vs. Policy as Configuration: Why the Distinction Matters

The term 'policy as code' has been diluted. At one end of the spectrum it means nothing more than storing YAML files in Git. At the other end it means applying the full software development lifecycle to policies: branching, review, automated testing, staged rollout, versioned releases, and deprecation processes. These are not the same thing, and the gap between them is where most Kyverno deployments go wrong.

Policy as configuration treats the YAML as an artifact to be deployed, like a Helm chart value or a ConfigMap. You write it, apply it, and move on. Policy as code treats the YAML as source that must be compiled, in the sense that it must be validated for correctness against known test cases before it can be trusted. The distinction becomes acute when a policy change would break an existing workload: without a test suite against known good and known bad resource fixtures, you cannot know whether a change to a validate rule regresses a team's deployment until it blocks their CI pipeline in production.

The practical implication is that every ClusterPolicy in your corpus needs a kyverno test suite in the same directory, committed together, gated by CI. This is not optional if you want a governance layer that teams can contribute to without fear. The test suite is the contract: it specifies exactly what the policy allows and denies, and any change that breaks a test case cannot merge. This is the same discipline applied to application code — it is just rarely applied to infrastructure policies.

Kyverno's Architecture: Admission, Background Scan, and the Generate/Mutate Pipeline

Before building the lifecycle, it is worth being precise about what Kyverno actually does. The architecture diagram below shows the full control loop. There are two distinct execution paths: the synchronous admission webhook, and the asynchronous background controller.

Click to zoom

The admission webhook intercepts CREATE, UPDATE, and DELETE operations against the API server and evaluates policies synchronously. A validate rule either allows or denies the request. A mutate rule patches the incoming object before it is persisted — the canonical use case being injection of resource limits, security context defaults, or required labels. A generate rule creates additional resources in response to a trigger: the most common pattern being automatic NetworkPolicy or ResourceQuota creation when a new namespace is provisioned.

The background controller runs independently of the admission path. It periodically lists existing resources against the cluster and evaluates them against policies that have background: true. The results are written to PolicyReport and ClusterPolicyReport CRDs. This is critical: resources that existed before a policy was installed are not automatically remediated at admission time. The background scan is the only way to surface pre-existing violations — and it is where most teams discover that their 'enforced' controls are actually only enforced for new resources.

A well-structured Kyverno ClusterPolicy for a regulated platform typically looks like the following. The policy enforces that every container declares resource requests and limits — the absence of which is the single most common cause of node-level overcommit incidents.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
  annotations:
    policies.kyverno.io/title: "Require Resource Limits"
    policies.kyverno.io/category: "Resource Management"
    policies.kyverno.io/severity: "high"
    policies.kyverno.io/subject: "Pod"
    policies.kyverno.io/minversion: "1.6.0"
    policies.kyverno.io/description: >
      Require all containers to declare CPU and memory requests and limits.
      Absence of limits enables unbounded resource consumption and is the
      primary cause of node-level overcommit incidents.
    compliance.stribog.io/control: "NIS2-RES-04, CIS-K8s-5.2.4"
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-container-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno
      validate:
        message: >
          Pod {{request.object.metadata.name}} has one or more containers that
          do not declare resources.requests.cpu, resources.requests.memory,
          resources.limits.cpu, or resources.limits.memory.
        pattern:
          spec:
            containers:
              - name: "*"
                resources:
                  requests:
                    memory: "?*"
                    cpu: "?*"
                  limits:
                    memory: "?*"
                    cpu: "?*"

Notice the annotation block. The compliance.stribog.io/control annotation links this policy to specific control identifiers from your compliance framework. This is not decorative. Tools like KubeVigil can query PolicyReport CRDs and cross-reference these annotations to produce per-control evidence reports for auditors — turning the policy corpus into a living compliance artifact rather than a static document.

Kyverno, CEL, and Native Kubernetes Admission APIs: The Convergence

Recent Kyverno releases promoted Common Expression Language (CEL) policies from beta to stable. This is architecturally significant. Kubernetes 1.30 introduced ValidatingAdmissionPolicy as a stable API — a native, CEL-based admission mechanism that does not require a third-party webhook. Current Kyverno can evaluate and generate both native ValidatingAdmissionPolicy objects and its own CRDs from the same policy authoring experience.

The practical implication is that organizations can write policies in Kyverno's familiar ClusterPolicy format but generate them as native Kubernetes ValidatingAdmissionPolicy objects for the rules that benefit from reduced webhook latency. CEL expressions execute in-process within the API server, with no network round-trip to the Kyverno pods. For high-frequency resources — like Pods in a busy cluster — this latency reduction is material.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root-cel
  annotations:
    policies.kyverno.io/title: "Require Non-Root Containers (CEL)"
    policies.kyverno.io/category: "Pod Security"
    policies.kyverno.io/severity: "critical"
    compliance.stribog.io/control: "NIS2-SEC-11, CIS-K8s-5.2.6"
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-run-as-non-root
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        cel:
          expressions:
            - expression: >
                object.spec.containers.all(c,
                  has(c.securityContext) &&
                  has(c.securityContext.runAsNonRoot) &&
                  c.securityContext.runAsNonRoot == true
                ) &&
                (!has(object.spec.initContainers) ||
                  object.spec.initContainers.all(c,
                    has(c.securityContext) &&
                    has(c.securityContext.runAsNonRoot) &&
                    c.securityContext.runAsNonRoot == true
                  )
                ) &&
                (!has(object.spec.ephemeralContainers) ||
                  object.spec.ephemeralContainers.all(c,
                    has(c.securityContext) &&
                    has(c.securityContext.runAsNonRoot) &&
                    c.securityContext.runAsNonRoot == true
                  )
                )
              message: >-
                All containers, initContainers, and ephemeralContainers must
                set securityContext.runAsNonRoot=true.
                Running as root is prohibited in production namespaces.

CEL expressions align directly with what Kubernetes itself uses internally for validating native API objects. Engineers who write CEL for custom resource validation in the API server can apply the same skill to Kyverno admission policies. This convergence reduces the cognitive overhead of maintaining a separate policy language and makes Kyverno a thinner abstraction layer over the platform rather than a competing system.

Building the Policy Corpus: Taxonomy, Naming, and Ownership at Scale

A policy corpus without a taxonomy is a collection of files. A taxonomy without ownership is a governance document no one reads. The two must be designed together, and both must be encoded in the repository structure and policy metadata — not in a separate document that will become stale.

The taxonomy that works at scale organizes policies by category and severity. Category maps to the compliance domain: pod-security, resource-management, network, rbac, image-integrity, namespace-hygiene. Severity is the business impact of a violation: critical (block immediately, Enforce mode), high (enforce after 30-day migration period), medium (audit mode, track in dashboard), low (informational, report only). Every policy gets both. This makes it possible to answer the question an auditor will ask: 'show me every critical control and confirm it is enforced on every cluster.'

Naming convention follows {category}-{control-name} in kebab-case: pod-security-require-non-root, resource-management-require-limits, network-default-deny-egress. The consistency matters because automated tooling — GitOps controllers, dashboard queries, kyverno test invocations — will grep on the name. Inconsistent names produce inconsistent automation.

• Repository structure: one directory per policy category; each policy has its own subdirectory containing the ClusterPolicy YAML and a kyverno-test.yaml test suite.
• Ownership annotation: policies.kyverno.io/owner: team-platform — the team accountable for the policy's correctness and the first point of contact for exception requests.
• Compliance mapping annotation: compliance.stribog.io/control: NIS2-RES-04 — links to a control registry that maps control IDs to audit evidence requirements.
• Minimum version annotation: policies.kyverno.io/minversion — prevents a policy from being deployed to a cluster running an older Kyverno version that may not support its syntax.
• Mode annotation: policies.kyverno.io/mode: Enforce|Audit — the intended production mode, distinct from validationFailureAction which controls the current deployed mode during rollout.

The mutate pipeline deserves separate treatment from validate policies, because the failure mode is different. A failing validate rule blocks a deploy with a visible error message. A failing mutate rule silently does nothing — the object is admitted without the expected patch. Always pair a mutate rule with a validate rule that confirms the mutation was applied, so the failure mode is visible rather than invisible.

# Policy 1: Mutate — inject default security context
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pod-security-inject-security-context
  annotations:
    policies.kyverno.io/title: "Inject Default Security Context"
    policies.kyverno.io/category: "pod-security"
    policies.kyverno.io/severity: "high"
spec:
  rules:
    - name: add-security-context
      match:
        any:
          - resources:
              kinds: [Pod]
      mutate:
        patchStrategicMerge:
          spec:
            containers:
              - (name): "*"
                securityContext:
                  +(allowPrivilegeEscalation): false
                  +(readOnlyRootFilesystem): true
                  +(runAsNonRoot): true
---
# Policy 2: Validate — confirm injection was applied or explicitly overridden
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pod-security-require-security-context
  annotations:
    policies.kyverno.io/title: "Require Security Context Fields"
    policies.kyverno.io/category: "pod-security"
    policies.kyverno.io/severity: "critical"
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-security-context
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "All containers must have allowPrivilegeEscalation=false, readOnlyRootFilesystem=true, and runAsNonRoot=true"
        pattern:
          spec:
            containers:
              - (name): "*"
                securityContext:
                  allowPrivilegeEscalation: false
                  readOnlyRootFilesystem: true
                  runAsNonRoot: true

Testing Policies Like Code: Kyverno CLI, Chainsaw, and PR Gates

The kyverno test command is the core of the policy testing pipeline. It evaluates a ClusterPolicy against a set of resource fixtures and an expected result manifest, confirming that the policy allows what it should allow, denies what it should deny, and mutates what it should mutate. A test suite that covers both the pass and fail paths for every rule is the contract between the policy author and every engineer whose workload will be subject to that policy.

The test fixture structure is straightforward. Each policy directory contains a kyverno-test.yaml that references resource fixture files and declares expected results. Running kyverno test . from the repository root executes every test suite and reports pass/fail with diff output on mutation failures. This is the gate that runs in CI on every pull request that modifies a policy.

name: require-resource-limits
policies:
  - require-resource-limits.yaml
resources:
  - good-pod.yaml
  - bad-pod-no-limits.yaml
  - bad-pod-no-requests.yaml
results:
  - policy: require-resource-limits
    rule: check-container-resources
    resource: good-pod
    kind: Pod
    result: pass
  - policy: require-resource-limits
    rule: check-container-resources
    resource: bad-pod-no-limits
    kind: Pod
    result: fail
  - policy: require-resource-limits
    rule: check-container-resources
    resource: bad-pod-no-requests
    kind: Pod
    result: fail

For more complex integration scenarios — particularly for generate policies that create dependent resources, or for policies that interact with existing cluster state — Kyverno's Chainsaw test framework provides end-to-end testing against a real cluster (typically a local kind cluster in CI). Chainsaw tests are declarative YAML that describe the sequence of apply operations and the expected cluster state after each step. They are slower than kyverno test but catch the class of bugs that unit tests cannot: race conditions in background scans, generate rule interactions with namespace selectors, and exception CRD precedence.

#!/usr/bin/env bash
# .github/workflows/policy-ci.yaml (excerpt — bash step)
set -euo pipefail

# Install kyverno CLI (pinned version matches cluster deployment)
KYVERNO_VERSION="v1.17.2"
curl -sSL "https://github.com/kyverno/kyverno/releases/download/${KYVERNO_VERSION}/kyverno_linux_amd64.tar.gz" \
  | tar xz kyverno
chmod +x kyverno && mv kyverno /usr/local/bin/kyverno

# Lint: confirm all YAML is parseable and ClusterPolicy schema is valid
kyverno lint policies/ --recursive

# Run unit tests: kyverno test against all policy suites
kyverno test policies/ --recursive -v 4

# Confirm every policy has a test suite (fail fast if missing)
for policy_dir in policies/*/; do
  policy_name=$(basename "$policy_dir")
  if [[ ! -f "${policy_dir}/kyverno-test.yaml" ]]; then
    echo "ERROR: ${policy_name} is missing a kyverno-test.yaml"
    exit 1
  fi
done

echo "All policy tests passed."

The diagram below shows the full policy lifecycle, including where CI testing sits in the flow and how exception requests are handled without bypassing the audit trail.

Click to zoom

Exception Management Without Loopholes: Time-Bound Exceptions with Audit Trails

Policy exceptions are not optional in a real organization. A team migrating a legacy workload will have a transitional period where they cannot comply with a new runAsNonRoot requirement. A vendor-supplied container image may not support read-only root filesystem. Pretending exceptions do not exist drives teams to bypass the enforcement layer entirely — which is worse than a structured exception because it produces no audit trail.

Kyverno's PolicyException CRD provides the correct abstraction. An exception is a Kubernetes object that grants a specific resource in a specific namespace an exemption from a named policy rule, for a defined time period, with a stated justification. The object is submitted as a pull request, reviewed by the policy-owning team, and merged to the policy repository like any other change. The Git history is the audit trail.

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: vendor-legacy-app-exception
  namespace: payments
  annotations:
    exception.stribog.io/requestor: "team-payments"
    exception.stribog.io/approver: "team-platform"
    exception.stribog.io/ticket: "JIRA-4821"
    exception.stribog.io/justification: >
      vendor/legacy-payment-processor:3.1.4 image runs as uid 0 by design.
      Vendor has committed to a non-root build in the next release (Q3 2026).
      Exception expires 2026-09-01. Team is pinned to this image version only.
    exception.stribog.io/expires: "2026-09-01"
spec:
  exceptions:
    - policyName: require-non-root-cel
      ruleNames:
        - require-run-as-non-root
  match:
    any:
      - resources:
          kinds:
            - Pod
          namespaces:
            - payments
          selector:
            matchLabels:
              app.kubernetes.io/name: legacy-payment-processor

The expiry annotation is enforced at the process level, not at the Kubernetes API level. A CI job that runs nightly queries all PolicyException objects across the cluster, compares exception.stribog.io/expires against the current date, and opens a Jira ticket (or equivalent) for any exception within 30 days of expiry. Expired exceptions are removed from the repository through a normal PR process — and if the workload has not been remediated by expiry, the exception removal PR is the forcing function that surfaces the conversation at the right organizational level.

Wiring Kyverno Reports to Your Compliance Dashboard

PolicyReport CRDs are the native output of Kyverno's background controller. A PolicyReport is scoped to a namespace and contains one result per evaluated resource per rule: the resource reference, the policy name, the rule name, the result (pass/fail/warn/skip/error), and the violation message. A ClusterPolicyReport covers cluster-scoped resources. These CRDs are machine-readable compliance evidence — the exact format that tools like KubeVigil consume when performing out-of-band audit passes against a cluster.

The compliance dashboard architecture for a multi-cluster deployment has three layers. The first layer is per-cluster: Kyverno writes PolicyReport CRDs continuously. The second layer is aggregation: a Prometheus scraper or a dedicated policy report aggregator (such as the Policy Reporter project) reads these CRDs across all clusters and exposes metrics. The third layer is presentation: a Grafana dashboard or custom compliance UI queries the metrics to produce a per-cluster, per-control, per-category view of compliance posture.

Click to zoom

The critical design decision is the independence of the audit layer. Kyverno's PolicyReport CRDs report what Kyverno has evaluated. They do not report what Kyverno has missed — gaps in policy coverage, policies set to Audit mode that should be in Enforce mode, or resources that fall into unintended namespace exclusions. An independent audit layer like KubeVigil re-evaluates the cluster against its own 110-check suite without relying on Kyverno's own reporting. The combination of Kyverno's continuous in-cluster reporting and KubeVigil's scheduled out-of-band scans provides defense in depth for the audit evidence chain.

For regulated fintech workloads specifically, the audit evidence requirement goes beyond a dashboard screenshot. An auditor needs to demonstrate that a specific control was enforced at a specific point in time against a specific resource. PolicyReport CRDs with timestamps and resource UIDs, stored in an immutable audit log (append-only object storage with write-once policy), satisfy this requirement. The timestamp on the PolicyReport, combined with the Git commit hash of the policy that produced it, gives you a complete provenance chain: which version of the policy was in effect, when it evaluated the resource, and what the result was.

Building the Governance Layer for Decades: Organizational Durability Over Tool Optimization

The Kyverno project will change. CEL is tightening the alignment with native Kubernetes admission APIs, and it is plausible that a future Kubernetes release makes a subset of Kyverno's validate capabilities redundant for simple rules. The policy engine is a commodity. What is not a commodity is the tested, annotated, ownership-tagged policy corpus that your organization has built, the CI pipeline that validates it, and the exception management process that has created three years of audit trail in your Git history.

This is what 'the long game' means in practice for governance. An organization that treats its policy corpus as a living software artifact — with ownership, with tests, with deprecation processes — can migrate between admission engines with much lower cost than an organization that has 200 manually applied ConstraintTemplates and a tribal knowledge base. The investment is in the lifecycle discipline, not in the tool.

The capabilities page describes how Stribog builds these governance layers for regulated platforms. The engagement model starts with an audit of the existing control landscape — identifying what is enforced, what is assumed to be enforced, and what is missing — and produces a prioritized remediation roadmap. The output is not a Kyverno installation. It is a policy corpus with tests, ownership, a delivery pipeline, and an exception management process that will still be running correctly long after the initial engagement ends.

Policy as code, executed rigorously, is one of the highest-leverage infrastructure investments a regulated platform team can make. The enforcement happens automatically on every admission request. The evidence is generated continuously without human intervention. The exceptions are documented and time-boxed. The corpus is tested and version-controlled. What you get — and what most teams never have — is a governance layer that an auditor can trust, that developers can understand, and that outlives any individual engineer who built it.