Kubernetes Secrets Are Still Broken: ESO Over Vault

Every Kubernetes secrets audit ends the same way. The findings report lists etcd encryption disabled, get secrets granted to namespace-wide service accounts, and a dozen application pods carrying database passwords that were rotated once, eighteen months ago, after the previous security review. The tooling to fix all of this has been production-grade for years. The problem is not a tooling gap. It is an architecture gap — specifically, the gap between teams that installed Vault because a blog post told them to, and teams that reasoned from first principles about what a secrets threat model actually requires.

The dominant posture is: install Vault, configure sidecar injection, call it done. That posture produces a high-operational-complexity single point of failure with most of the original risk intact. Vault's token TTLs are set to 24 hours because nobody wanted to tune them. The etcd backend is still unencrypted because the Kubernetes control plane was stood up by a bootstrap script that skipped the EncryptionConfiguration step. Service accounts have secrets: get because the application developer copied a RBAC snippet from Stack Overflow in 2023 and nobody has touched it since.

This article builds the architecture that actually addresses the threat model: External Secrets Operator (ESO) as the abstraction layer with pluggable backends, etcd encryption-at-rest as the non-negotiable baseline, and dynamic short-lived credentials as the control that renders long-lived secrets structurally impossible. The goal is not compliance checkbox coverage. The goal is a cluster where etcd encryption closes the offline-backup axis, and where any credential that still lands in a Kubernetes Secret (ESO) or a pod-local file (Agent) is useless once its Vault lease TTL elapses — not once a rotation ticket is filed.

The Kubernetes Secrets Original Sin: base64 Is Encoding, Not Encryption

When Kubernetes introduced the Secret resource type, the design decision was explicitly to not encrypt the stored data — only to base64-encode it to avoid YAML parsing edge cases. The Kubernetes documentation has said plainly for years that Secrets are not encrypted by default, and that the encoding provides no security. Despite this, a significant fraction of production clusters still store secrets in the default configuration: base64 plaintext, committed to etcd, accessible to any principal with secrets: get RBAC permission in the right namespace.

The practical consequences are not theoretical. If an attacker gains read access to etcd — through a misconfigured backup job, an exposed etcd client endpoint, or a node-level compromise — every secret in the cluster is immediately readable. The base64 decoding step is one command. This is not a Kubernetes-specific failure; it is the predictable outcome of storing credentials in a distributed key-value store without encryption. The original sin is accepting that default, and the ecosystem has built an enormous number of "secrets management" solutions on top of it without fixing the baseline.

There is a second layer of exposure that etcd encryption does not address: RBAC. A correctly encrypted etcd store is irrelevant if a service account with cluster-wide secrets: get runs in a compromised pod. The threat model has two independent axes — data at rest (etcd exposure) and data in transit/use (RBAC over-privilege plus long-lived credential lifetimes) — and solutions that only address one axis leave the other fully open.

What etcd Encryption-at-Rest Actually Protects (and What It Doesn't)

The Kubernetes EncryptionConfiguration API provides server-side encryption for specific resource types before they are written to etcd. When configured correctly with a KMS provider, the encryption keys are managed outside the cluster, meaning etcd data is useless without access to the KMS — the encrypted data and the key never co-locate. This is the correct architecture for etcd-level protection. The aescbc and secretbox providers store the key in the EncryptionConfiguration file on the control plane node, which is weaker but still meaningfully better than no encryption.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps          # add if ConfigMaps also carry sensitive data
    providers:
      - kms:
          apiVersion: v2
          name: aws-kms-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}        # fallback: allows reading pre-migration secrets

# Mount the KMS plugin socket into the kube-apiserver static pod.
# After applying, force re-encryption of existing secrets:
#   kubectl get secrets --all-namespaces -o json | kubectl replace -f -

What etcd encryption protects: offline etcd backups, direct etcd reads via misconfigured client endpoints, and backup storage exposure. What it does not protect: in-flight secret reads via the Kubernetes API by authenticated principals, pod environment variable exposure after a process dump, and secrets that applications log accidentally. Etcd encryption is a necessary baseline, not a sufficient defense. The CIS Kubernetes Benchmark v1.8 lists 5.4.1 — Prefer using Secrets as files over Secrets as environment variables (Manual) and 1.2.28 — Ensure that the --encryption-provider-config argument is set as appropriate (Manual) among the controls assessors will check for secrets posture. A cluster that fails 1.2.28 fails any credible audit.

The ESO Architecture: Why a Secret Store Abstraction Layer Is Worth the Indirection

External Secrets Operator is a CNCF project (Apache 2.0 license). Its core CRDs are ClusterSecretStore (or namespace-scoped SecretStore) for the credential store backend and ExternalSecret for mapping external data into a Kubernetes Secret. The controller watches ExternalSecret resources, fetches from the configured backend (or from a generator CR such as VaultDynamicSecret for non-KV engines), and writes or updates the target Secret on a configured interval. Application pods continue reading Kubernetes Secrets as before — no sidecar injection, no library changes, no application-layer awareness that the credential came from Vault rather than a human-created Secret.

The abstraction is the point. A ClusterSecretStore backed by HashiCorp Vault can be replaced with one backed by AWS Secrets Manager, GCP Secret Manager, or Infisical without any change to application manifests or ExternalSecret definitions. This is the correct separation of concerns: the application declares what secret it needs, the ESO configuration declares where secrets come from, and the two are independently changeable. When HashiCorp changed Vault's license to Business Source License (BSL) in 2023, teams that had coupled their application layer directly to Vault's agent injection had a significant migration ahead of them. Teams running ESO changed one ClusterSecretStore YAML.

Click to zoom

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-cluster-store
spec:
  provider:
    vault:
      server: "https://vault.internal.example.com:8200"
      path: "secret"          # KV v2 mount
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"        # Vault auth mount
          role: "external-secrets"       # Vault role bound to ESO service account
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
      # caBundle is format:byte — base64 of the PEM file, not armored PEM inline:
      #   caBundle: $(base64 -w0 < vault-ca.pem)
      caProvider:
        type: ConfigMap
        name: vault-ca
        key: ca.crt
        namespace: external-secrets

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: payments
spec:
  refreshInterval: 1h          # re-sync interval for this KV-backed ExternalSecret
  secretStoreRef:
    name: vault-cluster-store
    kind: ClusterSecretStore
  target:
    name: database-credentials  # Kubernetes Secret name
    creationPolicy: Owner        # ESO owns and manages this Secret lifecycle
    deletionPolicy: Retain       # keep Secret if ExternalSecret is deleted
  data:
    - secretKey: username        # key in the Kubernetes Secret
      remoteRef:
        key: payments/database   # Vault path under the KV mount
        property: username       # field within the Vault secret
    - secretKey: password
      remoteRef:
        key: payments/database
        property: password

The refreshInterval is the first tuning decision that matters on a KV-backed ExternalSecret like the one above. For static credentials in Vault KV, an hourly or daily re-sync is reasonable: ESO re-reads the same path and rewrites the target Secret. A failed sync logs an event and preserves the last known good Secret value, which is the correct behavior for availability. That interval is not how you reach dynamic credentials. Database, PKI, and cloud engines are not available on the Vault provider's KV remoteRef path — shortening refreshInterval here only re-polls KV faster. Dynamic issuance under ESO is a separate CR path (VaultDynamicSecret + dataFrom.generatorRef, next), where refreshInterval must sit below the issued lease TTL so the controller re-issues before Vault revokes the prior lease. On either path, refreshInterval and Vault ttl are independent knobs; they only couple correctly when the CR type matches the engine.

The Vault provider on ClusterSecretStore/SecretStore only reads the KV secrets engine. Dynamic engines (database, PKI, cloud credentials) need a VaultDynamicSecret generator referenced from ExternalSecret.spec.dataFrom[].sourceRef.generatorRef, not a shorter refreshInterval on a KV remoteRef.

apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
metadata:
  name: payments-db-dynamic
  namespace: payments
spec:
  path: database/creds/payments-role   # Vault database secrets engine path
  method: GET
  provider:
    server: "https://vault.internal.example.com:8200"
    auth:
      kubernetes:
        mountPath: kubernetes
        # Same Vault role name as ClusterSecretStore above.
        # VaultDynamicSecret is namespaced: serviceAccountRef.namespace is
        # ignored, so this resolves to payments/external-secrets. Create that
        # SA (or reuse one) and extend bound_service_account_namespaces on the
        # Vault role (or add a second role) so payments is allowed.
        role: external-secrets
        serviceAccountRef:
          name: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db-dynamic
  namespace: payments
spec:
  refreshInterval: 10m                 # must be shorter than the role TTL
  target:
    name: payments-db-dynamic
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: VaultDynamicSecret
          name: payments-db-dynamic

HashiCorp Vault: Operational Weight vs. Capability, and the BSL Procurement Question

Vault is a genuinely capable secrets platform. Its dynamic secrets engine, fine-grained policy system, audit log device, and extensive authentication backend support make it the most feature-complete option in the open-source-adjacent space. The argument here is not that Vault is the wrong tool — it is that Vault is the wrong interface for most Kubernetes workloads, and that its operational weight is frequently underestimated before the first production incident.

Operating Vault in high-availability mode requires a Raft storage cluster (or external Consul), careful management of unseal keys or auto-unseal via a KMS, explicit token renewal pipelines, and a response plan for the scenario where Vault becomes unavailable before ESO can refresh credentials. This is not an indictment of Vault — it reflects the complexity inherent in running a distributed secrets service. Teams that have absorbed this operational weight report it as manageable but non-trivial. Teams that underestimated it have experienced outages during Vault restarts because the unseal ceremony was not automated, or because a token TTL expired and the application could not restart without manual intervention.

The license question is real and should be answered before procurement, not discovered during an audit. In August 2023, HashiCorp moved Vault (and Terraform) from the Mozilla Public License to the Business Source License. The BSL permits use for non-competitive purposes but prohibits commercial competitors from using the code. For most enterprises, the BSL does not restrict operational use of Vault — but it does affect internal tooling that redistributes Vault components, and it creates a procurement conversation for compliance-sensitive environments that previously assumed perpetual open-source availability. OpenBao, the Linux Foundation fork of Vault created in response to the license change, is the MPL 2.0 (OSI-approved) alternative — though an existing Vault cluster only qualifies for the documented in-place migration path if its version, storage backend and seal type clear three specific gates. ESO supports OpenBao via the same Vault provider configuration, making a backend swap possible without disrupting the ESO abstraction layer.

Dynamic Credentials: The Pattern That Eliminates Long-Lived Secrets from the Cluster

Static credentials — a database password stored in Vault KV, synced to a Kubernetes Secret, mounted into a pod — are better than plaintext environment variables and better than an unencrypted etcd Secret. They are not the ceiling. The ceiling is dynamic credentials: credentials that Vault generates on-demand, scoped to a specific role, with a short lease TTL matched to risk (minutes to hours, not months). Vault leases expire when their TTL elapses — not because a pod exits. Under Vault Agent, operators often size that TTL to the pod's expected lifetime so a dead pod's credential is useless soon after. Under ESO Level 4, the controller re-issues on each ExternalSecret refreshInterval into a short-lived Kubernetes Secret, and the prior lease lapses on its own TTL; pod exit alone does not revoke it. Either path leaves nothing long-lived to rotate by ceremony, and nothing useful to an attacker after the lease window.

Vault's database secrets engine is the canonical implementation. The pattern: Vault holds a privileged connection to the target database, and when a caller authenticates via Kubernetes JWT and requests credentials, Vault creates a temporary database user with a scoped role and a TTL of, say, 15 minutes. If the credential is re-issued (pod restart under Vault Agent, or ESO generator refresh), a new user is created. If the pod is compromised and the credential is extracted, it is useless within 15 minutes. The application still reads a username and password from a file or volume (prefer files over env vars — CIS 5.4.1) — under Agent often pod-local injection, under ESO a short-lived Kubernetes Secret; only the issuer and lifetime change.

Click to zoom

The Vault Kubernetes authentication backend is the bridge between a projected service account token and Vault's policy system. A workload (or the ESO controller acting for it) presents a ServiceAccount JWT to Vault's auth/kubernetes/login endpoint. Vault validates the JWT against the cluster's TokenReview API, maps the ServiceAccount to a Vault role, and returns a Vault token scoped to the policies defined for that role. That token is then used to request dynamic credentials. With Vault Agent, the exchange often runs in-pod on startup. With ESO at Level 4, the same Vault auth model runs in the controller: a VaultDynamicSecret generator issues the credential and an ExternalSecret materializes it as a Kubernetes Secret via dataFrom.sourceRef.generatorRef — the pod still consumes short-lived credentials, but re-issue cadence is the ExternalSecret's refreshInterval, not a pod-local renew loop.

Secret Rotation Without Restarts: How ESO + Vault Dynamic Secrets Achieves Zero-Downtime Rotation

The common objection to dynamic credentials is restart sensitivity: if the credential changes, the application must reconnect to the database. For a well-written application that handles connection pool errors and reconnects gracefully, this is a non-issue — the application will naturally pick up the new credential on its next connection attempt. For legacy applications that hold a single long-lived database connection and never reconnect, this requires mitigation.

With the default target.creationPolicy: Owner, ESO rewrites the target Secret's data in place; mounted volumes pick up the new bytes without a remount or pod restart. Use creationPolicy: Merge only when the Secret must already exist and carry keys ESO does not own — it is not a restart-avoidance mechanism. Secret volume mounts receive updates via the kubelet (default configMapAndSecretChangeDetectionStrategy: Watch) without a pod restart. This live reload applies only to secrets mounted as volumes; secrets injected as environment variables at pod start are never updated in a running pod and always require a restart. Applications that read credentials from files rather than environment variables benefit from this path: the credential updates in the mounted volume, and the application can be written or configured to detect and reload credential files on a change signal.

For applications that genuinely cannot reconnect without a restart, set the dynamic credential TTL longer than pod churn and, on the ESO Level 4 path (VaultDynamicSecret + ExternalSecret dataFrom.generatorRef), set refreshInterval comfortably below that TTL so each controller sync *re-issues* a fresh credential into the Kubernetes Secret before the prior lease expires (ESO does not call Vault lease renew; each generator invocation produces a new set of values). Superseded leases remain valid until their own TTL lapses unless something revokes them. If you need true lease renewal at a fraction of TTL, that is Vault Agent / application-side renew, not ESO.

The capabilities work that Stribog delivers in regulated environments consistently involves this TTL tuning conversation. The default Vault settings are not optimized for Kubernetes workload lifecycles. The configuration must be intentional, documented, and tied to the threat model — and that documentation becomes the audit evidence.

Audit Evidence: What a Secrets Compliance Audit Requires and How to Produce It

A secrets compliance audit under NIS2 Article 21 security measures, SOC 2 Type II CC6.1-CC6.3, or internal security policy will ask for evidence in four categories: inventory (what secrets exist and where), access control (who can read them and under what conditions), rotation records (when was each credential last changed and what triggered the change), and breach detection (would you know if a credential was exfiltrated). Most Kubernetes clusters can partially answer the first two and cannot answer the last two at all.

Vault's audit device provides the foundation for the third and fourth categories. When configured to write to a file or syslog backend, Vault logs every auth event, every secret read, and every dynamic credential issuance with timestamp, requesting entity, and the policy path that authorized the access. This log is the rotation record — every dynamic credential issuance is an implicit rotation event with a consistent, structured timestamp. Tamper-evidence for this log comes not from the local file itself, which Vault does not write in an append-only or cryptographically chained format, but from shipping it immediately to an immutable SIEM or log management platform. An audit log that stays on the Vault host is only as trustworthy as the host. Ship it out, and alerting on unexpected secret access patterns addresses breach detection.

ESO contributes to the inventory layer. The ExternalSecret CRDs in the cluster are a machine-readable manifest of every external secret reference in use — what backend it comes from, which namespace and workload owns it, and what rotation interval applies. This is the secrets inventory that auditors request and that most teams produce by grepping through manifests the night before the audit. When the inventory is a live CRD query, it is always current.

The audit question is never 'do you have a secrets manager?' It is always 'can you show me the last time credential X was rotated, who had access to it, and what would have alerted you if it had been read by an unauthorized principal?' Vault answers the rotation and access-history questions. ESO answers the inventory question. etcd encryption answers the unasked question: 'what would an attacker get from your backup storage?'

For a regulated fintech environment built this way, the audit evidence package from the ESO + Vault + etcd encryption architecture answers every secrets-related question with machine-readable output rather than screenshots or manual attestations. The controls are structural, not procedural. That is the difference between passing an audit and being audit-grade.

The Minimum Viable Secrets Architecture for a Regulated Workload

The full ESO + Vault dynamic credentials stack is the right end state. Getting there from a default Kubernetes installation requires sequenced steps, and the sequence matters because each step improves security posture independently — you do not need Vault operational before you can encrypt etcd, and you do not need dynamic credentials before you can deploy ESO. The maturity model below maps the progression explicitly.

Click to zoom

Level 0 and Level 1 are the current state for a significant fraction of clusters identified in security assessments — not because engineers are unaware of the problem but because the path forward appears monolithic. The sequence is: implement etcd EncryptionConfiguration immediately (it is a control-plane config change, not an application change), then tighten RBAC to eliminate cluster-wide secret access for application service accounts. These two steps together constitute Level 2 and address the most critical audit finding in nearly every cluster.

Level 3 — deploying ESO with an external store backend — is where the pluggable architecture pays off. Start with Vault KV as the backend if Vault is already in the environment, or with AWS Secrets Manager or GCP Secret Manager if the cluster runs in a cloud environment with native IAM. The ESO ClusterSecretStore provides the abstraction that makes future backend changes non-disruptive. At Level 3, secrets are no longer created manually in Kubernetes by humans with cluster-admin — every secret has a machine-readable origin in an external store, managed by ESO, with an audit log.

Level 4 — short-lived dynamic credentials with zero long-lived static secrets — applies to secrets that Vault's dynamic engines support: database credentials, cloud provider credentials via Vault's AWS/GCP/Azure secrets engines, PKI certificates via Vault's PKI engine, and SSH certificates. Lease TTL is matched to risk and the workload window; under Vault Agent that often means sizing TTL near expected pod lifetime, while under ESO re-issue cadence is the ExternalSecret's refreshInterval (keep it below the lease TTL) via a VaultDynamicSecret generator, not a tighter poll on a KV remoteRef. Level 4 under ESO is that generator wired through ExternalSecret.spec.dataFrom[].sourceRef.generatorRef, so each refresh re-issues from the dynamic engine rather than re-reading a static KV path. Not every secret type is dynamically generatable. Static API keys issued by third-party SaaS systems are the common exception. For those, Level 3 plus short *manual* rotation intervals (30 days or less), tracked against KV metadata (created_time / your change ticket), is the practical ceiling—Vault max_ttl / max_lease_ttl bound leases and tokens, not a static value in KV. Where the backend is a database Vault manages, prefer a static role with a configured rotation period instead. The goal is zero long-lived credentials wherever the backend supports dynamic issuance, and minimized lifetime everywhere else.

The minimum viable architecture for a regulated workload is Level 2 as an immediate prerequisite and Level 3 as the target state within one sprint. Level 4 for database credentials is a two-week project given a functioning Vault or OpenBao instance. The gap between most organizations and this architecture is not technical — the tools are mature, documented, and CNCF-maintained. The gap is prioritization and sequencing. Start with etcd encryption today. Everything else follows.