Kubernetes Secrets Management Is Still Broken: ESO + Short-Lived Credentials Over Vault-Everything

Every Kubernetes secrets audit ends the same way. The findings report lists etcd encryption disabled, get secrets granted to namespace-wide service accounts, and a dozen application pods carrying database passwords that were rotated once, eighteen months ago, after the previous security review. The tooling to fix all of this has been production-grade for years. The problem is not a tooling gap. It is an architecture gap — specifically, the gap between teams that installed Vault because a blog post told them to, and teams that reasoned from first principles about what a secrets threat model actually requires.

The dominant posture is: install Vault, configure sidecar injection, call it done. That posture produces a high-operational-complexity single point of failure with most of the original risk intact. Vault's token TTLs are set to 24 hours because nobody wanted to tune them. The etcd backend is still unencrypted because the Kubernetes control plane was stood up by a bootstrap script that skipped the EncryptionConfiguration step. Service accounts have secrets: get because the application developer copied a RBAC snippet from Stack Overflow in 2023 and nobody has touched it since.

This article builds the architecture that actually addresses the threat model: External Secrets Operator (ESO) as the abstraction layer with pluggable backends, etcd encryption-at-rest as the non-negotiable baseline, and dynamic short-lived credentials as the control that renders long-lived secrets structurally impossible. The goal is not compliance checkbox coverage. The goal is a cluster where a full dump of etcd produces nothing useful to an attacker, and where compromising a pod's credentials accomplishes nothing after the pod is gone.

The Kubernetes Secrets Original Sin: base64 Is Encoding, Not Encryption

When Kubernetes introduced the Secret resource type, the design decision was explicitly to not encrypt the stored data — only to base64-encode it to avoid YAML parsing edge cases. The Kubernetes documentation has said plainly for years that Secrets are not encrypted by default, and that the encoding provides no security. Despite this, a significant fraction of production clusters still store secrets in the default configuration: base64 plaintext, committed to etcd, accessible to any principal with secrets: get RBAC permission in the right namespace.

The practical consequences are not theoretical. If an attacker gains read access to etcd — through a misconfigured backup job, an exposed etcd client endpoint, or a node-level compromise — every secret in the cluster is immediately readable. The base64 decoding step is one command. This is not a Kubernetes-specific failure; it is the predictable outcome of storing credentials in a distributed key-value store without encryption. The original sin is accepting that default, and the ecosystem has built an enormous number of "secrets management" solutions on top of it without fixing the baseline.

There is a second layer of exposure that etcd encryption does not address: RBAC. A correctly encrypted etcd store is irrelevant if a service account with cluster-wide secrets: get runs in a compromised pod. The threat model has two independent axes — data at rest (etcd exposure) and data in transit/use (RBAC over-privilege plus long-lived credential lifetimes) — and solutions that only address one axis leave the other fully open.

What etcd Encryption-at-Rest Actually Protects (and What It Doesn't)

The Kubernetes EncryptionConfiguration API provides server-side encryption for specific resource types before they are written to etcd. When configured correctly with a KMS provider, the encryption keys are managed outside the cluster, meaning etcd data is useless without access to the KMS — the encrypted data and the key never co-locate. This is the correct architecture for etcd-level protection. The aescbc and secretbox providers store the key in the EncryptionConfiguration file on the control plane node, which is weaker but still meaningfully better than no encryption.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps          # add if ConfigMaps also carry sensitive data
    providers:
      - kms:
          apiVersion: v2
          name: aws-kms-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}        # fallback: allows reading pre-migration secrets

# Mount the KMS plugin socket into the kube-apiserver static pod.
# After applying, force re-encryption of existing secrets:
#   kubectl get secrets --all-namespaces -o json | kubectl replace -f -

What etcd encryption protects: offline etcd backups, direct etcd reads via misconfigured client endpoints, and backup storage exposure. What it does not protect: in-flight secret reads via the Kubernetes API by authenticated principals, pod environment variable exposure after a process dump, and secrets that applications log accidentally. Etcd encryption is a necessary baseline, not a sufficient defense. The CIS Kubernetes Benchmark v1.8 lists 5.4.1 — Ensure that secrets are not stored as environment variables and 1.2.31 — Ensure that the --encryption-provider-config argument is set as appropriate as required controls. A cluster that fails 1.2.31 fails any credible audit.

The ESO Architecture: Why a Secret Store Abstraction Layer Is Worth the Indirection

External Secrets Operator is a CNCF project (Apache 2.0 license) that introduces two CRDs: ClusterSecretStore (or namespace-scoped SecretStore) representing the credential store backend, and ExternalSecret representing a mapping from an external secret path to a Kubernetes Secret. The ESO controller watches ExternalSecret resources, reads the specified data from the configured backend, and writes or updates the corresponding Kubernetes Secret on a configured interval. Application pods continue reading Kubernetes Secrets as before — no sidecar injection, no library changes, no application-layer awareness that the credential came from Vault rather than a human-created Secret.

The abstraction is the point. A ClusterSecretStore backed by HashiCorp Vault can be replaced with one backed by AWS Secrets Manager, GCP Secret Manager, or Infisical without any change to application manifests or ExternalSecret definitions. This is the correct separation of concerns: the application declares what secret it needs, the ESO configuration declares where secrets come from, and the two are independently changeable. When HashiCorp changed Vault's license to Business Source License (BSL) in 2023, teams that had coupled their application layer directly to Vault's agent injection had a significant migration ahead of them. Teams running ESO changed one ClusterSecretStore YAML.

Click to zoom

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-cluster-store
spec:
  provider:
    vault:
      server: "https://vault.internal.example.com:8200"
      path: "secret"          # KV v2 mount
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"        # Vault auth mount
          role: "external-secrets"       # Vault role bound to ESO service account
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
      caBundle: |
        -----BEGIN CERTIFICATE-----
        # your Vault CA cert (base64-encoded PEM)
        -----END CERTIFICATE-----

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: payments
spec:
  refreshInterval: 1h          # re-sync interval; lower for dynamic creds
  secretStoreRef:
    name: vault-cluster-store
    kind: ClusterSecretStore
  target:
    name: database-credentials  # Kubernetes Secret name
    creationPolicy: Owner        # ESO owns and manages this Secret lifecycle
    deletionPolicy: Retain       # keep Secret if ExternalSecret is deleted
  data:
    - secretKey: username        # key in the Kubernetes Secret
      remoteRef:
        key: payments/database   # Vault path under the KV mount
        property: username       # field within the Vault secret
    - secretKey: password
      remoteRef:
        key: payments/database
        property: password

The refreshInterval is the first tuning decision that matters. For static credentials stored in Vault KV, an hourly or daily sync interval is reasonable. For dynamic credentials with short TTLs, the interval must be shorter than the TTL — otherwise the pod refreshes a credential that Vault has already revoked. ESO handles this gracefully: a failed sync logs an event and preserves the last known good Secret value, which is the correct behavior for availability. The operational insight is that refreshInterval and Vault ttl are two independent knobs that must be set in relationship to each other.

HashiCorp Vault: Operational Weight vs. Capability, and the BSL Procurement Question

Vault is a genuinely capable secrets platform. Its dynamic secrets engine, fine-grained policy system, audit log device, and extensive authentication backend support make it the most feature-complete option in the open-source-adjacent space. The argument here is not that Vault is the wrong tool — it is that Vault is the wrong interface for most Kubernetes workloads, and that its operational weight is frequently underestimated before the first production incident.

Operating Vault in high-availability mode requires a Raft storage cluster (or external Consul), careful management of unseal keys or auto-unseal via a KMS, explicit token renewal pipelines, and a response plan for the scenario where Vault becomes unavailable before ESO can refresh credentials. This is not an indictment of Vault — it reflects the complexity inherent in running a distributed secrets service. Teams that have absorbed this operational weight report it as manageable but non-trivial. Teams that underestimated it have experienced outages during Vault restarts because the unseal ceremony was not automated, or because a token TTL expired and the application could not restart without manual intervention.

The license question is real and should be answered before procurement, not discovered during an audit. In August 2023, HashiCorp moved Vault (and Terraform) from the Mozilla Public License to the Business Source License. The BSL permits use for non-competitive purposes but prohibits commercial competitors from using the code. For most enterprises, the BSL does not restrict operational use of Vault — but it does affect internal tooling that redistributes Vault components, and it creates a procurement conversation for compliance-sensitive environments that previously assumed perpetual open-source availability. OpenBao, the Linux Foundation fork of Vault created in response to the license change, is the Apache 2.0 alternative. ESO supports OpenBao via the same Vault provider configuration, making a backend swap possible without disrupting the ESO abstraction layer.

Dynamic Credentials: The Pattern That Eliminates Long-Lived Secrets from the Cluster

Static credentials — a database password stored in Vault KV, synced to a Kubernetes Secret, mounted into a pod — are better than plaintext environment variables and better than an unencrypted etcd Secret. They are not the ceiling. The ceiling is dynamic credentials: credentials that Vault generates on-demand, scoped to a specific role, with a time-to-live set to the pod's expected lifetime. When the pod exits, the credential expires. There is nothing to rotate, nothing to revoke, nothing to leak that is still useful after the workload ends.

Vault's database secrets engine is the canonical implementation. The pattern: Vault holds a privileged connection to the target database, and when a Kubernetes workload authenticates via K8s JWT and requests credentials, Vault creates a temporary database user with a scoped role and a TTL of, say, 15 minutes. If the pod restarts, a new credential is issued. If the pod is compromised and the credential is extracted, it is useless within 15 minutes. The application never knows the difference — it still reads a username and password from its environment.

Click to zoom

The Vault Kubernetes authentication backend is the bridge between a pod's projected service account token and Vault's policy system. A pod presents its ServiceAccount JWT to Vault's auth/kubernetes/login endpoint. Vault validates the JWT against the cluster's TokenReview API, maps the ServiceAccount to a Vault role, and returns a Vault token scoped to the policies defined for that role. This Vault token is then used to request dynamic credentials. The critical property is that this entire exchange is automated — it happens on pod startup, without human involvement, and produces a credential that exists only for the pod's operational window.

Secret Rotation Without Restarts: How ESO + Vault Dynamic Secrets Achieves Zero-Downtime Rotation

The common objection to dynamic credentials is restart sensitivity: if the credential changes, the application must reconnect to the database. For a well-written application that handles connection pool errors and reconnects gracefully, this is a non-issue — the application will naturally pick up the new credential on its next connection attempt. For legacy applications that hold a single long-lived database connection and never reconnect, this requires mitigation.

ESO addresses this with the target.creationPolicy: Merge option combined with Vault's credential renewal path. Instead of replacing the entire Kubernetes Secret (which triggers a volume remount and potential pod restart depending on the application's restart policy), ESO can update individual keys within an existing Secret. Secret volume mounts have supported kubelet-driven live reload since the early releases of Kubernetes — the kubelet syncs mounted Secret volumes on a configurable interval (default ~60 seconds) without a pod restart. This live reload applies only to secrets mounted as volumes; secrets injected as environment variables at pod start are never updated in a running pod and always require a restart. Applications that read credentials from files rather than environment variables benefit from this path: the credential updates in the mounted volume, and the application can be written or configured to detect and reload credential files on a change signal.

For applications that genuinely cannot reconnect without a restart, the operational strategy is lease duration management at the Vault layer. Set the dynamic credential TTL long enough that ESO can renew it before expiry, and configure Vault's max_ttl to enforce a hard ceiling that still prevents indefinite credential accumulation. A TTL of 24 hours with renewal at 75% of TTL (18 hours), and a max_ttl of 7 days, provides 7-day rotation guarantees without requiring application restarts on every pod startup. This is a practical degradation from the ideal but still eliminates the 90-day or 365-day static rotation problem entirely.

The capabilities work that Stribog delivers in regulated environments consistently involves this TTL tuning conversation. The default Vault settings are not optimized for Kubernetes workload lifecycles. The configuration must be intentional, documented, and tied to the threat model — and that documentation becomes the audit evidence.

Audit Evidence: What a Secrets Compliance Audit Requires and How to Produce It

A secrets compliance audit under NIS2 Article 21 security measures, SOC 2 Type II CC6.1-CC6.3, or internal security policy will ask for evidence in four categories: inventory (what secrets exist and where), access control (who can read them and under what conditions), rotation records (when was each credential last changed and what triggered the change), and breach detection (would you know if a credential was exfiltrated). Most Kubernetes clusters can partially answer the first two and cannot answer the last two at all.

Vault's audit device provides the foundation for the third and fourth categories. When configured to write to a file or syslog backend, Vault logs every auth event, every secret read, and every dynamic credential issuance with timestamp, requesting entity, and the policy path that authorized the access. This log is the rotation record — every dynamic credential issuance is an implicit rotation event with a consistent, structured timestamp. Tamper-evidence for this log comes not from the local file itself, which Vault does not write in an append-only or cryptographically chained format, but from shipping it immediately to an immutable SIEM or log management platform. An audit log that stays on the Vault host is only as trustworthy as the host. Ship it out, and alerting on unexpected secret access patterns addresses breach detection.

ESO contributes to the inventory layer. The ExternalSecret CRDs in the cluster are a machine-readable manifest of every external secret reference in use — what backend it comes from, which namespace and workload owns it, and what rotation interval applies. This is the secrets inventory that auditors request and that most teams produce by grepping through manifests the night before the audit. When the inventory is a live CRD query, it is always current.

For the regulated fintech environments where Stribog has deployed this stack, the audit evidence package from the ESO + Vault + etcd encryption architecture answers every secrets-related question with machine-readable output rather than screenshots or manual attestations. The controls are structural, not procedural. That is the difference between passing an audit and being audit-grade.

The Minimum Viable Secrets Architecture for a Regulated Workload

The full ESO + Vault dynamic credentials stack is the right end state. Getting there from a default Kubernetes installation requires sequenced steps, and the sequence matters because each step improves security posture independently — you do not need Vault operational before you can encrypt etcd, and you do not need dynamic credentials before you can deploy ESO. The maturity model below maps the progression explicitly.

Click to zoom

Level 0 and Level 1 are the current state for a significant fraction of clusters identified in security assessments — not because engineers are unaware of the problem but because the path forward appears monolithic. The sequence is: implement etcd EncryptionConfiguration immediately (it is a control-plane config change, not an application change), then tighten RBAC to eliminate cluster-wide secret access for application service accounts. These two steps together constitute Level 2 and address the most critical audit finding in nearly every cluster.

Level 3 — deploying ESO with an external store backend — is where the pluggable architecture pays off. Start with Vault KV as the backend if Vault is already in the environment, or with AWS Secrets Manager or GCP Secret Manager if the cluster runs in a cloud environment with native IAM. The ESO ClusterSecretStore provides the abstraction that makes future backend changes non-disruptive. At Level 3, secrets are no longer created manually in Kubernetes by humans with cluster-admin — every secret has a machine-readable origin in an external store, managed by ESO, with an audit log.

Level 4 — dynamic credentials with TTL equal to pod lifetime — applies to secrets that Vault's dynamic engines support: database credentials, cloud provider credentials via Vault's AWS/GCP/Azure secrets engines, PKI certificates via Vault's PKI engine, and SSH certificates. Not every secret type is dynamically generatable. Static API keys issued by third-party SaaS systems are the common exception. For those, Level 3 plus short manual rotation intervals (30 days or less, enforced by Vault's max_ttl) is the practical ceiling. The goal is zero long-lived credentials wherever the backend supports dynamic issuance, and minimized lifetime everywhere else.

The minimum viable architecture for a regulated workload is Level 2 as an immediate prerequisite and Level 3 as the target state within one sprint. Level 4 for database credentials is a two-week project given a functioning Vault or OpenBao instance. The gap between most organizations and this architecture is not technical — the tools are mature, documented, and CNCF-maintained. The gap is prioritization and sequencing. Start with etcd encryption today. Everything else follows.