Internal PKI with step-ca and cert-manager: Private ACME

Every serious Kubernetes estate ends up with two populations of certificates. The first is small and well-understood: the handful of public names terminating browser traffic, which Let's Encrypt automated so thoroughly that most teams have forgotten certificates were ever a manual job. The second is an order of magnitude larger and almost always worse-managed — the internal names: service-to-service mTLS, database and etcd peer certs, fail-closed admission webhooks, brokers, admin consoles on .internal domains. These are the certificates that expire at 03:00 on a Sunday and take a payment path down with them.

The reflex is to reach for the same public CA that solved the first problem. It does not work: a public CA will not sign a name it cannot validate, cannot reach an air-gapped cluster, and — if it did sign — would publish that hostname to a permanent public log. The usual fallback is a shell script with openssl and a ten-year self-signed root nobody can rotate. This article describes the third option: run your own ACME CA with step-ca, point cert-manager at it, and get the Let's Encrypt operational model for infrastructure that never touches the public internet.

Three problems a public CA cannot solve for internal names

The first is disclosure. Since April 2018, Chrome has required every publicly-trusted TLS certificate to be logged to Certificate Transparency logs — the append-only, publicly-readable structure in RFC 6962. CT is an unambiguous win for the public web: it is how mis-issuance gets caught. It is also a permanent public record of every name in every certificate. Put payments-reconciliation.internal.example.com in a publicly-trusted certificate and you have published the existence, naming convention, and rough function of an internal service to anyone running a CT-log scraper — attackers do, because enumerating internal topology from public logs costs nothing. For an internal name, transparency is not a feature but a leak.

The second is reachability. Every public ACME challenge requires the CA to reach you: http-01 needs an inbound request to port 80 on a public address, dns-01 a public TXT record. An air-gapped estate satisfies neither — and building an egress exception for your CA is precisely the hole the air gap existed to prevent.

The third is that the lifetime is no longer yours to choose. In April 2025 the CA/Browser Forum approved Ballot SC-081v3, scheduling reductions to the maximum validity of publicly-trusted TLS certificates: from 398 days to today's 200-day maximum (in force since 15 March 2026), then 100 days on 15 March 2027 and 47 days on 15 March 2029. For public certificates this is correct and overdue. But it means your renewal cadence is now set by a vendor ballot. Internal PKI should take that number from your own threat model — and for workloads that can rotate in seconds, the right answer is very often far shorter than 47 days.

The hierarchy: an offline root, an online intermediate

The design decision separating a credible internal PKI from a liability is the two-tier hierarchy. The root key signs only intermediate CA certificates — perhaps once every few years, in a deliberate ceremony. The rest of the time it is powered off: on a YubiKey in a safe, or an HSM that is not network-attached. The intermediate lives online inside step-ca, signing thousands of short-lived leaves a day.

The reason is blast radius. An online signing key is reachable from the network by definition, so its compromise is a scenario to plan for. If that key is the root, every trust store is wrong, and fixing it means redistributing a new root to every workload, laptop, and appliance at once. If it is an intermediate, you revoke it, sign a new one from the offline root, and reissue — while every client's trust store, pinning only the root, stays valid. The two-tier split converts an estate-wide catastrophe into a bad afternoon.

# 1. Generate root and intermediate keys on a PKCS#11 HSM (requires the step-kms plugin).
export PKCS_URI='pkcs11:module-path=/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so;token=YubiHSM'
step kms create --json --kms "$PKCS_URI" "pkcs11:id=7331;object=root-ca"
step certificate create --profile root-ca \
  --kms "$PKCS_URI" --key "pkcs11:id=7331;object=root-ca" \
  "Example Internal Root CA" root_ca.crt
step kms create --json --kms "$PKCS_URI" "pkcs11:id=7332;object=intermediate-ca"
step certificate create --profile intermediate-ca \
  --kms "$PKCS_URI" --ca-kms "$PKCS_URI" \
  --ca root_ca.crt --ca-key "pkcs11:id=7331;object=root-ca" \
  --key "pkcs11:id=7332;object=intermediate-ca" \
  "Example Internal Intermediate CA" intermediate_ca.crt

# 2. Initialise a basic step-ca config, install the PEMs, then point ca.json at the HSM intermediate key.
step ca init \
  --name        "Example Internal PKI" \
  --dns         "step-ca.pki.svc.cluster.local" \
  --address     ":9000" \
  --provisioner "platform@example.com"
cp root_ca.crt intermediate_ca.crt "$(step path)/certs"
# Edit $(step path)/config/ca.json so the intermediate signing key is the HSM object:
#   "root": "…/certs/root_ca.crt",
#   "crt":  "…/certs/intermediate_ca.crt",
#   "key":  "pkcs11:id=7332;object=intermediate-ca",
#   "kms":  { "type": "pkcs11", "uri": "pkcs11:module-path=…;token=YubiHSM" }

# 3. Verify the chain before anything depends on it.
step certificate inspect --short $(step path)/certs/intermediate_ca.crt

step-ca supports pkcs11, yubikey, awskms, cloudkms, and azurekms as key backends — a sovereignty decision as much as a security one. An HSM or YubiKey keeps the root inside hardware you hold; a cloud KMS puts ultimate authority inside a provider's control plane. Both are defensible, but only one survives "what happens to our internal PKI if this account is suspended?" For an estate premised on not depending on infrastructure you do not control, the hardware answer is the consistent one.

Click to zoom

step-ca as an ACME server on your own network

The pivotal property of step-ca is that it implements RFC 8555 — the same ACME protocol Let's Encrypt speaks. The integration surface becomes a protocol with several mature independent implementations, not a vendor SDK. Every ACME client already written — cert-manager, certbot, acme.sh, Caddy, Traefik — works against your CA with one change to the directory URL. You are not adopting a product; you are standing up an endpoint that speaks an IETF standard.

step-ca (Apache 2.0, currently v0.30.2) supports http-01, dns-01, and tls-alpn-01, plus device-attest-01 for hardware-attested enrolment — still an IETF draft rather than a ratified RFC, and disabled by default. Inside a cluster, http-01 is usually right: the proof-of-control loop a public CA must run across the internet runs entirely within your own pod network.

step ca provisioner add acme --type ACME

# Lifetime from your threat model, not a browser vendor's ballot.
# 24h default, 72h ceiling: short enough that a leaked key expires
# before it is useful, long enough to survive a weekend outage.
step ca provisioner update acme \
  --x509-default-dur 24h \
  --x509-min-dur     12h \
  --x509-max-dur     72h

# Equivalent claims block in ca.json:
#   "claims": {
#     "minTLSCertDuration":     "12h",
#     "maxTLSCertDuration":     "72h",
#     "defaultTLSCertDuration": "24h"
#   }

That lifetime choice is the whole argument for owning the CA, made concrete. A 24-hour certificate is not merely shorter — it changes what revocation means. Public PKI's revocation story has always been weak: CRLs and OCSP are checked inconsistently and fail open more often than anyone admits. A certificate that expires tomorrow needs no revocation infrastructure, because expiry *is* the revocation mechanism — the one that never fails open. The public web cannot get there quickly; your internal estate can, because you control both ends.

cert-manager as an ordinary ACME client

cert-manager graduated from the CNCF in 2024 and is the de facto certificate controller for Kubernetes; the current stable line is 1.21.x. Pointing it at a private ACME server is deliberately unremarkable — same ACME issuer type, different directory URL, plus fields public CAs do not need. Since cert-manager 1.11, spec.acme.caBundle carries the base64-encoded PEM of the CA that signed the ACME endpoint's own certificate, letting the controller trust an endpoint no public root vouches for.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-acme
spec:
  acme:
    # Your CA, on your network. No egress required.
    server: https://step-ca.pki.svc.cluster.local:9000/acme/acme/directory
    email: platform@example.com
    privateKeySecretRef:
      name: internal-acme-account-key
    # Base64 PEM of your root — trusts an endpoint no public root
    # vouches for. Available since cert-manager 1.11.
    caBundle: <base64-encoded PEM of root_ca.crt>
    # Required for Certificate.spec.duration to be sent as ACME notAfter.
    # step-ca honours notBefore/notAfter; public CAs like Let's Encrypt do not.
    enableDurationFeature: true
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: payments-api-tls
  namespace: payments
spec:
  secretName: payments-api-tls
  duration: 24h       # inside the provisioner's min/max window
  renewBefore: 8h     # renew at 2/3 of lifetime; room to retry
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always   # new key each renewal, not just a new cert
  dnsNames:
    - payments-api.payments.svc.cluster.local
    - payments-api.internal.example.com
  issuerRef:
    name: internal-acme
    kind: ClusterIssuer

Two fields deserve more attention than they get. renewBefore must leave genuine slack — cert-manager renews at that threshold then retries with backoff, so a value leaving only minutes turns a transient CA outage into an expiry. And since cert-manager v1.18.0, rotationPolicy: Always is the default (set it explicitly so the intent stays visible — and on v1.17 and earlier, where the default is still Never, omitting it means cert-manager reuses the existing private key on every renewal, so a key that leaked in January is still the key in December). Short lifetimes only bound a key compromise if the key actually changes.

Distributing trust is the hard half

Issuing certificates is the easy half. The hard half is trust distribution: every client verifying a certificate signed by your root needs that root in its trust store, in the right namespace, updated when the root rotates. Hand-rolled internal PKIs collapse here — not at issuance, but when someone must add a root to four hundred namespaces and finds there is no mechanism.

trust-manager, a sibling project to cert-manager, exists for exactly this: it watches a source of CA certificates and materialises them as ConfigMaps or Secrets in every namespace matching a selector. The API is in motion — the Bundle resource in trust.cert-manager.io/v1alpha1 is being superseded by a cluster-scoped ClusterBundle in trust-manager.io/v1alpha2, announced September 2025. Pin manifests to the version you run and plan that migration rather than meeting it mid-upgrade.

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: internal-ca-bundle
spec:
  sources:
    # Your internal root — public certificate only. Never the key.
    - secret:
        name: internal-root-ca
        key: ca.crt
    # Keep the public roots, or every egress call fails verification.
    - useDefaultCAs: true
  target:
    configMap:
      key: ca-bundle.crt
    namespaceSelector:
      matchLabels:
        trust: internal-pki

The failure modes that actually bite

A private CA moves risk from a vendor's operations team onto yours — worth doing only when failure modes are named in advance and instrumented before anything depends on them.

• The intermediate expires and nobody notices. The most common internal-PKI outage. Leaves renew daily, so automation looks healthy until the intermediate signing them expires — years after anyone who set it up moved on. Alert on the issuing chain, not just leaves.
• The root expires. Same failure, decade-long fuse, estate-wide blast radius. Root rotation must be rehearsed before it is needed; an unexecuted runbook is a hypothesis.
• A trust store somewhere never got the bundle. A JVM truststore, a Go binary with an embedded CA pool, an appliance with a web UI. trust-manager covers Kubernetes namespaces only — the rest is your inventory.
• The CA's own database is not backed up. step-ca's store holds the issuance and revocation record. That is audit evidence; losing it means losing the ability to answer "what did we issue, to whom, and when."
• cert-manager renews faster than the CA can sign. With 24-hour lifetimes across a large estate, issuance volume runs orders of magnitude above a public-CA workload. Load-test at your real certificate count first.

All five are observable, so all five belong on the self-hosted observability stack that carries the estate. cert-manager exports certmanager_certificate_expiration_timestamp_seconds and certmanager_certificate_ready_status.

groups:
  - name: internal-pki
    rules:
      # Renewal is failing while the cert is still valid — the window in
      # which the problem is still cheap to fix.
      - alert: CertificateNotReady
        expr: certmanager_certificate_ready_status{condition="False"} == 1
        for: 1h
        labels: { severity: critical }

      # Last line of defence on the leaf.
      - alert: CertificateExpiringSoon
        expr: |
          (certmanager_certificate_expiration_timestamp_seconds - time()) / 3600 < 8
        for: 15m
        labels: { severity: warning }

      # The intermediate itself — scrape the intermediate PEM (e.g. x509-certificate-exporter),
      # not a blackbox probe of the CA endpoint. probe_ssl_earliest_cert_expiry is the minimum
      # NotAfter across the whole presented chain and is dominated by step-ca's own ≤24h leaf.
      - alert: IssuingChainExpiringSoon
        expr: (x509_cert_not_after{secret_name="intermediate-ca",secret_key="tls.crt"} - time()) / 86400 < 90
        for: 1h
        labels: { severity: critical }

Where a private CA does not belong

Sovereignty is practised in proportion, and a private CA is not a universal upgrade. Public names still belong on a public ACME CA — an internal root gains nothing for a hostname unmodified browsers must validate, and costs support the moment a customer's device distrusts it. The ACME Renewal Information extension, RFC 9773, is making public renewal more robust still.

Click to zoom

The subtler boundary is workload identity. SPIFFE and SPIRE solve an adjacent but different problem: they issue attested identities based on what a workload *is* — node, service account, image — rather than what hostname it answers to, with SVIDs rotating in minutes. A private PKI issues certificates for names. The two compose cleanly; most mature estates run both: SPIRE inside the mesh, step-ca for hostname TLS that ingress, databases, and appliances speak. Beneath both, an eBPF dataplane like Cilium enforces identity-aware policy — the certificate proves who you are, the dataplane decides who you may reach.

And the honest early exit on the decision flow's critical path: with no team to own key ceremonies, intermediate rotation, and expiry alerting, a managed CA remains the right answer. An internal root that expires because its creator left is worse than any amount of CT-log disclosure. Build the capability first, then take the responsibility.

Audit-grade means hardening the CA itself

A certificate authority is the highest-value target in an estate: whoever controls it can mint an identity for anything. Both projects' advisory history is instructive rather than disqualifying. step-ca has carried authentication-bypass issues permitting issuance while skipping authorization — CVE-2025-44005 in the ACME and SCEP provisioners, and CVE-2026-30836 in SCEP's UpdateReq handling, fixed in 0.30.0. cert-manager shipped an advisory where the default edit ClusterRole let namespace users create Challenge and Order resources with attacker-controlled solver configuration, risking DNS-provider credential disclosure.

None of that argues against self-hosting; every CA has an advisory history, and the ones that publish theirs are the ones you can audit. Treat the CA as tier-zero: disable every unused provisioner — SCEP especially, the surface behind both step-ca CVEs above — patch promptly, isolate it under a Kyverno policy set that blocks workloads from creating issuers, and verify the image's signature and SBOM before it runs. A supply-chain compromise of the binary holding your intermediate key is the worst version of this story.

The audit-grade payoff is the issuance record. Because step-ca writes every issuance to its database, you can answer — from your own systems, without a support ticket — which certificates exist, which workload requested each, and when each expires. That is the evidence an auditor asks for under NIS2 and DORA's third-party evidence standard, held rather than requested. The logic making secrets a self-hosted concern applies with more force here: a secret is a credential; a CA manufactures credentials.

The exit ramp and the long game

The optionality argument is unusually clean, because the exit ramp is the protocol itself. Moving from step-ca to another private or commercial ACME CA, or back to a public CA for a name that became public, is a change to server and caBundle. Workloads never know which CA signed them; they read a kubernetes.io/tls Secret either way. That is the opposite of usual PKI lock-in — traditional enterprise PKI binds you through enrolment agents and per-certificate pricing that makes short lifetimes economically irrational. An Apache-2.0 CA speaking an IETF standard has no such incentive, which is why a 24-hour default is even a conversation: depend on protocols, not products.

The long game is where internal PKI stops being infrastructure and becomes institutional memory. A trust hierarchy is a decade-scale commitment — longer than the workloads it protects, longer than the Kubernetes distribution beneath it, often longer than the team that stood it up. The root you generate this quarter will still sign intermediates after the architecture has been replaced twice. That is why it deserves the offline key, the rehearsed rotation, and the alert on the chain rather than the leaf. Certificates are the infrastructure whose neglect is scheduled in advance, to the second, by the certificate itself.

That pattern is worth naming on its own terms: a script-and-wiki-page internal CA where nothing alerts on the intermediate, so an absence of incidents gets mistaken for evidence that everything is fine. It isn't evidence of anything — an unmonitored certificate authority is a countdown that has not gone off yet, not a system that has been proven safe. Leaves renew; the intermediate is the one nobody is watching.

That is the sovereign internal PKI in one sentence: not a cheaper CA, but one whose root you hold, whose lifetimes come from your threat model, and whose exit is a configuration change. step-ca and cert-manager are both open, mature, and in front of you. The decision that remains is the one this practice keeps returning to — whether you own the systems you depend on, or keep renting the one that decides who your systems are.