Your Own ACME: Sovereign Internal PKI with step-ca and cert-manager

Every serious Kubernetes estate ends up with two populations of certificates. The first is small and well-understood: the handful of public names terminating browser traffic, which Let's Encrypt automated so thoroughly that most teams have forgotten certificates were ever a manual job. The second is an order of magnitude larger and almost always worse-managed — the internal names. Service-to-service mTLS, Postgres client certificates, etcd peer certs, the admission webhook that fails closed, the Kafka broker listener, the admin console on a .internal domain that resolves nowhere public. These are the certificates that expire at 03:00 on a Sunday and take a payment path down with them.

The reflex is to reach for the same public CA that solved the first problem. It does not work, for structural reasons: a public CA will not sign a name it cannot validate, cannot reach an air-gapped cluster, and — if it did sign — would publish that hostname to a permanent public log. The usual fallback is a shell script with openssl and a ten-year self-signed root nobody can rotate. This article describes the third option: run your own ACME certificate authority with step-ca, point cert-manager at it, and get the Let's Encrypt operational model for infrastructure that will never touch the public internet.

Three problems a public CA cannot solve for internal names

The first is disclosure. Since April 2018, Chrome has required every publicly-trusted TLS certificate to be logged to Certificate Transparency logs — the append-only, publicly-readable structure specified in RFC 6962. CT is an unambiguous win for the public web: it is how mis-issuance gets caught. It is also a permanent public record of every name in every certificate. Put payments-reconciliation.internal.example.com in a publicly-trusted certificate and you have published the existence, naming convention, and rough function of an internal service to anyone running a CT-log scraper — and attackers do run them, because enumerating a target's internal topology from public logs is reconnaissance that costs nothing. For an internal name, transparency is not a feature but a leak.

The second is reachability. Every public ACME challenge requires the CA to reach you: http-01 needs an inbound request to port 80 on a public address, dns-01 a public TXT record. An air-gapped estate satisfies neither — and building an egress exception for your CA is precisely the hole the air gap existed to prevent.

The third is that the lifetime is no longer yours to choose. In April 2025 the CA/Browser Forum approved Ballot SC-081v3, scheduling reductions to the maximum validity of publicly-trusted TLS certificates: from today's 398 days to 200 days on 15 March 2026, 100 days on 15 March 2027, and 47 days on 15 March 2029. For public certificates this is correct and overdue. But it means your renewal cadence is now set by a vendor ballot. Internal PKI should take that number from your own threat model — and for workloads that can rotate in seconds, the right answer is very often far shorter than 47 days.

The hierarchy: an offline root, an online intermediate

The design decision separating a credible internal PKI from a liability is the two-tier hierarchy. The root key signs exactly one kind of thing — intermediate CA certificates — perhaps once every few years, in a deliberate ceremony. The rest of the time it is powered off: on a YubiKey in a safe, or an HSM that is not network-attached. The intermediate lives online inside step-ca, signing thousands of short-lived leaves a day.

The reason is blast radius. An online signing key is reachable from the network by definition, so its compromise is a scenario to plan for. If the compromised key is the root, every trust store in the estate is wrong, and fixing it means physically redistributing a new root to every workload, laptop, and appliance at once. If it is an intermediate, you revoke it, sign a new one from the offline root, and reissue — while every client's trust store, pinning only the root, stays valid throughout. The two-tier split converts an estate-wide catastrophe into a bad afternoon.

# 1. Generate root and intermediate keys on a PKCS#11 HSM.
step-pkcs11-init \
  --kms "pkcs11:module-path=/usr/lib/libyubihsm_pkcs11.so;token=YubiHSM" \
  --root 10 --crt 20 --name "Example Internal Root CA"

# 2. Initialise step-ca. The root key is referenced, not copied —
#    step-ca signs with the intermediate only.
step ca init \
  --name        "Example Internal PKI" \
  --dns         "step-ca.pki.svc.cluster.local" \
  --address     ":9000" \
  --provisioner "[email protected]" \
  --kms pkcs11 --root "pkcs11:id=10" --key "pkcs11:id=20"

# 3. Verify the chain before anything depends on it.
step certificate inspect --short $(step path)/certs/intermediate_ca.crt

step-ca supports pkcs11, yubikey, awskms, cloudkms, and azurekms as key backends — a sovereignty decision as much as a security one. An HSM or YubiKey keeps the root of your trust hierarchy inside hardware you physically hold; a cloud KMS puts ultimate authority over it inside a provider's control plane. Both are defensible, but only one survives the question "what happens to our internal PKI if this account is suspended?" For an estate premised on not depending on infrastructure you do not control, the hardware answer is the consistent one.

Click to zoom

step-ca as an ACME server on your own network

The pivotal property of step-ca is that it implements RFC 8555 — the same ACME protocol Let's Encrypt speaks. The integration surface for your private CA therefore becomes a protocol with several mature independent implementations, rather than a vendor SDK. Every ACME client already written — cert-manager, certbot, acme.sh, Caddy, Traefik — works against your CA with one change to the directory URL. You are not adopting a product; you are standing up an endpoint that speaks an IETF standard.

step-ca (Apache 2.0, currently v0.30.2) supports http-01, dns-01, and tls-alpn-01, plus device-attest-01 for hardware-attested enrolment — still an IETF draft rather than a ratified RFC, and disabled by default. Inside a cluster, http-01 is usually right: the proof-of-control loop a public CA must run across the internet runs entirely within your own pod network.

step ca provisioner add acme --type ACME

# Lifetime from your threat model, not a browser vendor's ballot.
# 24h default, 72h ceiling: short enough that a leaked key expires
# before it is useful, long enough to survive a weekend outage.
step ca provisioner update acme \
  --x509-default-dur 24h \
  --x509-min-dur     12h \
  --x509-max-dur     72h

# Equivalent claims block in ca.json:
#   "claims": {
#     "minTLSCertDuration":     "12h",
#     "maxTLSCertDuration":     "72h",
#     "defaultTLSCertDuration": "24h"
#   }

That lifetime choice is the whole argument for owning the CA, made concrete. A 24-hour certificate is not merely shorter — it changes what revocation means. Public PKI's revocation story has always been weak: CRLs and OCSP are checked inconsistently and fail open more often than anyone admits. A certificate that expires tomorrow needs no revocation infrastructure, because expiry *is* the revocation mechanism, and it is the one that never fails open. The public web cannot get there quickly because it must carry every client on earth; your internal estate can, because you control both ends.

cert-manager as an ordinary ACME client

cert-manager graduated from the CNCF in 2024 and is the de facto certificate controller for Kubernetes; the current stable line is 1.20.x. Pointing it at a private ACME server is deliberately unremarkable — same ACME issuer type, different directory URL, one extra field. Since cert-manager 1.11, spec.acme.caBundle carries the base64-encoded PEM of the CA that signed the ACME endpoint's own certificate, letting the controller trust an endpoint no public root vouches for.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-acme
spec:
  acme:
    # Your CA, on your network. No egress required.
    server: https://step-ca.pki.svc.cluster.local:9000/acme/acme/directory
    email: [email protected]
    privateKeySecretRef:
      name: internal-acme-account-key
    # Base64 PEM of your root — trusts an endpoint no public root
    # vouches for. Available since cert-manager 1.11.
    caBundle: <base64-encoded PEM of root_ca.crt>
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: payments-api-tls
  namespace: payments
spec:
  secretName: payments-api-tls
  duration: 24h       # inside the provisioner's min/max window
  renewBefore: 8h     # renew at 2/3 of lifetime; room to retry
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always   # new key each renewal, not just a new cert
  dnsNames:
    - payments-api.payments.svc.cluster.local
    - payments-api.internal.example.com
  issuerRef:
    name: internal-acme
    kind: ClusterIssuer

Two fields deserve more attention than they get. renewBefore must leave genuine slack — cert-manager renews at that threshold then retries with backoff, so a value leaving only minutes turns a transient CA outage into an expiry. And rotationPolicy: Always is the difference between rotating certificates and rotating key material: without it, cert-manager reuses the existing private key, so a key that leaked in January is still the key in December. Short lifetimes only bound a key compromise if the key actually changes.

Distributing trust is the hard half

Issuing certificates is the easy half. The hard half is trust distribution: every client verifying a certificate signed by your root needs that root in its trust store, in the right namespace, updated when the root rotates. This is where hand-rolled internal PKIs collapse — not at issuance, but when someone must add a root to four hundred namespaces and finds there is no mechanism.

trust-manager, a sibling project to cert-manager, exists for exactly this: it watches a source of CA certificates and materialises them as ConfigMaps or Secrets in every namespace matching a selector. Note the API is in motion — the Bundle resource in trust.cert-manager.io/v1alpha1 is being superseded by a cluster-scoped ClusterBundle in trust-manager.io/v1alpha2, announced September 2025. Pin manifests to the version you run and plan that migration rather than meeting it during an upgrade.

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: internal-ca-bundle
spec:
  sources:
    # Your internal root — public certificate only. Never the key.
    - secret:
        name: internal-root-ca
        key: ca.crt
    # Keep the public roots, or every egress call fails verification.
    - useDefaultCAs: true
  target:
    configMap:
      key: ca-bundle.crt
    namespaceSelector:
      matchLabels:
        trust: internal-pki

The failure modes that actually bite

A private CA moves risk from a vendor's operations team onto yours — worth doing, but only with the failure modes named in advance and instrumented before anything depends on them.

• The intermediate expires and nobody notices. The most common internal-PKI outage. Leaves renew daily, so the automation looks healthy right up until the intermediate signing them expires — years after anyone who set it up has moved on. Alert on the issuing chain, not just on leaves.
• The root expires. The same failure with a decade-long fuse and estate-wide blast radius. Root rotation must be rehearsed before it is needed; an unexecuted runbook is a hypothesis.
• A trust store somewhere never got the bundle. A JVM truststore, a Go binary with an embedded CA pool, an appliance with a web UI. trust-manager covers Kubernetes namespaces and nothing else — the rest is your inventory.
• The CA's own database is not backed up. step-ca's store holds the issuance and revocation record. That is audit evidence, and losing it means losing the ability to answer "what did we issue, to whom, and when."
• cert-manager renews faster than the CA can sign. With 24-hour lifetimes across a large estate, issuance volume runs orders of magnitude above a public-CA workload. Load-test at your real certificate count first.

All five are observable, so all five belong on the same self-hosted observability stack that carries the estate. cert-manager exports certmanager_certificate_expiration_timestamp_seconds and certmanager_certificate_ready_status.

groups:
  - name: internal-pki
    rules:
      # Renewal is failing while the cert is still valid — the window in
      # which the problem is still cheap to fix.
      - alert: CertificateNotReady
        expr: certmanager_certificate_ready_status{condition="False"} == 1
        for: 1h
        labels: { severity: critical }

      # Last line of defence on the leaf.
      - alert: CertificateExpiringSoon
        expr: |
          (certmanager_certificate_expiration_timestamp_seconds - time()) / 3600 < 8
        for: 15m
        labels: { severity: warning }

      # The intermediate — via a blackbox probe against the CA endpoint,
      # because no leaf-level metric will tell you the chain is dying.
      - alert: IssuingChainExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 90
        for: 1h
        labels: { severity: critical }

Where a private CA does not belong

Sovereignty is practised in proportion, and a private CA is not a universal upgrade. Public names still belong on a public ACME CA — an internal root gains nothing for a hostname unmodified browsers must validate, and costs a support burden the moment a customer's device distrusts it. The ACME Renewal Information extension, RFC 9773, is making public renewal more robust still.

Click to zoom

The subtler boundary is workload identity. SPIFFE and SPIRE solve an adjacent but different problem: they issue attested identities based on what a workload *is* — its node, service account, and image — rather than what hostname it answers to, with SVIDs rotating in minutes. A private PKI issues certificates for names. The two compose cleanly, and most mature estates run both: SPIRE inside the mesh, step-ca for the hostname TLS that ingress, databases, and appliances actually speak. Beneath both, an eBPF dataplane like Cilium enforces identity-aware policy — the certificate proves who you are, the dataplane decides who you may reach.

And the honest early exit, which the decision flow puts in the critical path: with no team to own key ceremonies, intermediate rotation, and expiry alerting, a managed CA remains the right answer. An internal root that expires because the person who created it left is worse than any amount of CT-log disclosure. Build the capability first, then take the responsibility.

Audit-grade means hardening the CA itself

A certificate authority is the highest-value target in an estate, because whoever controls it can mint an identity for anything. Both projects' advisory history is instructive rather than disqualifying. step-ca has carried authentication-bypass issues permitting issuance while skipping authorization checks — CVE-2025-44005 in the ACME and SCEP provisioners, and CVE-2026-30836 in SCEP's UpdateReq handling, fixed in 0.30.0. cert-manager shipped an advisory where the default edit ClusterRole let namespace users create Challenge and Order resources with attacker-controlled solver configuration, risking disclosure of DNS-provider credentials.

None of that argues against self-hosting; every CA has an advisory history, and the ones that publish theirs are the ones you can audit. It argues for treating the CA as tier-zero infrastructure: disable every provisioner you do not use — SCEP in particular, the surface behind both step-ca CVEs above — patch promptly, isolate the CA in its own namespace under a Kyverno policy set that prevents workloads from creating issuers, and verify the image's signature and SBOM before it runs. A supply-chain compromise of the binary holding your intermediate key is the worst version of this story.

The audit-grade payoff is the issuance record. Because step-ca writes every issuance to its own database, you can answer — from your own systems, without a support ticket — which certificates exist, which workload requested each, and when each expires. That is the evidence an auditor asks for under NIS2 and DORA, held rather than requested. The logic making secrets a self-hosted concern applies with more force here: a secret is a credential, but a CA is the ability to manufacture credentials.

The exit ramp and the long game

The optionality argument here is unusually clean, because the exit ramp is the protocol itself. Moving from step-ca to a different private CA, to a commercial CA that speaks ACME, or back to a public CA for a name that became public, is a change to one server URL and one caBundle field. Your workloads never know which CA signed them; they read a kubernetes.io/tls Secret, the same shape regardless. That is the opposite of how PKI lock-in usually works — traditional enterprise PKI binds you through enrolment integrations, agent software, and per-certificate pricing that makes short lifetimes economically irrational. An Apache-2.0 CA speaking an IETF standard has no such incentive, which is why a 24-hour default is even a conversation: depend on protocols, not products.

The long game is where internal PKI stops being infrastructure and becomes institutional memory. A trust hierarchy is a decade-scale commitment — longer than the workloads it protects, longer than the Kubernetes distribution beneath it, quite possibly longer than the team that stood it up. The root you generate this quarter will still be signing intermediates after the current architecture has been replaced twice. That is why it deserves the offline key, the rehearsed rotation, and the alert on the chain rather than the leaf. Certificates are the one piece of infrastructure whose neglect is scheduled in advance, to the second, by the certificate itself.

That is the sovereign internal PKI in one sentence: not a cheaper certificate authority, but one whose root you physically hold, whose lifetimes come from your own threat model, and whose exit is a configuration change. step-ca and cert-manager are both open, both mature, both in front of you. The decision that remains is the one this practice keeps returning to — whether you own the systems you depend on, or keep renting the one that decides who your systems are.