Self-Managed Kafka: Strimzi, Redpanda, NATS Off Confluent Cloud

The event backbone is the most load-bearing piece of infrastructure most organisations never chose to own. It starts as one Kafka topic behind a feature, and within two years it is the nervous system — every service publishes to it, every analytics job reads from it, and an outage of it is an outage of everything. That is precisely the layer teams most often rent from Confluent Cloud, and precisely the layer where renting compounds into the deepest lock-in: not because the software is proprietary (Kafka is open), but because the operational muscle to run it has been outsourced, the data has an egress meter attached, and the wire your whole estate speaks now belongs to someone else's balance sheet.

This article is the production ops playbook for taking that layer back — not as a cost-cutting stunt, but as an engineering decision with its eyes open. It covers three tools that together span the whole design space: Strimzi, which turns Apache Kafka into declarative Kubernetes objects; Redpanda, the single-binary, Kafka-compatible engine for teams that want the protocol without the JVM; and NATS JetStream, for the large class of workloads that never needed Kafka at all. It also covers the part most migration pitches skip: the honest cost model, the day-2 work that does not disappear when you leave the managed service, and the gate that tells you when staying on Confluent is still the right call.

The Confluent Cloud meter, and what it actually costs

Confluent Cloud's serverless tiers price three dimensions: data ingested, data egressed, and data retained. On the Basic tier the published rates are roughly $0.11 per GB in, $0.11 per GB out, and $0.10 per GB-month retained — before the three-times replication factor that durable topics require. Those numbers look small until you multiply them by a real firehose with real fan-out. In one independent breakdown of a 300 MiB/s workload, egress alone reached tens of thousands of dollars a month and became the single largest line item on the bill — because every consumer group, every mirror, every analytics reader pays the meter again on the same bytes.

The infrastructure gap is real and it widens with scale. AxonOps' 2026 cost analysis put a modest 3-broker, 45 MB/s cluster at roughly $571/month self-hosted against $1,216/month on Confluent Cloud — a meaningful but survivable difference. At 30 brokers and 1.875 GB/s the same analysis put self-hosting near $8,820/month against a Confluent Cloud range of $42,093–$53,773/month. That is not a rounding error; it is the difference between a line item and a headcount, and it is the same repatriation calculus that governs any serious cloud exit — the premium is structural at scale, not marginal.

So the decision is not "managed is a rip-off." It is narrower and more useful: at what scale, and with what team, does owning the backbone become the disciplined choice rather than the reckless one? The rest of this article answers that by tool — because the right answer for a 200-topic Kafka estate with Connect and Streams is not the right answer for a fleet of edge services trading small messages.

Strimzi: Kafka broker lifecycle as declarative objects

Strimzi is the reason self-managed Kafka on Kubernetes stopped being a synonym for pain. It is a CNCF Incubating project (accepted 2019, incubating since 2024) whose operator models an entire Kafka cluster as a set of custom resources: a Kafka for the cluster, KafkaNodePool resources for the broker and controller groups, and KafkaTopic / KafkaUser for topics and principals. You declare intent — this many brokers, this storage class, these listeners, TLS-authenticated clients only — and the operator's reconciliation loop drives the cluster to that state and keeps it there through node failures and rolling upgrades. The whole thing is GitOps-native: the desired state of your event backbone lives in a repository, reviewed like any other code.

The single most important recent change is the removal of ZooKeeper. Strimzi reached its 1.0 release with KRaft — Kafka's own Raft-based consensus — as the only supported metadata mode, and dropped the older CRD API versions in the process. In practice this halves the number of distributed systems you operate: there is no separate ZooKeeper ensemble to size, secure, back up, and reason about during an incident. A KRaft controller quorum lives inside the Kafka cluster itself, and Strimzi models it as a dedicated node pool. Fewer moving parts is not a cosmetic win — it is one fewer consensus system in your audit scope and one fewer thing to page on at 3 a.m.

# KRaft controller quorum — replaces the old ZooKeeper ensemble entirely.
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controllers
  namespace: events
  labels: { strimzi.io/cluster: backbone }
spec:
  replicas: 3
  roles: [controller]           # dedicated KRaft metadata quorum
  storage:
    type: persistent-claim
    size: 20Gi
    class: local-nvme
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: brokers
  namespace: events
  labels: { strimzi.io/cluster: backbone }
spec:
  replicas: 3
  roles: [broker]
  storage:
    type: persistent-claim
    size: 1Ti
    class: local-nvme
    deleteClaim: false          # keep the data if the pool is deleted
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: backbone
  namespace: events
  annotations:
    strimzi.io/kraft: enabled          # ZooKeeper-free
    strimzi.io/node-pools: enabled
spec:
  kafka:
    version: 4.0.0
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication: { type: tls }   # mutual TLS: only authenticated clients
    config:
      default.replication.factor: 3
      min.insync.replicas: 2            # acks=all needs 2 in-sync replicas to commit
      offsets.topic.replication.factor: 3
    authorization: { type: simple }     # topic/group ACLs enforced by the broker
  entityOperator:
    topicOperator: {}                   # KafkaTopic CRs become real topics
    userOperator: {}                    # KafkaUser CRs mint TLS certs + ACLs

Two settings in that manifest carry the whole durability story. min.insync.replicas: 2 with a replication factor of 3 means a producer using acks=all is never told its write succeeded until at least two brokers have it — the streaming equivalent of quorum-synchronous commit, and the difference between "we failed over" and "we failed over and lost four seconds of orders." The mutual-TLS listener and simple authorization mean the brokers speak only to clients holding a certificate the User Operator minted, with ACLs enforced per topic and consumer group. That is audit-grade access control expressed as Kubernetes objects, not a console someone clicked through once. Broker metrics land in the same observability stack you already run, so consumer lag and under-replicated partitions are graphed next to everything else.

Click to zoom

The one architectural decision Strimzi does not make for you is where the bytes ultimately rest. Kafka's tiered-storage support lets brokers keep only hot segments on local NVMe and offload cold segments to object storage — which means the retention window that would blow up your local disk instead lands in an S3-compatible store you own. That object store is the sovereignty boundary: your history lives where you decide, under your keys, with no provider able to meter reads against it. It is the same pattern that makes the stateful data layer sovereign — the compute is cattle; the durable state sits in a boundary you control.

Redpanda: the single-binary bet

Strimzi runs Apache Kafka, the JVM and all. Redpanda takes the opposite bet: reimplement the Kafka wire protocol in C++ so that any existing Kafka client, tool, or connector works unmodified, but the engine underneath is a single binary with no ZooKeeper, no JVM, and no external dependencies. Internally it uses a thread-per-core architecture on the Seastar framework and Raft consensus per partition, which eliminates the JVM garbage-collection pauses and lock contention that make Kafka's tail latencies unpredictable. The practical effect is that a broker starts in milliseconds instead of minutes, uses far less memory, and ships a schema registry and HTTP proxy inside the same process.

The reason this matters for a sovereignty practice is operational surface, not benchmarks. A single binary is dramatically easier to own end to end: one thing to build reproducibly, one thing to sign and attest in your supply chain, one thing to patch, one process to reason about when it misbehaves. For a team taking on a self-managed backbone for the first time — especially one without deep JVM-tuning experience — fewer moving parts is worth more than a latency percentile. You are trading the vast Kafka ecosystem's maturity for a thinner, more legible system, and for many mid-scale estates that is the right trade.

The interchangeability is the point worth dwelling on. Because Strimzi-run Kafka, Redpanda, and Confluent all implement the same Produce/Fetch/Metadata protocol and consumer-group coordination, your producers and consumers do not know or care which one they are talking to. That is what a real exit ramp looks like: you can stand up Redpanda beside a Confluent cluster, mirror the topics, cut the client bootstrap servers over, and decommission the old one — with no application rewrite. The lock-in that remains lives in the proprietary layers above the protocol (fully-managed connectors, ksqlDB-specific features, Stream Governance), so the discipline is simple: stay on the open wire, and keep the vendor-specific surface as thin as you can live with.

NATS JetStream: when you never needed Kafka

A large fraction of what gets built on Kafka is not a durable, replayable, partitioned commit log at all — it is service-to-service messaging that acquired Kafka because Kafka was already there. For that workload, NATS is a different and often better shape. Core NATS is a tiny, high-performance pub/sub and request-reply system; JetStream adds persistence on top — replicated, file-backed streams with at-least-once and exactly-once delivery, consumer acknowledgement, and retention policies — without the operational weight of a Kafka cluster. The whole server is a single small Go binary, trivial to run at the edge or embed alongside services, with subject-based addressing instead of Kafka's partition model.

The cost argument here is sharp because the comparison is against a managed service with no self-hosted option. Synadia — the company that maintains NATS — published a benchmark putting NATS 59–87% below Amazon Kinesis on cost depending on message size and scale, with a 3-node NATS cluster on c5n.xlarge instances across three availability zones costing about $473/month in compute, and single-digit-millisecond delivery latency against Kinesis's hundreds of milliseconds. Treat the exact percentages as a vendor benchmark and measure your own workload — but the structural point holds: NATS's default consumer model already does what Kinesis charges a premium (Enhanced Fan-Out) to provide. When your requirement is many low-latency consumers of the same subject, that is a fundamental architectural advantage, not a tuning delta.

# A JetStream stream is a replicated, file-backed log — no Kafka, no JVM.
# Three replicas, a 7-day retention window, discard oldest when full.
nats stream add ORDERS \
  --subjects "orders.>" \
  --storage file \
  --replicas 3 \
  --retention limits \
  --max-age 168h \
  --discard old

# A durable, explicitly-acknowledged consumer — at-least-once by default.
nats consumer add ORDERS fulfilment \
  --ack explicit \
  --deliver all \
  --max-deliver 5 \
  --filter "orders.created"

NATS is not a Kafka replacement and pretending it is will burn you. It has no equivalent of Kafka Connect's connector ecosystem, its stream-processing story is thinner, and very high-throughput analytical fan-out is Kafka's home turf. The discipline is to match the tool to the workload: use NATS for the messaging, eventing, and control-plane traffic that never justified a Kafka cluster, and keep Kafka (via Strimzi or Redpanda) for the durable, high-throughput, replayable log that feeds analytics. Many sovereign estates end up running both — NATS as the lightweight nervous system, Kafka as the system of record — and that is a coherent architecture, not a failure to standardise.

Choosing the backbone: a decision framework

The three tools are not competitors so much as answers to three different questions, and a single decision path routes you cleanly. The first gate is the one every honest self-hosting argument has to lead with, and it is not technical.

Gate one — is there a team to own day-2? Running a streaming backbone means owning topic and subject design, ACL and account governance, consumer-lag monitoring, connector and schema evolution, capacity planning, and incident response. If no one will own that, the managed service is the correct answer today and the work is to build the capability before you migrate — exactly as it is for any repatriation. Gate two — do you need the Kafka wire protocol and a durable, replayable, partitioned log? If yes, you are choosing between Strimzi and Redpanda. If no — if you need messaging, not a log — NATS is lighter and cheaper. Gate three — do you need the full Kafka ecosystem (Connect, Streams, mirroring, hundreds of topics)? If yes, Strimzi's Apache Kafka is the safe choice; if you want the protocol with minimal ops, Redpanda's single binary wins.

Click to zoom

Notice what the framework does not do: it never tells you to self-host everything because self-hosting is virtuous. Sovereignty is a goal pursued in proportion, not a mandate. A four-person startup with one Kafka topic should stay on the managed service and spend its scarce engineering time on the product; the calculus flips when the backbone becomes load-bearing, the bill becomes a headcount, and the team has the depth to own it. Getting that timing right — neither renting past the point it makes sense nor self-hosting before you are ready — is the actual skill.

GitOps the fleet, and design the exit ramp in

The operational win of the operator model is that the entire backbone becomes Git-managed state. A Kafka, its node pools, every KafkaTopic, and every KafkaUser are YAML in a repository, applied by the same GitOps controller that manages the rest of your clusters. A new topic is a pull request with a reviewer and an audit trail, not a click in a console. An ACL change is a diff. A broker version bump is a commit that the operator rolls out one pod at a time, respecting min.insync.replicas so the cluster never drops below quorum during the upgrade. This is what audit-grade rigor looks like for streaming infrastructure: every change to who-can-read-what and what-topics-exist is reviewable, reversible, and attributable.

The exit ramp must be designed in before you need it — the day you need it is the worst day to find it missing. Concretely: keep producers and consumers on the open Kafka or NATS protocol and off vendor-only client extensions; hold your retention in an object store you own rather than a provider's opaque tier; pin schemas in a registry you run; and rehearse the failover you claim to have. Because all three engines here are open and protocol-compatible, moving between them — Confluent to Redpanda, self-hosted Kafka to another cluster, one region to another — is a mirror-and-cutover, not a rewrite. That property is the whole reason to do this: not to save a line item this quarter, but to never again be where leaving is too expensive to contemplate.

The long-game framing is where it lands. An event backbone is a decade-scale commitment — the topics you define and the semantics your services assume outlive the team that wrote them. Building it on open protocols, on infrastructure you own, with its lifecycle in version control and its history in your own object store, is what lets it last without becoming a hostage negotiation at renewal time. The events themselves feed everything downstream, from real-time services to the sovereign lakehouse where they become analytics — so the backbone's sovereignty propagates to every system built on top of it. Own the wire, and you own the estate.

That is the sovereign event backbone in one sentence: not the cheapest streaming platform, but the one you run on an open protocol, whose lifecycle is code you review and whose history is data you hold. Strimzi, Redpanda, and NATS are mature, open, and in front of you. The decision that remains is the one this practice keeps returning to — whether you own the systems you depend on, or keep renting the one every other system depends on in turn.