Redpanda vs Kafka: The Same Wire Protocol, Another System

Most comparisons of the two still lead with ZooKeeper and a throughput chart. Both framings have aged badly. Apache Kafka removed ZooKeeper entirely in 4.0, so most older articles' opening argument no longer describes the current release. And the benchmark charts come almost exclusively from vendors with a product to sell on one side of the line.

What is left is more useful to a platform lead. Redpanda publishes its own list of places where it departs from the Kafka protocol. It publishes a licence whose use grant, conversion clock and feature gates a procurement team has to read. And it documents three different migration paths, each with its own version and licence prerequisites. This article works through each, then the host, today's failure modes, and a decision rule that outlasts one renewal cycle.

ZooKeeper is no longer the difference

Kafka 4.0, announced on 18 March 2025, was the first major release to operate entirely without ZooKeeper. The upgrade guide is blunt: 4.0 only supports KRaft mode, ZooKeeper mode has been removed, and clusters in ZooKeeper mode have to be migrated to KRaft before they can be upgraded to 4.0.x. The 3.9 announcement names 3.9 as the bridge in its upgrade examples, because 4.0 supports neither running in ZooKeeper mode nor migrating from it. The same release added KIP-853, which made controller quorum membership dynamic — with the stated limitation that 3.9 cannot convert an existing static quorum to a dynamic one.

The current release, 4.4.0, shipped on 22 October 2026. A single-node broker-and-controller lab has no ZooKeeper setting left to compare:

# config/server.properties — Kafka 4.4, combined broker + controller (lab only)
process.roles=broker,controller
node.id=1
controller.quorum.bootstrap.servers=localhost:9093

listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT

log.dirs=/var/lib/kafka/data

# One node cannot hold three replicas of the internal topics.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

A new log directory must be formatted before the first start, as the 4.4 quickstart does — KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)", then bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties, then bin/kafka-server-start.sh config/server.properties. The quickstart treats that format as a required step, not an option.

What still differs is the runtime underneath the protocol. Kafka is a JVM broker with a heap and a garbage collector you tune. Redpanda is built in C++ and packaged as a single binary that does not rely on external systems, and it implements a thread-per-core model through the Seastar library, pinning each application thread to a CPU core. Its replication unit is also different: every topic partition forms its own Raft group, with one elected leader and zero or more followers. Redpanda's own materials claim up to 10x lower p99 latency than other platforms; that is a vendor claim this article cannot vouch for, and not the axis the decision should turn on.

Click to zoom

Where wire compatibility holds, and where it does not

Redpanda's compatibility page states that it is compatible with the Kafka protocol, then lists exceptions. They are short, specific, and each one maps to a line in a client or platform configuration somebody will have to change:

• SCRAM: a user cannot hold both a SCRAM-SHA-256 and a SCRAM-SHA-512 credential. Redpanda supports one SASL/SCRAM mechanism per user, so any estate that issued both during a rotation has to pick one before cutover.
• Quotas: the request-rate quota, request_percentage, is not supported. Redpanda supports byte-rate quotas (producer_byte_rate, consumer_byte_rate) and the topic-mutation quota (controller_mutation_rate). A noisy-neighbour policy built on request share has to be re-expressed in bytes.
• Transactions: Redpanda does not implement the server-side portion of KIP-890, Transactions Server-Side Defense. The page says its own transaction implementation is not susceptible to the class of errors that KIP addresses, and that Kafka 4.x clients detect that Transactions V2 is unsupported and fall back to the original transaction protocol.
• HTTP Proxy: it does not support topic or ACL create, read, update and delete. It is designed for clients that produce and consume, not for administrative tooling.

The list is also notable for what it does not mention. Kafka 4.0 made the next-generation consumer rebalance protocol, KIP-848, generally available: it is enabled by default on the server side, and consumers opt in with group.protocol=consumer. Kafka 4.2 declared Kafka Queues — share groups, KIP-932 — production-ready. Redpanda's compatibility page names neither. Absence from an exception list is not a support statement, so treat both as things to test against your target version before any consumer is migrated, not as settled in either direction.

What each one asks of the host

Redpanda's production requirements are a floor, stated as rules. Each broker must have at least 2 GB of memory per core. Each production node must have at least two physical cores, and four are strongly recommended. The filesystem is XFS or ext4, with XFS recommended for its performance with Redpanda workloads, and NFS is unsupported. Because each application thread is pinned to a core, the core count is the unit of capacity planning, not an afterthought.

Partition count is the second unit. The same page says each broker must have at least 2 MB of memory for each topic partition replica. It then describes a separate mechanism: memory available to partition replicas is a pool sized as a percentage of total memory by topic_partitions_memory_allocation_percent, and each replica consumes topic_memory_per_partition bytes from that pool. The cluster-property reference gives the defaults — 10 percent, and 204800 bytes — and says that if the pool cannot cover a new partition, topic creation or alteration fails. The 2 MB figure is a sizing rule; the pool is the admission check that turns an undersized node into a failed topic creation.

Kafka's documentation sets no comparable floor. Its hardware and OS guidance (4.2 documentation) notes that Kafka uses regular files and has no hard dependency on a filesystem, with EXT4 and XFS the two most used. Its Java guide offers -Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 as typical arguments, illustrated with one of LinkedIn's busiest clusters — an example, not a sizing requirement. The same guide fully supports Java 17, 21 and 25, and recommends the most recent LTS, Java 25 at the time it was written. Java 17 has been the broker floor since 4.0, when brokers, Connect and tools moved to it while clients and Kafka Streams stayed on Java 11.

The practical difference is who carries the tuning. Kafka hands you a JVM to size and a garbage collector to watch, and documents a typical configuration rather than a rule. Redpanda hands you hard minimums and core pinning, which make capacity easier to reason about and harder to run on shared or oversubscribed hosts.

The licence is an operational fact

Apache Kafka is entirely Apache 2.0. Redpanda is two editions. According to its licensing overview, the Community Edition is under the Redpanda Business Source License, free and source-available; the Enterprise Edition requires a licence key and adds features. The licence text grants use provided you do not use the work for a "Streaming or Queuing Service" . The file defines one as "a commercial offering that allows third parties (other than your employees and individual contractors) to access the functionality of the Licensed Work by performing an action directly or indirectly that causes the creation of a topic in the Licensed Work." The next sentence says that includes providers of infrastructure, cloud, hosting and data-center services, and their affiliates, that would offer the work as part of a broader service to their own customers. Whether an internal platform serving other business units falls inside that line is a question for counsel.

The conversion clock is stated two ways, and a procurement brief should quote both rather than pick one. The licensing overview says BSL code converts to Apache 2.0 four years after each code merge. The licence file itself sets the Change License to Apache 2.0 and the Change Date at four years from release date, alongside the other conversion terms the BSL template carries. Do not compute a conversion date for any specific version from either sentence; read the file shipped with the version you run.

The enterprise boundary matters more day to day than the conversion date, because each gated feature fails differently when a licence lapses. From the licensing table:

• Tiered Storage: topics cannot be created or modified to enable it.
• Role-Based Access Control: roles and the ACLs associated with roles cannot be created or modified; role deletion is allowed.
• OAUTHBEARER/OIDC authentication: no change.
• Remote Read Replicas: replica topics cannot be created or modified.
• Shadowing: new shadow links cannot be created; existing links continue operating and can be updated.

That is not a list of things that stop. The documentation says the cluster continues to operate without data loss while further use of enterprise features is restricted. Two other rules carry the real risk. Since 24.3, a new cluster automatically receives a 30-day trial licence with unrestricted enterprise features; when it expires, inactive enterprise features are disabled and active ones enter a restricted state. And Redpanda blocks upgrades to new feature releases if enterprise features are active without a valid licence. A team can adopt a gated feature during a trial without deciding to buy, and discover the decision at its next upgrade. Check the per-release table against the version you run.

On the Kafka side, tiered storage (KIP-405) was declared production-ready in 3.9 under the same Apache 2.0 licence as everything else. Production-ready is not the same as finished: Kafka 4.3.2 fixed KAFKA-20732, in which size-based retention could delete data still within retention from both local and remote tiers after a leader election. The difference between the two projects here is not maturity; it is that one gate is a licence key and the other is your own upgrade discipline. Stribog's notes on the OpenTofu licence migration and the OpenBao exit from Vault walk through what a BSL change does to a dependency you already run.

Failure modes and upgrade risk

The first failure mode is licence drift, and it is cheap to detect. Run a pre-flight before go-live and before every upgrade:

#!/usr/bin/env bash
# Redpanda pre-flight: make licence state and enterprise feature use visible
# before an upgrade is attempted, not after it is refused.
set -euo pipefail

rpk cluster info
rpk cluster license info --format json
rpk cluster config get cloud_storage_enabled
rpk cluster config get sasl_mechanisms

The second is a bad patch release, which both projects ship. Redpanda's GitHub release notes for 26.1.16 told users not to install it: a critical bug could destabilise long-running clusters, fixed in 26.1.17, and users should upgrade directly to 26.1.17 or 26.2.x. The 26.1.17 release itself reverted the c-ares DNS resolver from 1.34.7 to 1.34.6 to avoid an upstream regression that could leave broker-to-broker RPC connections unable to reconnect after a peer restart — and the revert reintroduces CVE-2026-33630 until upstream fixes the regression. So 26.1.17 is a trade, and a security team should see that line before signing off.

Kafka's equivalent is in the 4.3.2 announcement, which fixed two separate data-loss bugs. KAFKA-20732 is the tiered-storage size-retention bug above. KAFKA-20979 is a different class: the time index could be truncated across a broker restart, so time-based retention deleted segments still within retention. It is a retention bug, not a tiered-storage one. The 4.4.0 release notes list both, so the current release carries both fixes.

The third is the support calendar. Per Redpanda's release notes, v26.2 (July 2026) is supported until 28 July 2027, and v25.3 until 19 November 2026 — weeks away at the time of writing. A cluster on 25.3 is choosing an upgrade path now, and with an enterprise feature active that path runs through the licence check above.

Moving between them is a migration, not a config change

Redpanda's Kafka migration guide documents three paths, and they differ on exactly the properties an exit ramp is judged by.

1. Shadowing is the recommended path for a running Kafka workload. The target needs Redpanda 25.3 or later and an Enterprise Edition licence. The link authenticates with SASL/SCRAM, SASL/PLAIN (25.3.2 or later), TLS, mutual TLS or nothing; the source Schema Registry must be Confluent-compatible with HTTP basic, mutual TLS or no authentication; an Amazon MSK source needs SASL/SCRAM-SHA-512 enabled. Each record is written at its source offset, so consumers resume at the same offsets. The guide qualifies the rest of what the link copies: committed consumer group offsets are capped at the target partition's high watermark, only ACLs that match the link's filters are replicated, and schemas move only when Schema Registry sync is configured — from a Confluent Schema Registry, only to a 26.2 or later target. The data-migration page sends Kafka workloads to the guide, and to MirrorMaker 2 only when neither Shadowing nor Migrator applies; its MM2 procedure downloads Kafka 3.3.1, so use the 4.4 file below. At cutover, before a moved group starts on the target, exclude it from the link's consumer offset sync or pause that sync; otherwise, whenever the group has no active members on the target, the link copies the source's stopped offsets over its progress. Before failing over, stop the moved producers and compare each partition's high watermark on the source directly with the target's: the link's reported lag comes from its last fetch and can read 0 while the source holds unfetched records, and a failover stops replication for good, stranding them. If those producers register schemas, first set paused: true in the link's schema_registry_sync_options; the target's synced registry contexts refuse new schemas while sync runs, even after failover. A shadow link replicates only into Redpanda, and a failed-over topic cannot return to shadow mode.
2. Redpanda Migrator is a Redpanda Connect pipeline that copies records, topic settings, schemas and consumer group offsets, and needs no Enterprise licence. It is the documented path when the target is older than 25.3, has no licence, or the source accepts only IAM or OAUTHBEARER authentication — Amazon MSK with IAM access control, for example — which a shadow link does not support and Migrator does. Separately, neither Shadowing nor Migrator reads schemas from AWS Glue Schema Registry, so with a Glue registry schemas do not move on either path, whatever the authentication. Records get new offsets on the target, so committed offsets are translated by timestamp; for a group with no active members the match is refined to the exact record using a source-offset header, named by offset_header in Redpanda Connect 4.71.0 and later. Schemas keep their IDs by default, which requires the target registry in IMPORT mode. Migrator can also write from Redpanda back to Kafka.
3. MirrorMaker 2, from the Kafka distribution, works too — and Redpanda's guidance is to use it only when neither of the above applies.

If MirrorMaker 2 is the path, keep the file one-directional. Properties are from the Kafka 4.4 MirrorMaker configuration reference:

# connect-mirror-maker.properties — ONE-WAY: Kafka -> Redpanda
clusters = kafka, redpanda
kafka.bootstrap.servers = kafka-1.example.internal:9092
redpanda.bootstrap.servers = redpanda-1.example.internal:9092

kafka->redpanda.enabled = true
kafka->redpanda.topics = .*
kafka->redpanda.groups = .*
redpanda->kafka.enabled = false

# Keep topic names identical on the target. One-way only: MirrorMaker
# cannot prevent replication cycles under this policy.
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy

replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3

emit.checkpoints.enabled = true
sync.group.offsets.enabled = true

Two lines in that file carry the risk. The IdentityReplicationPolicy javadoc says it does not rename remote topics, suits one-way replication, and that MirrorMaker cannot prevent cycles when it is used — so enabling redpanda->kafka in the same file would loop records. And sync.group.offsets.enabled defaults to false; when on, it writes translated offsets to the target's __consumer_offsets only while no consumers in that group are connected to the target. Keeping source offsets is what Shadowing sells.

Click to zoom

Whichever path, the cutover is a rehearsal list, not a bootstrap-server swap: every client configuration audited, transactional producers tested against the fallback protocol, SCRAM credentials recreated with one mechanism, request-share quotas re-expressed in bytes. The same discipline applies leaving a managed service — see leaving Kinesis and leaving SQS, where the semantics gap is wider still.

Choosing for the next decade

A decision rule that survives contact with procurement:

• Choose Apache Kafka when the team already carries JVM operating skill, when the licence must be one an auditor can approve without a use-grant review, or when tiered storage and access control must never depend on a key that can lapse.
• Choose Redpanda when hard per-core sizing and a single binary fit the team better than JVM tuning, the BSL use grant has been read against your actual service model, and either the Community feature set is enough or the enterprise licence is budgeted as a permanent line, not a trial.
• Either way, pick the exit path before the entry path: know which of Shadowing, Migrator or MirrorMaker 2 would carry you out, and what it would cost in offsets.

Over a decade, the commitments that are cheap to reverse are the ones on the wire: client code written against the Kafka protocol moves between engines. The expensive ones sit below it — a gated feature woven into daily operations, credentials minted under one model, offsets that a migration will translate rather than keep. Owning the system means knowing which of those you have taken on, and keeping the protocol, not the product, as the thing your applications depend on.