Leaving SQS: Kafka Is Not a Drop-In Queue Replacement

Search this comparison and you get platform selection: throughput tables, ordering guarantees, a winner. Wrong layer. Managed-queue exits overrun because four SQS primitives leaked into application code and nobody costed the rewrite: a handler calling ChangeMessageVisibility is written against SQS whether or not anyone says so.

What follows maps those primitives onto Kafka's real configuration names and effective bounds, so the mapping breaks here rather than mid-cutover — the same discipline as pricing a cloud exit as a number.

Two Different Slots, One Search Query

SQS is a per-message broker: it keeps state for every message — in flight or not, how many times received, when the lease expires. Kafka is an append-only partitioned log: consumption removes nothing, and a consumer group's entire state is one committed offset per partition. Two data structures, not two takes on one idea. (Leaving a managed *log* is a different problem again — see why Kinesis shards are not Kafka partitions.)

The test: does order matter across the whole stream, or is each unit independent work? Kafka's own 4.3 upgrade documentation draws that line introducing share groups: use them in cases where records are processed one at a time, rather than as part of an ordered stream. A migration replaces not a transport but guarantees your business logic already depends on silently.

The Four SQS Primitives That Live in Your Application Code

The lease. An SQS visibility timeout defaults to 30 seconds, minimum 0, maximum 12 hours, and ChangeMessageVisibility sets one for an individual message without changing the queue's overall setting. It is a per-message lease a handler renews as it works, with one edge Kafka cannot reproduce: the 12-hour maximum runs from first receipt, and extending does not reset it.

The acknowledgement. DeleteMessage is the acknowledgement. There is no separate commit — not deleting is how you retry, and the message returns when the lease lapses.

The poison counter and its destination. A redrive policy's maxReceiveCount is how many times a consumer can receive a message before it moves to a dead-letter queue; SetQueueAttributes documents a default of 10, and the dead-letter queue configuration page a valid range of 1 to 1,000. Every receive counts, including one where the handler crashed and the timeout lapsed — and where maxReceiveCount exceeds 3, a message received three or more times without being deleted is moved to the back of the queue, so later attempts arrive later than a flat retry estimate suggests. Two more details surface late: the DLQ must be in the same AWS account and Region as the source queue, and a standard-queue message's expiration is always based on its original enqueue timestamp, unchanged by the move. The DLQ is not a holding pen with a fresh clock.

The duplicate contract. Standard queues ensure at-least-once delivery, but more than one copy might be delivered and messages may occasionally arrive out of order. Your handlers are therefore already idempotent, or already quietly wrong — the one primitive that transfers cleanly.

Two numbers the genre still quotes wrongly: maximum message size is 1,048,576 bytes — 1 MiB, not 256 KB — and most standard queues cap at approximately 120,000 in-flight messages.

{
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-central-1:123456789012:orders-intake",

    "VisibilityTimeout": "300",

    "MessageRetentionPeriod": "345600",

    "ReceiveMessageWaitTimeSeconds": "20",

    "DelaySeconds": "0",

    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-central-1:123456789012:orders-intake-dlq\",\"maxReceiveCount\":5}"
  }
}

What a Kafka Consumer Group Gives You Instead

A consumer group has an offset per partition. Nothing in that state model expresses that one record failed while the other 499 succeeded — the only thing a consumer tells the broker is how far it has got, over a log that consumption does not shorten.

Failure is also detected at the member, not the message. max.poll.interval.ms is the maximum delay between invocations of poll() under group management, default 300000 ms; if poll() is not called before it expires the consumer is considered failed and the group rebalances, reassigning its partitions. session.timeout.ms, default 45000 ms, does the same when heartbeats stop — though that client setting is not supported when group.protocol is consumer, where group.consumer.session.timeout.ms governs instead.

Set that beside the SQS lease. A handler needing twenty minutes renews its own visibility timeout and nothing notices; the same handler on a consumer group blows the poll deadline and every partition it held is reassigned. Workarounds pay elsewhere: a thread pool keeps poll() responsive but breaks the meaning of a committed offset. Share groups exist because of that misfit.

Click to zoom

Share Groups: Kafka Grows a Queue, with Different Bounds

Status, and how to check it on your own cluster

Share groups are no longer a preview: the Apache Kafka 4.2.0 release announcement of 17 February 2026 states that Kafka Queues (share groups) is now production-ready, and the 4.3 upgrade documentation repeats it independently. Anything saying otherwise quotes KIP-932 on the 4.1 preview. Production-ready is not switched on, though: KIP-932 says share.version=1 is enabled in the latest production metadata version once share groups are generally available. One prerequisite catches small clusters: share groups create an internal topic __share_group_state whose default configuration uses 3 replicas and needs 3 brokers.

#!/usr/bin/env bash
# Share groups on a self-managed Apache Kafka 4.3 cluster.
set -euo pipefail

BOOTSTRAP=localhost:9092
GROUP=orders-intake

# 1. What feature versions has this cluster actually finalized? GA means
#    share.version=1 rides the latest production metadata version, which is
#    not the same as saying every cluster is already there.
bin/kafka-features.sh --bootstrap-server "$BOOTSTRAP" describe

# 2. Finalize the 4.3 feature versions. The upgrade docs run this only after
#    the cluster's behaviour and performance have been verified.
bin/kafka-features.sh --bootstrap-server "$BOOTSTRAP" upgrade --release-version 4.3

# 3. Confirm, then look at the group itself.
bin/kafka-features.sh --bootstrap-server "$BOOTSTRAP" describe

bin/kafka-share-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" --describe --state
bin/kafka-share-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" --describe --members
bin/kafka-share-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" --describe --offsets

# --describe --offsets prints GROUP, TOPIC, PARTITION, START-OFFSET and LAG
# (and LEADER-EPOCH with --verbose). That is the lag view. There is no
# delivery-count column -- and no metric publishes one either. The documented
# share-group series are rates and times: AcquisitionLockTimeoutPerSec,
# InFlightMessageCount, RecordAcknowledgementsPerSec by ack type. So nothing
# reports how close a given record sits to share.delivery.count.limit. Alarm
# on the lock-timeout rate and treat it as a proxy, not the counter.
# KIP-932 notes it is possible to downgrade to share.version=0 to
# disable the feature, which is the documented off switch.

The mapping, primitive by primitive

The 4.3 AcknowledgeType enum has four constants: ACCEPT (consumed successfully), REJECT and RELEASE (not consumed successfully), and RENEW (still being processed). KIP-932 specified only three — RENEW arrived in 4.2 through KIP-1222, letting applications extend acquisition lock timeouts on records needing longer processing. It is the ChangeMessageVisibility analogue, not a drop-in: KafkaShareConsumer's documentation requires calling poll() regularly and acknowledge() with RENEW every iteration, the renewed record returned again by poll(). And if partition leadership changes, the current delivery attempt ends — a clause SQS lacks.

DeleteMessage maps to ACCEPT, with a qualification that changes your disk bill: ACCEPT marks the record Acknowledged, not deleted. The record moves to Archived when the share-partition start offset passes it, and the bytes leave only when topic retention removes them — and because share groups all read the same log, that retention applies to every group. A lapsed timeout maps to RELEASE, or to doing nothing: the lock releases automatically once the lock duration elapses. maxReceiveCount maps to share.delivery.count.limit, whose counter increments by 1 on every acquisition.

REJECT has no SQS analogue — the record is unprocessable and is not made available for another delivery attempt; in KIP-932's worked example a rejected record is Archived. None of this works on a stock client: share.acknowledgement.mode defaults to implicit, and in implicit mode the consumer must not use acknowledge() at all.

// EXCERPT. Modelled on the KafkaShareConsumer renewal example in the 4.3 javadoc.
Properties props = new Properties();
props.setProperty("bootstrap.servers", "<BOOTSTRAP_SERVERS>");
props.setProperty("group.id", "orders-intake");
props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

// The shipped default is "implicit", and in implicit mode acknowledge() must
// not be called at all. Without this line, every call below is illegal.
props.setProperty("share.acknowledgement.mode", "explicit");

// Renewal needs the poll loop to see the same record again, one at a time.
props.setProperty("share.acquire.mode", "record_limit");
props.setProperty("max.poll.records", "1");

KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props);
consumer.subscribe(List.of("orders.intake"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));

    // Renew at half the lock the broker actually gave us, not at a guess.
    int waitMs = consumer.acquisitionLockTimeoutMs().orElse(10_000) / 2;

    for (ConsumerRecord<String, String> record : records) {
        startProcessingOnAnotherThreadIfNew(record);

        switch (awaitOutcome(record, waitMs)) {
            case SUCCEEDED ->
                // ACCEPT is the DeleteMessage analogue, with one difference
                // that matters: it marks the record Acknowledged. It does not
                // delete it. The bytes go when topic retention says so.
                consumer.acknowledge(record, AcknowledgeType.ACCEPT);

            case RETRYABLE ->
                // RELEASE is letting the visibility timeout lapse. The
                // delivery count was already incremented by the acquisition.
                consumer.acknowledge(record, AcknowledgeType.RELEASE);

            case UNPROCESSABLE -> {
                // REJECT has no SQS analogue. The record goes to Archived and
                // is copied nowhere. If you need the DLQ back, build it here:
                //   dlq.send(new ProducerRecord<>("orders.intake.dlq",
                //                                 record.key(), record.value()));
                // Note what that does NOT cover: a record exhausted by lapsed
                // acquisition locks or by RELEASE is archived by the broker
                // without this branch ever running. No operator copy exists.
                consumer.acknowledge(record, AcknowledgeType.REJECT);
            }

            case STILL_WORKING ->
                // RENEW is the ChangeMessageVisibility analogue. Each call
                // buys another lock duration; repeated renewal has no
                // documented total ceiling.
                consumer.acknowledge(record, AcknowledgeType.RENEW);
        }
    }

    consumer.commitSync();
}

The three places the bounds do not line up

The lease is bounded per renewal and unbounded in total. A group's share.record.lock.duration.ms defaults to 30000 ms under a ceiling of group.share.max.record.lock.duration.ms — a read-only broker config shipping at 60000 ms, settable in [30000,...,3600000]. But KIP-1240 is explicit that KIP-1222's renewal extends processing indefinitely by repeatedly renewing the acquisition locks; the only off switch is the group config share.renew.acknowledge.enable, default true. Against SQS's absolute 12 hours from first receipt, the asymmetry runs opposite to expectation: 3600000 ms bounds the ceiling, not how long a record stays acquired.

The poison counter has different bounds. A group's share.delivery.count.limit defaults to 5, and the group configuration reference prints open-ended [2,...] — not the bound you get, since KIP-1240 caps a group config's effective value by the broker limits. The real ceiling is group.share.max.delivery.count.limit, shipping at 10 and settable in [5,...,25], over a group.share.min.delivery.count.limit floor of 2 — raised only by editing a read-only broker config and restarting. The counter is also explicitly approximate: those updates are not performed with exactly-once semantics, so the delivery count cannot be relied on to be precise.

The in-flight ceilings only compare after a multiplication. Most standard SQS queues cap acquired-but-undeleted messages at approximately 120,000; Kafka caps them per share-partition, share.partition.max.record.locks defaulting to 2000 under a group ceiling of 4000 from group.share.max.partition.max.record.locks. The Kafka number scales with partition count; the SQS one does not. At that default a 64-partition topic holds 128,000 acquired records, past the approximate SQS quota.

# --- The defaults a share group inherits when it overrides nothing ---

# ~ SQS RedrivePolicy maxReceiveCount (AWS: default 10, range 1-1,000).
group.share.delivery.count.limit=5

# ~ SQS VisibilityTimeout (default 30s, maximum 12h from FIRST receipt).
group.share.record.lock.duration.ms=30000

# ~ the SQS in-flight ceiling (~120,000 per QUEUE on most standard
#   queues). This one is per share-PARTITION, so the comparable figure
#   is this times partition count.
group.share.partition.max.record.locks=2000


# --- KIP-1240: the bounds that decide what a per-group override may be ---
# A group config's effective value is capped by these. Quoting the group
# config's own open-ended range as "the" bound is the error to avoid.

group.share.min.delivery.count.limit=2
group.share.max.delivery.count.limit=10

group.share.min.record.lock.duration.ms=15000
# Raise this before a group may hold a lock longer than one minute:
#   group.share.max.record.lock.duration.ms=600000
group.share.max.record.lock.duration.ms=60000

group.share.min.partition.max.record.locks=100
group.share.max.partition.max.record.locks=4000


# --- Commented ON PURPOSE: uncomment ONLY below three brokers ---
# Share groups create __share_group_state, which defaults to 3 replicas.
# Fewer than 3 brokers: uncomment both BEFORE first using share groups.
# 3 or more brokers: LEAVE THEM COMMENTED. The shipped defaults are
# replication factor 3 and min.insync.replicas 2, and forcing 1/1 here
# leaves the share-group state topic single-replica on a cluster that
# could have replicated it.

# share.coordinator.state.topic.replication.factor=1
# share.coordinator.state.topic.min.isr=1

One loss mode belongs in the retention review, not the queue review: under size-based segment retention the share-partition start offset advances past a deleted segment, which as KIP-932 puts it potentially silently removes records that were eligible for delivery.

When the Right Answer Is a Broker, Not a Log

If the workload is queue-shaped, a broker is the closer semantic match. RabbitMQ quorum queues give you the destination Kafka lacks, but only once configured: a message redelivered more times than the delivery limit, or negatively acknowledged with requeue false, is routed to a dead letter exchange if one exists and discarded otherwise. Running the broker itself — topology, backpressure and 4.3's Raft-only failure model — is covered in the RabbitMQ operations article.

Two asymmetries still break a lift-and-shift, both recent enough that older write-ups miss them. First, the counter is not maxReceiveCount. Since RabbitMQ 4.3 the delivery limit is based on delivery-count rather than acquired-count, so explicit returns — via nack, or AMQP 1.0 modify with delivery_failed=false — no longer count towards it; x-acquired-count tracks assignments, x-delivery-count actual failed deliveries. It increments only for genuine failures — a client crash or connection loss, modified with delivery-failed=true, or basic.reject — so a consumer that nacks forever is capped by nothing. The limit defaults to 20 since RabbitMQ 4.0.

Second, the lease is provisioned, not renewed. RabbitMQ's delivery acknowledgement timeout defaults to 30 minutes, is evaluated at one-minute intervals, and since 4.3 is supported only by quorum queues. It sits on five configuration surfaces, from consumer_timeout in rabbitmq.conf to the rabbitmq:consumer-timeout AMQP 1.0 attach property — none renewable mid-delivery. Expiry is protocol-dependent: an AMQP 1.0 client gets a DISPOSITION with state=released and keeps its link, assigned nothing further until it settles the timed-out message; an AMQP 0.9.1 consumer is cancelled if it supports consumer_cancel_notify, and only otherwise is its channel closed with precondition_failed. Either way the message returns to the queue, incrementing x-acquired-count only — the quorum-queues table marks a consumer timeout as not a failed delivery — so a slow handler can be requeued indefinitely without moving toward the delivery limit or a dead-letter exchange. An SQS visibility lapse does count.

NATS JetStream is the other candidate; the honest citation is its consumer-create JSON Schema, not the docs site. ack_wait — how long a message may remain un-acknowledged before redelivery — is in nanoseconds, defaulting to 30000000000: easy to misread beside a visibility timeout in seconds. max_deliver defaults to -1 and ack_policy defaults to none; the schema records both defaults but defines neither, and nats-server's own consumer.go is where -1 is set up as meaning no limit and AckNone as requiring no acks. So redelivery continues until the message is acknowledged or terminated — no poison cap unless you set one — and none is not Kafka's implicit mode, which still acknowledges on the next poll or commit: none assumes acknowledgement on delivery and never redelivers. The self-managed event backbone comparison makes the platform choice.

The Cutover: Dual-Write, Drain, and the Idempotency You Cannot Skip

Run it in three stages and make the first two boring. Stage one is dual-write with single-read: producers write to SQS and to the self-hosted broker in the same unit of work, SQS stays authoritative, nothing consumes the new side. It buys two numbers no spreadsheet gives you: real ingest volume on real disks, and whether retention means what you assumed.

Stage two is dual-read behind an idempotency store, the only stage where correctness is bought rather than configured. Key it on a producer-assigned business identifier, never a broker-assigned message ID: those differ per transport and stop deduplicating once the second source arrives. Both sides are at-least-once. SQS standard queues state it outright; on a share group, an unsuccessful attempt below share.delivery.count.limit — a RELEASE, or an acquisition lock that lapses — returns the record to Available to be delivered again. Its delivery count stays a poison guard, never a precise receive count.

Stage three is drain and gate. Producers stop and the backlog drains against the ~120,000 in-flight ceiling most standard queues carry — a cap on messages received but not yet deleted, not a rate. Drain rate is that cap divided by each message's time in flight, so extra consumers stop helping once the cap saturates. Drain the DLQ too: a standard-queue message's expiration runs from its original enqueue timestamp, so one sitting there is still ageing. Gate on four conditions, abort on any failure: message-count metrics at zero across a full retention period, broker-side lag flat for the same window, zero duplicate-effect alarms, and a poison path exercised rather than configured.

Click to zoom

If you bridge the two sides with Kafka Connect, read the error defaults first. errors.tolerance defaults to none, meaning any error results in an immediate connector task failure, and errors.deadletterqueue.topic.name is blank by default, so one malformed record stops the bridge and nothing is captured. Connect is the one place in Kafka where a real dead-letter topic ships.

name=orders-archiver
connector.class=com.example.connect.OrdersArchiverSinkConnector
tasks.max=4
topics=orders.intake

# Default is "none": ANY error fails the connector task immediately.
errors.tolerance=all

# Default is blank. No name here means no dead-letter queue at all.
errors.deadletterqueue.topic.name=orders.intake.dlq

# Default is 3, which cannot be created on a two-broker cluster.
# Set this to YOUR cluster's replication factor; 3 is only right at 3+.
errors.deadletterqueue.topic.replication.factor=<REPLICATION_FACTOR>

# Without this you get the failed record and no reason for the failure.
errors.deadletterqueue.context.headers.enable=true

# Log the failure, but not the payload: a dead-letter path is not a reason
# to start writing customer data into broker logs.
errors.log.enable=true
errors.log.include.messages=false

What the Self-Hosted Bill Actually Contains

Fix the SQS side before comparing totals: the invoice is not counting what most teams assume. Each 64 KB chunk of a payload is billed as 1 request, so a 1 MiB API action is 16 requests. One request carries 1 to 10 messages up to 1 MiB total, so chunking applies to that combined payload, not the message count — payload growth alone grows the bill.

Fair queues add a second multiplier, with a trap. API actions for sending, receiving, deleting and changing message visibility in standard queues incur fair queue rates when at least one message carries a message group ID — and are charged both fair queue and standard queue rates. The trap: on standard queues with fair queues, MessageGroupId is only a tenant identifier and does not enforce message ordering. Add one expecting ordering and you get double rating and no ordering.

Model the self-hosted side with the same honesty: brokers sized for peak, replicated storage at your replication factor, inter-zone network, an on-call rotation that is now a salary line rather than infrastructure. Add a cost the SQS model lacks: an ACCEPTed share-group record is not deleted, so a drained queue holds disk for as long as topic retention says — size storage against retention, not queue depth. Any total from today's rates is a model, not a measurement; label it as one for finance. The broader exercise is the cloud repatriation playbook; if consumers scale to zero between bursts, see event-driven autoscaling too.

Security and the Audit Conversation

Know what you stop being able to point at. SQS server-side encryption protects the contents of messages using SQS-managed encryption keys (SSE-SQS) or keys managed in AWS KMS (SSE-KMS); it encrypts messages as soon as SQS receives them and decrypts them only when they are sent to an authorized consumer.

Self-hosted, the same four questions arrive and every answer is an artefact you now produce: who can read the volume the log sits on; how long a message lives, enforced by configuration you can show; who consumed from this queue; and where a failed message went. That last one is what the migration quietly breaks, so raise it with your auditor first: on SQS the answer was a dead-letter queue you could open; on a share group it is Archived, copied nowhere, unless somebody built the republish branch.

Designing the Next Exit In, and the Long Game

The expensive coupling was never the API — SDK calls are a day's work to replace — but the semantics those calls implied, spread through the handlers unlabelled. Define the semantics as your own vocabulary and let an adapter map them onto whichever broker sits underneath: lease, acknowledge, fail-retryable, fail-permanent. When an adapter cannot implement one, you have found the gap on a whiteboard instead of in production.

Keep one abstraction above that: the transactional outbox. The business transaction writes its event into the same database transaction that changes state, and a relay publishes it after. It costs a table and a process, and makes the broker replaceable: the source of truth is a database you already run yourself. For multi-step work, workflow orchestration you host is the better abstraction and the queue becomes an implementation detail.

Protocol is the other lever. RabbitMQ has spoken AMQP 1.0 natively since 4.0, serving it and AMQP 0.9.1 on port 5672 without any additional plugins — and AMQP 1.0 is an independent standard. The Kafka wire protocol is Apache Kafka’s own, but other brokers implement it: Redpanda’s architecture documentation says producers and consumers interact with it using the Kafka API — which of those you run is a separate choice. Plan for the semantics to version, too: group.coordinator.rebalance.protocols is deprecated and will be removed in Kafka 5.0, after which all protocols are controlled solely by feature versions via kafka-features.sh.

Over a decade, queue choice is not about throughput but about who owns the semantics your business logic depends on, and what changing your mind costs. Sovereignty here is unglamorous: the four verbs are yours, the mapping is written down, the next exit a known number rather than a discovery.