RabbitMQ Message Broker: Topology, Backpressure, Failure

RabbitMQ is an open-source message broker: publishers send messages to exchanges, exchanges route them through bindings to queues, and consumers take them off. That is the whole definition, and it is where most writing about it stops. The interesting part starts on the day the broker is yours to run — when a memory alarm has frozen every publisher in the estate, or a node has dropped off the network and somebody has to say whether the orders written in the last ten seconds still exist.

What You Are Actually Choosing When You Run RabbitMQ

Start with the inventory, because it is shorter than it used to be. Quorum queues and streams are the two replicated data structures available, and classic queue mirroring was removed starting with RabbitMQ 4.0. Classic queues remain, but their data is not replicated — and they are still the default queue type unless the virtual host's default is overridden. A cluster nobody configured hands out unreplicated queues. That is a decision too; it is just made by omission.

Three decisions follow, and each is inherited by every service that connects afterwards: which structure holds each workload's data, what happens when a queue or a node fills up, and what the minority side of a network partition does. Get them right on day one and they are invisible. Get them wrong and the fix means re-declaring queues under live traffic.

Scope every piece of failure-mode advice by version before acting on it. RabbitMQ 4.3, released 23 April 2026, made Khepri the only supported metadata store and entirely removed Mnesia, along with the pause_if_all_down, pause_minority and autoheal partition-handling strategies. Guidance that tells you to set cluster_partition_handling describes 4.2 and earlier. Those versions are not gone: as of October 2026 the release table still lists 4.2 patch releases, but its community-support column for 4.2 reads 31 Jul 2026.

Topology Is an Architectural Decision, Not a Diagram

Topology gets drawn once on a whiteboard and then copied by every team that follows, so the exchange choices are worth more thought than they usually get. The exchange types map onto distinct routing jobs:

• direct — routes on an exact routing-key match. One flow, one key, no ambiguity.
• topic — routes on dotted patterns such as orders.*.created. The default choice for event flows, because a new consumer subscribes by adding a binding rather than asking the publisher to change.
• fanout — copies every message to every bound queue and ignores the key. Cheap, and blunt.
• headers — routes on message header values instead of the key. Rarely the clearest option.
• consistent hashing — spreads keys across queues for partitioned consumption; it ships as the rabbitmq_consistent_hash_exchange plugin and is not enabled by default.

One exchange deserves a rule of its own. The default exchange is a pre-declared direct exchange whose name is the empty string, and RabbitMQ automatically binds every queue to it using the queue's own name as the routing key. Publishing to it is the fastest path from zero to working — and it welds the publisher to a queue name. Keep it for what it is good at, such as replies to a known queue, and give every real flow a named exchange. As a judgement call, that is the cheapest optionality in the whole broker: a second consumer becomes a binding, not a code change.

The second rule is RabbitMQ's own: if an optional argument can be set via a policy, consider a policy before hardcoding it in application code. Policies match queues by name pattern per virtual host and can be changed at runtime, which makes length limits and retention reviewable configuration rather than constants scattered across services. The exception is the queue type itself — a policy cannot set or change it. The operator's lever there is the virtual host default queue type, which a client's x-queue-type overrides at declaration. So clients that care declare x-queue-type; operators own the vhost default and everything else:

#!/usr/bin/env bash
# Operator-owned limits. Clients declare the queues with x-queue-type
# (quorum or stream); a policy can never set or change the type.
set -euo pipefail

# Work queues: cap the length and make the broker refuse new publishes
# (and nack them under publisher confirms) instead of dropping the oldest.
rabbitmqctl set_policy -p orders orders-limits '^orders\.' \
  '{"max-length": 500000, "overflow": "reject-publish"}' \
  --priority 10 --apply-to quorum_queues

# Streams: retention is the only bound. Without it the log grows until
# the disk runs out.
rabbitmqctl set_policy -p orders audit-retention '^audit\.' \
  '{"max-age": "7D", "max-length-bytes": 50000000000}' \
  --priority 10 --apply-to streams

Quorum Queue or Stream: the Two Replicated Structures

The quorum queue is a durable, replicated queue based on the Raft consensus algorithm, and RabbitMQ says it should be considered the default choice when you need a replicated, highly available queue. Its guarantee is precise and worth quoting to anyone who asks whether the broker loses messages: a message confirmed back to the publisher should not be lost as long as a majority of the nodes hosting the queue are not permanently made unavailable. Messages that were never confirmed carry no guarantee at all. The same page states the trade-off plainly — quorum queues generally favour data consistency over availability.

Its intended home is narrow and explicit: queues that exist for a long time, need high availability and are critical to the application. The documentation lists where it fits poorly — temporary queues and high declaration churn, the lowest possible latency, very long backlogs of 5M+ messages, and large fan-outs — and points the last two at streams. Quorum queues are always durable and cannot be exclusive. Replica count comes from x-quorum-initial-group-size; three is the minimum that makes practical sense, tolerating the failure of one node, and the group size should be an odd number. Delivery limits and dead-lettering belong to the same queue type and are covered in the SQS exit article; they are not repeated here.

A stream is the other shape. Consumers read non-destructively: any number of them subscribe and read the same messages as many times as they want. Underneath, a stream is an immutable append-only disk log that will grow indefinitely until the disk runs out, unless retention is set with max-age and/or max-length-bytes — and neither is set by default.

• Read semantics. Quorum queue: an acknowledgement removes the message. Stream: reading leaves it in place for the next reader and for replay.
• Backlog. Quorum queue: meant to drain; a 5M+ backlog is outside its intended use. Stream: long backlogs are the point.
• Fan-out. Quorum queue: one copy per bound queue. Stream: one log, many readers.
• What bounds it. Quorum queue: a length limit plus an overflow mode. Stream: retention, and nothing else.

The verdict is that these are different jobs, not competing products. Work that must be done exactly by someone — charge the card, send the invoice — is a quorum queue. Facts that several systems need to read, or that an auditor may need replayed next quarter, belong in a stream. Classic queues keep a job too: data you can afford to lose, such as reply queues that die with their connection. If stream workloads grow until the broker becomes a log platform in all but name, that is a different decision, compared in the Kafka and NATS backbone article.

Click to zoom

Backpressure: What the Broker Does When It Fills Up

RabbitMQ has two resource alarms, and both stop publishers rather than accept data they cannot hold. By default, a node uses about 60% of available RAM before it raises a memory alarm and blocks all connections that are publishing. That limit is not a cap — nodes can use more than it — it is merely the point at which publishers are throttled. In containerised environments such as Kubernetes, RabbitMQ advises against the relative threshold and prefers an absolute one. The second alarm watches disk: when free space drops below a configured limit, 50 MB by default, all producers are blocked. On a node that also holds streams, that default deserves an explicit, larger value.

The property that surprises people is scope. Memory and disk alarms are cluster-wide: if one node goes over the limit, publishing connections on all nodes are blocked. Connections that only consume are not blocked, and deliveries to them continue as usual. The block itself is TCP back pressure, so publishes eventually time out or fail outright — RabbitMQ is explicit that applications must handle this and use publisher confirms to track what the broker actually accepted. Modern clients support the connection.blocked notification, so an application can see the block instead of inferring it from timeouts. Because throttling applies per connection, RabbitMQ advises using individual connections for either producing or consuming, where other design considerations permit. On a shared connection the throttle mostly shows up as delay, but a dedicated consuming connection keeps draining the backlog the alarm is about.

Queue length limits are the second layer. When a limit is reached, the default behaviour drops or dead-letters messages from the front of the queue — the oldest ones. With reject-publish or reject-publish-dlx the most recently published messages are discarded instead, and with publisher confirms the publisher is told via basic.nack. For a work queue, the oldest message is usually the one a customer has waited on longest, so dropping it silently is the wrong default; reject-publish turns overflow into a signal the publisher can act on. Quorum queues support drop-head and reject-publish, do not support reject-publish-dlx, and their reject-publish can overshoot the limit by at least one message — more while in-flight publishes are still arriving, so do not size it as exactly N+1.

Prefetch is the consumer-side valve. RabbitMQ's guidance is that values in the 100 through 300 range usually offer optimal throughput, while a prefetch of 1 significantly reduces it. Quorum queues cap consumer prefetch at 2,000 to limit runaway Raft log growth, and they do not support global QoS — consuming from a channel with it set returns a channel error. Prefetch must be per consumer. Here is a publisher and a consumer that respect all of this, using pika:

"""Publish and consume on separate connections (pika 1.4).

The publisher uses confirms and watches connection.blocked; the consumer
uses a per-consumer prefetch and manual acks. Queue type is declared here,
by the client, because a policy cannot set it.
"""
import logging
import os

import pika
from pika.exceptions import AMQPConnectionError, NackError, UnroutableError

log = logging.getLogger("orders")

HOST = "broker.messaging.svc.cluster.local"  # the RabbitmqCluster below
EXCHANGE = "orders"
QUEUE = "orders.created"
# Topic keys: "*" is exactly one dot-separated word, so the binding
# orders.*.created matches orders.web.created but not orders.created.
ROUTING_KEY = "orders.web.created"
BINDING_KEY = "orders.*.created"


def params(name: str) -> pika.ConnectionParameters:
    return pika.ConnectionParameters(
        host=HOST,
        # The vhost must already exist. Never the default guest user: it is
        # refused from any address other than localhost.
        virtual_host="orders",
        credentials=pika.PlainCredentials(
            os.environ["RABBITMQ_USER"], os.environ["RABBITMQ_PASSWORD"]
        ),
        heartbeat=30,
        # Tear the connection down if the broker keeps it blocked this long.
        blocked_connection_timeout=60,
        client_properties={"connection_name": name},
    )


def declare_topology(channel) -> None:
    """Idempotent, so both sides run it and either may start first.

    A mandatory publish is unroutable until the queue and binding exist.
    """
    channel.exchange_declare(EXCHANGE, exchange_type="topic", durable=True)
    channel.queue_declare(
        QUEUE, durable=True, arguments={"x-queue-type": "quorum"}
    )
    channel.queue_bind(QUEUE, EXCHANGE, routing_key=BINDING_KEY)


class Publisher:
    def __init__(self) -> None:
        self.blocked = False
        self.conn = pika.BlockingConnection(params("orders-publisher"))
        self.conn.add_on_connection_blocked_callback(self._on_blocked)
        self.conn.add_on_connection_unblocked_callback(self._on_unblocked)
        self.channel = self.conn.channel()
        declare_topology(self.channel)
        self.channel.confirm_delivery()

    def _on_blocked(self, _conn, frame) -> None:
        self.blocked = True
        log.warning("broker blocked publishing: %s", frame.method.reason)

    def _on_unblocked(self, _conn, _frame) -> None:
        self.blocked = False
        log.info("broker unblocked publishing")

    def publish(self, body: bytes) -> bool:
        """Return True only when the broker confirmed the message."""
        while self.blocked:
            self.conn.sleep(1.0)  # keeps servicing I/O so the unblock arrives
        try:
            self.channel.basic_publish(
                exchange=EXCHANGE,
                routing_key=ROUTING_KEY,
                body=body,
                properties=pika.BasicProperties(
                    delivery_mode=pika.DeliveryMode.Persistent
                ),
                mandatory=True,
            )
            return True
        except (NackError, UnroutableError):
            log.error("broker refused or could not route the message")
            return False
        except AMQPConnectionError:
            log.error("connection lost; outcome unknown, republish after reconnect")
            raise


def handle(body: bytes) -> None:
    """Idempotent business logic: a redelivered message must be harmless."""
    log.info("processing %d bytes", len(body))


def consume() -> None:
    conn = pika.BlockingConnection(params("orders-consumer"))
    channel = conn.channel()
    declare_topology(channel)
    # Per-consumer prefetch; global QoS is a channel error on quorum queues.
    channel.basic_qos(prefetch_count=200)

    def on_message(ch, method, _properties, body) -> None:
        handle(body)
        ch.basic_ack(delivery_tag=method.delivery_tag)

    channel.basic_consume(QUEUE, on_message_callback=on_message, auto_ack=False)
    channel.start_consuming()

Clients that never set a prefetch are the common failure, and they are rarely yours to fix. A server-wide default applies whenever a consumer does not specify one, set in the advanced configuration file:

%% advanced.config
[
  {rabbit, [
    {default_consumer_prefetch, {false, 250}}
  ]}
].

Failure Modes After 4.3: Everything Replicated Is Raft

The reworked partitions guide states it in one line: starting with 4.3.0, the metadata store (Khepri), quorum queues and stream coordinators are all Raft-based. That removes a whole class of operator decisions. Mnesia's weakness was its partition recovery — its replication assumed the system could afford to throw away all data on one side of a partition. That page still describes Mnesia's removal as future work; the partitions guide and the 4.3 release post are the current word. On 4.3 the old cluster_partition_handling keys are still accepted but have no effect, and the guide says to remove them. A config file that still sets them is documentation of a cluster that no longer exists.

What happens instead depends on which side you are on, and on whether a leader was lost. Where a majority is missing, operations that need Raft consensus — publishes, consumer acknowledgements, metadata store updates — cannot make progress: they are retained until a majority is restored, or fail with a timeout. When only a follower is cut off, the majority side keeps reading normally; on the disconnected side, Khepri serves local cached reads, quorum queue deliveries pause, and stream consumers can still read locally available data. When the link returns, the reconnected replica receives the missing log entries from the leader and updates its local state to match it. Do not read that as a promise that the cut-off side loses nothing: the leader's log wins, and an operation that timed out has to be retried by the client.

If a queue's leader was on the lost side, the majority elects a new one, and that window has its own rules. During it, write operations are rejected until a new leader is elected; for publishes pending a confirm, that can mean a negative acknowledgement. Yet for quorum queues most published messages do not fail outright — they are retained and forwarded to the new leader once it is elected, with confirms arriving late or, in some scenarios, as basic.nack. That retain-and-forward behaviour is documented for quorum queues. Quorum queue deliveries always go through the leader, so their consumers see no new deliveries, and consumer registration (basic.consume) and basic.get block until a leader exists or fail with a client-side timeout. Streams differ: stream consumers can continue reading already-replicated data from any replica they can reach, and are unaffected by the leader change. Start to finish an election usually takes a few seconds. Then the part that matters for application code: messages delivered but not yet acknowledged are re-queued and redelivered by the new leader, so a quorum queue consumer may receive the same message more than once. Publisher confirms tell you what was accepted; idempotent consumers make the duplicates harmless. Neither is optional.

Click to zoom

Sizing follows from majorities. The clustering guide stresses odd node counts and strongly discourages two-node clusters: two nodes have no majority once either is lost. Growth has a trap of its own — a new node hosts no quorum queue members unless the operator explicitly adds them: rabbitmq-queues add_member per queue, rabbitmq-queues grow for a set, or the continuous membership reconciliation the same page offers as a more automated route. Until one of those happens, a fourth node adds no resilience to any existing queue.

Kubernetes adds one more. An operator-defined readiness probe can stall a cluster restart under the OrderedReady pod management policy; the Parallel policy avoids it. If you run on Kubernetes, the RabbitMQ Cluster Operator defaults to one replica, highly discourages even counts and says odd numbers must be used, and cannot scale a cluster down — only to zero. Choose the replica count as if it were permanent, because it nearly is. The operator already sets Parallel pod management, and it still writes pause_minority into its own defaults file; on 4.3 that key is accepted and ignored, so leave it rather than fight it. Storage deserves the same care as any stateful workload on Kubernetes:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: broker
  namespace: messaging
spec:
  image: rabbitmq:4.3.4-management # pin it: older operators default older images
  replicas: 3 # odd, and effectively permanent: the operator cannot scale down
  resources:
    requests:
      cpu: "2"
      memory: 8Gi
    limits:
      cpu: "2"
      memory: 8Gi
  persistence:
    storageClassName: fast-local # your StorageClass
    storage: 200Gi
  rabbitmq:
    additionalConfig: |
      # Absolute thresholds: RabbitMQ advises against relative ones in containers.
      vm_memory_high_watermark.absolute = 4Gi
      disk_free_limit.absolute = 10GB

Watch the signals, not just the pods. Alarm state, queue depth against its limit and unconfirmed publishes belong on the same dashboards as everything else; the self-hosted observability stack is where they live. Queue depth is also the honest input for scaling consumers, and queue length is the RabbitMQ scaler mode KEDA aims at consumer replicas.

Upgrades and the Support Clock

The upgrade path is fixed: upgrading to 4.3.x is only possible from 4.2.x, so a 3.13.x cluster goes to 4.2 first, then to 4.3. All stable feature flags must be enabled before an upgrade, or the upgrade may fail — check that as a separate, earlier step, not inside the maintenance window. RabbitMQ recommends a rolling, in-place upgrade, with nodes upgraded one by one, and offers a blue-green deployment as the safer option at the cost of running a second cluster while traffic moves.

Then put the dates on the calendar, quoted as RabbitMQ's release table publishes them as of October 2026. 4.3.0 shipped 23 Apr 2026, and community support for 4.3 ends 31 Jan 2027. Community support for 4.2 ended 31 Jul 2026; its commercial support is listed to 30 Jun 2030, and the table labels every commercial date as indicative. The community window for a release is measured in months, not years, so plan upgrades as a recurring operation with an owner, not a project that happens when something breaks.

What an Auditor Is Shown

A broker under audit has to show that accepted messages are not silently lost and that configuration is controlled. The evidence maps cleanly:

• Delivery safety — publisher confirms on every publishing path, plus the documented guarantee: a confirmed message in a quorum queue should not be lost as long as a majority of the nodes hosting it are not permanently made unavailable. That guarantee does not cover drop-head overflow, message TTL, delivery-limit dead-lettering or deleting the queue.
• Configuration control — policies exported with the broker's definitions and kept in version control, so length limits, overflow modes and retention have a review history.
• Node authentication — nodes and CLI tools share an Erlang cookie whose file must be accessible only to its owner, permissions 600 or similar. Treat it as a credential.
• Dependency register — the RabbitMQ server and its tier-1 core plugins are licensed under MPL 2.0.

Exit Ramps and the Long Game

The cheapest exit is the one designed in early. Speak an open protocol — AMQP 1.0 is native since RabbitMQ 4.0, as the SQS exit article covers — so clients are not tied to one broker's library. Keep topology and policies in version control, so a second cluster can be stood up from text. Blue-green, the safer upgrade strategy, doubles as a migration rehearsal: running two clusters side by side is the same skill as leaving one.

Know, too, when a workload has outgrown the broker. A stream that has become the system of record for many readers is a log platform in all but name, and the Kinesis-to-Kafka comparison covers what that move costs. Over a decade the queue types, the partition model and the CLI will all change again — 4.3 just proved it. What lasts is the discipline underneath: replicated structures chosen per workload, overflow that signals instead of drops, clients that confirm and tolerate duplicates, and an upgrade cadence somebody owns.