Retry and Dead-Letter Queues¶
RabbitKit provides a built-in retry system based on RabbitMQ dead-letter exchanges and per-queue TTL. Messages that fail are routed through a series of delay queues before being promoted to a dead-letter queue (DLQ) if all retry attempts are exhausted.
Durability, sanitized triage headers, and handoff failures (0.12)¶
Three RetryConfig knobs make the retry chain's reliability explicit. All
defaults preserve the legacy topology, so upgrading never re-declares a queue:
| Knob | Values | Default |
|---|---|---|
delay_queue_type |
classic, quorum, inherit (match the source queue) |
classic |
dlq_queue_type |
inherit (quorum when the source is quorum), classic, quorum |
inherit |
error_detail |
sanitized (redacted + capped), omit, raw (legacy) |
sanitized |
handoff |
RetryHandoffConfig(...) — bounded backoff when the retry publish itself fails |
bounded |
The DLQ triage headers carry x-rabbitkit-error-category (the failure
category, see Error Classification) alongside
x-rabbitkit-error-type, and
x-rabbitkit-error-message is redacted (URL passwords, password=/token=
pairs, bearer tokens, long opaque tokens) and length-capped. A failed
delay-queue handoff never acks the source; it is nack-requeued and counted in
rabbitkit_retry_handoff_failures_total with capped exponential backoff. See
Bulk Operations & Reliability Profiles.
Retry policy¶
| Field | Meaning | Default |
|---|---|---|
max_retries / delays |
the ladder; one fixed-TTL delay queue per tier (whole seconds) | 4 / (5, 30, 120, 600) |
RetryConfig.exponential(max_retries=, initial_delay=, multiplier=, max_delay=) |
builds delays as min(max_delay, initial_delay * multiplier**i) |
— |
max_retry_age |
seconds since the FIRST failure after which a transient failure is terminal (retry-age-exceeded) |
None (unbounded) |
terminal_action |
"dead_letter": confirmed, metadata-stamped copy to <queue>.dlq, then ack; "reject": legacy broker DLX |
"dead_letter" |
jitter_mode / jitter_shards / jitter_factor |
TTL spread across shard queues (see the end of this page) | off |
unknown_policy |
severity for exceptions no predicate/type tuple classifies | PERMANENT |
from rabbitkit import RetryConfig
retry = RetryConfig.exponential(max_retries=5, initial_delay=2, multiplier=3, max_delay=300, max_retry_age=3600)
# delays == (2, 6, 18, 54, 162)
There is no immediate-retry mode: every retry waits in a delay queue, and a failed handoff to the delay queue is braked before the message can come back (see Handoff failures below), so a failure can never spin the handler.
Handler settlement signals are not failures. AckMessage,
NackMessage and RejectMessage raised by a handler pass straight through
RetryMiddleware — whatever unknown_policy or a predicate says — and the
pipeline settles exactly as asked.
Handoff failures¶
If the publish to the delay queue is returned, nacked, times out or raises
(the delay queue was deleted, a broker alarm), the source is never acked. On
async routes the middleware holds the delivery for the backoff
(RetryHandoffConfig: capped exponential, backoff_max 30 s by default,
cancellable) and only then nack-requeues it, so the broker cannot hand it
straight back. Once the tracker is EXHAUSTED (max_consecutive_failures or
recovery_deadline), every failing delivery is held for the full
backoff_max — consumption of that queue is effectively paused at
prefetch messages per backoff_max — and one CRITICAL log line is written
per outage; wire on_handoff_exhausted to stop the route if you prefer.
The sync path never sleeps on pika's I/O thread; it reports the backoff via
on_handoff_failure.
Every Route Gets a Dead-Letter Path by Default¶
In RabbitMQ, a message rejected with requeue=False is permanently discarded unless the queue has a dead-letter exchange. A plain subscriber with no retry can still reject — a handler raising ValueError on a malformed payload is classified as a permanent error and rejected. To prevent silent loss, RabbitKit auto-provisions a {queue}.dlq for every route that can reject, controlled by SafetyConfig.reject_without_dlx:
| Policy | Behavior | Use |
|---|---|---|
"auto_provision" (default) |
Declares {queue}.dlq and wires the source queue's DLX to it |
Most consumers |
"error" |
Startup fails with UnsafeTopologyError if a rejecting route has no DLX |
Topology managed externally (Terraform/definitions) |
"discard" |
Rejected messages may be discarded; warns once per route | Low-value/ephemeral data only — explicit opt-in to loss |
from rabbitkit import RabbitConfig, SafetyConfig, RejectWithoutDLXPolicy
config = RabbitConfig(safety=SafetyConfig(reject_without_dlx=RejectWithoutDLXPolicy.ERROR))
# Or per route:
@broker.subscriber(queue="low-value-telemetry", reject_without_dlx="discard")
def handle(body: bytes) -> None: ...
Retry-enabled routes and queues with a manually configured dead_letter_exchange already have a dead-letter path and are left untouched. ACK_FIRST routes (which ack before the handler and can never reject) are skipped. The policy applies only under TopologyMode.AUTO_DECLARE — in PASSIVE_ONLY/MANUAL modes RabbitKit does not own queue arguments and cannot know whether an externally managed DLX exists.
Upgrading an existing deployment: auto-provisioning adds x-dead-letter-exchange/x-dead-letter-routing-key to the source queue's declare arguments. RabbitMQ rejects re-declaring an existing queue with different arguments (406 PRECONDITION_FAILED → a clear ConfigurationError at startup). Either delete/re-create the queue, apply the matching arguments via a policy on the broker, switch that route to reject_without_dlx="discard"/"error", run with TopologyMode.PASSIVE_ONLY, or set SafetyConfig(on_topology_conflict="warn_continue") — which warns and continues using the existing queue definition for just the conflicting queues while still declaring the rest (unlike PASSIVE_ONLY, which skips declaration entirely).
A missing DLQ never swallows a message¶
The dead-letter path only protects a message if the DLQ exists when the
message is rejected. A classic queue (and a quorum queue without
dead-letter-strategy: at-least-once) dead-letters at most once: if the
target queue is gone, RabbitMQ drops the message and nobody is told.
Measured on 3.13 and 4.1: a permanent failure after the auto-provisioned
{queue}.dlq was deleted lost the message on both queue types.
The async broker therefore checks that the DLQ exists before it
dead-letters anything (handler failures, retry-exhausted messages,
RejectMessage(requeue=False), NackMessage(requeue=False),
NACK_ON_ERROR). If it is missing, the message is requeued instead, after
a backoff that starts at 0.5 s and doubles to 5 s while the DLQ stays
missing, and an ERROR is logged once per 10 s window
(HandlerPipeline.dead_letter_blocked_total counts them). The message
survives until an operator restores the DLQ; restarting the service with
TopologyMode.AUTO_DECLARE recreates it.
Limits: the check covers dead-letter targets reached through the default
exchange (x-dead-letter-exchange: "" with the DLQ name as the routing
key), which is what the retry and auto_provision paths declare. A
dead-letter exchange with its own bindings is not verified. A positive
check is cached for 5 s, so a DLQ deleted inside that window can still drop
a message. The sync broker does not run the check yet.
A quorum source declared with dead_letter_strategy="at-least-once" is not
probed: the broker itself keeps a dead-lettered message on the source until
the DLQ exists again (messages_dlx in the management API), so the message
is rejected at once and the handler is not run again. Every other source
gets the requeue-with-backoff above; once the DLQ is back, the next failure
lands there with the full metadata. Verified on RabbitMQ 4.1 for both
strategies.
Full Topology Example¶
For a queue named orders.created, RabbitKit declares the following topology automatically when RetryConfig is provided:
orders.exchange (topic)
|
| routing key: orders.created
v
orders.created.queue ──(transient failure: middleware republishes)──┐
^ v
| orders.created.queue.retry.1
| (TTL=5000ms, DLX="" routed
| back by QUEUE NAME)
└──(TTL expiry dead-letters via the ────┘
DEFAULT exchange, routing key =
"orders.created.queue")
|
| (after max_retries exhausted: reject → source queue's own DLX)
v
orders.created.queue.dlq
There is no retry exchange. Each delay queue is a standard durable
classic queue with a uniform queue-level x-message-ttl and
x-dead-letter-exchange: "" (the default exchange) with the source
queue's name as the dead-letter routing key. On the default exchange a
routing key that equals a queue's name always delivers directly to that
queue — completely independent of how the source queue is bound to its
real exchange. (Dead-lettering back through the source's own exchange —
what an earlier design did — silently loses the message whenever the queue
is bound by a pattern like orders.* rather than literally by its name.)
How Retry Queues Are Declared with TTL¶
For RetryConfig(max_retries=3, delays=(5, 30, 120)) on a queue named
orders.created.queue, RabbitKit declares one delay queue per attempt:
| Queue | x-message-ttl | x-dead-letter-exchange | x-dead-letter-routing-key |
|---|---|---|---|
orders.created.queue.retry.1 |
5000 ms | "" (default) |
orders.created.queue |
orders.created.queue.retry.2 |
30000 ms | "" (default) |
orders.created.queue |
orders.created.queue.retry.3 |
120000 ms | "" (default) |
orders.created.queue |
One delay queue per retry attempt, each with a uniform queue-level TTL
(never per-message TTL — RabbitMQ only expires messages at the queue head,
so mixed TTLs in one queue would let a long delay block expired short ones).
By default RetryConfig requires len(delays) >= max_retries
(strict_delays=True raises ConfigValidationError otherwise); pass
strict_delays=False to reuse the last delay for the remaining attempts.
How Retry Count Is Tracked¶
The retry count is stored in the AMQP message header x-rabbitkit-retry-count (configurable via RetryConfig.retry_header). On each retry delivery:
- RabbitKit reads
x-rabbitkit-retry-countfrom the incoming message headers and takes the larger of it and the delay tiers the broker'sx-deathhistory shows (<queue>.retry.<n>⇒ at leastnretries happened). - It increments the value and sets it on the outbound retry message — never lowering a larger value already on the message.
- The incremented value is used to select the appropriate delay queue for the current attempt.
Headers are preserved across re-deliveries. If x-rabbitkit-retry-count is absent, it is treated as 0 (first attempt).
Metadata on every retry and DLQ copy¶
| Header | Meaning |
|---|---|
x-rabbitkit-retry-count |
retries done so far (selects the delay tier) |
x-rabbitkit-attempts |
handler executions including this one (never lowered) |
x-rabbitkit-original-message-id |
the id every copy carries; an id-less message gets one on its first handoff |
x-rabbitkit-original-queue / -exchange / -routing-key |
where the message was consumed / originally published |
x-rabbitkit-first-failed-at / -last-failed-at |
ISO-8601; first is set once (the max_retry_age clock) |
x-rabbitkit-error-class / -error-type / -error-message |
the CURRENT attempt's exception class and sanitized message |
x-rabbitkit-error-category |
the CURRENT attempt's failure category: retryable, permanent, invalid_message, unknown_outcome, infrastructure (see Error Classification) |
x-rabbitkit-failure-category |
transient / timeout-unknown on a retry copy; permanent / exhausted / retry-age-exceeded on a DLQ copy |
x-rabbitkit-dead-lettered-at |
DLQ copies only |
x-rabbitkit-original-expiration |
the producer's expiration, which is not copied onto delay/DLQ copies |
The AMQP properties are preserved too: message_id, correlation_id,
timestamp, type, app_id, user_id, priority, reply_to,
content_type, delivery_mode, and every application header (trace context
such as traceparent, idempotency keys).
The body and content_encoding are the ones that came off the wire.
CompressionMiddleware decompresses message.body in place (and leaves
content_encoding as delivered), so a copy built from message.body would
carry a decoded body labelled gzip; before this was fixed every retried
compressed message failed to decompress on its next delivery and went to the
DLQ. Every retry and DLQ copy is built from RabbitMessage.wire_body /
wire_content_encoding, which an on_receive hook cannot change, so the copy
decodes exactly like the original (verified on a real broker: a compressed
message that fails once succeeds on attempt 2, and its terminal DLQ copy is
still gzip).
expiration is the one exception: on a delay queue it would expire the copy
early and collapse the backoff; on the DLQ it would delete the failure
record.
Trust model. These headers are client-controlled. rabbitkit never lowers
them and clamps the count to max_retries, but a producer that strips every
header (including x-death) restarts the budget. Bound that case with
max_retry_age and, against crash loops, a quorum x-delivery-limit.
A producer that sets x-rabbitkit-retry-count HIGH skips the retries. With
RetryConfig(trust_retry_header=False) the header is not read at all: the
count comes only from the x-death entries RabbitMQ adds when a copy expires
out of <queue>.retry.<n>, so a forged count changes nothing (verified on a
real broker: a message published with x-rabbitkit-retry-count: 99 still
gets every retry). The header is still written for observability. The
default stays True because TestBroker writes no x-death, and suites
that simulate a later attempt by setting the header rely on it.
The Retry-Count Header Is Not Trusted Input (H5)¶
The retry-count header is read verbatim from an inbound AMQP message — there
is no broker-side attestation distinguishing a value this middleware wrote
during its own delay-queue round trip from one a producer set directly.
RetryMiddleware clamps every value read from the header to [0,
max_retries] before using it, regardless of what the header claims:
- A negative value (e.g. a producer trying to reset the counter for
unbounded retries) clamps to
0and is treated as a fresh first attempt — it can never produce a negative attempt number, which would otherwise target a delay queue likeorders.retry.-4that was never declared (the publish would silently target a non-existent queue on the default exchange and the message would be lost, not retried). - An absurdly large value (e.g. a producer trying to force every message
straight to the DLQ, skipping retries) clamps to
max_retriesand is treated as exhausted. - A non-numeric/malformed value is treated the same as a missing header (
0) rather than raising, so a garbage header degrades gracefully instead of crashing the pipeline.
This makes max_retries an enforced ceiling independent of the header's
configured value being read from a trusted or untrusted source — but it is
still an application-level check, not a broker-enforced one. For a
broker-enforced backstop on top of this (e.g. against a misbehaving consumer
that never settles a message, independent of anything RetryMiddleware does),
prefer quorum queues for the source queue and set x-delivery-limit — the
broker itself dead-letters a message after that many redeliveries, with no
dependency on any header at all.
Stripping reserved headers from application publishes¶
RetryConfig(trust_retry_header=False) ignores the header and counts only the
broker's x-death history, but a producer can put a forged x-death (or
x-rabbitkit-first-failed-at, which drives max_retry_age) on a publish too.
PublisherConfig(strip_reserved_headers=True) removes rabbitkit's
retry/DLQ/replay headers, x-death and a custom retry_header from every
broker.publish() (and publish_many / iter_publish), logging one WARNING
per header name. rabbitkit's own retry and DLQ copies and DLQInspector
replays do not go through broker.publish() and keep them. Both settings are
off by default and on in the CRITICAL profile (see
Reliable Messaging).
Crash-loop backstop (M5)¶
The header count only advances on the delay-queue round trip
(RetryMiddleware acks the source and republishes). A handler that crashes
the process mid-message (OOM, segfault, SIGKILL) never acks, so the broker
redelivers the same message forever — the header never increments and the
retry ladder is never engaged. The broker-side backstop for this is a quorum
source queue with x-delivery-limit: it counts redeliveries and
dead-letters once the limit is hit, regardless of the header. rabbitkit
preserves both the quorum type and the delivery limit when retry re-declares
the source queue with its DLX routing, so the two compose:
from rabbitkit import RabbitQueue, QueueType, RetryConfig
@broker.subscriber(
queue=RabbitQueue(name="orders", queue_type=QueueType.QUORUM, delivery_limit=20),
retry=RetryConfig(max_retries=4, delays=(5, 30, 120, 600)),
)
def handle(order: Order) -> None:
...
# Normal retries ack+republish (source redelivery count stays ~0 per cycle),
# so delivery_limit only trips on a true crash-loop → message dead-lettered
# to orders.dlq by the broker. Keep delivery_limit comfortably above the
# redeliveries a normal restart might cause.
What Happens After max_retries¶
A failure is terminal when the error is permanent, the retries are exhausted
(x-rabbitkit-retry-count equals max_retries), or max_retry_age has
passed since the first failure. With the default
terminal_action="dead_letter", RetryMiddleware then publishes a copy to
<queue>.dlq (mandatory, confirm-gated) carrying the CURRENT attempt's
metadata (table above), and only after the broker confirms it acks the
source — the same confirmed-publish-then-ack pattern as a retry. The
exception is re-raised tagged _rabbitkit_terminal (so an outer
ExceptionMiddleware still sees it) and _rabbitkit_dead_lettered, which
tells the pipeline the message is already settled.
If that DLQ publish fails (DLQ missing, returned, nacked, timed out), the middleware hands the delivery back to the pipeline, which settles it exactly once:
| DLQ | Source | What happens |
|---|---|---|
| exists (publish failed for another reason) | any | reject; RabbitMQ moves it via the source's x-dead-letter-* arguments (only x-death, not the terminal metadata) |
| missing | quorum, dead_letter_strategy="at-least-once" |
reject; the broker keeps it on the source until the DLQ is back; the handler is not run again |
| missing | anything else (async broker) | requeued after a 0.5 s to 5 s backoff (see A missing DLQ never swallows a message); once the DLQ is back the next failure lands there with metadata |
Failures that never reach RetryMiddleware¶
An on_receive failure (a body that does not decompress, a bad
signature), a body over ConsumerConfig.max_body_bytes, a failure on a
route without retry, and a transient failure on redelivery are settled by
the pipeline, not by RetryMiddleware. By default they are rejected into
the source's DLX and their DLQ entry carries only x-death. With
SafetyConfig(dead_letter_with_metadata=True) (async broker; set by the
CRITICAL profile) the pipeline first publishes the same metadata-stamped
copy (x-rabbitkit-error-category such as invalid_message,
x-rabbitkit-original-queue, ..., the wire body and encoding) to the
route's DLQ, mandatory=True and confirmed, and acks the delivery only
then. If the copy is not confirmed, the table above applies. Only a DLQ
reached through the default exchange (the retry DLQ or the auto_provision
safety DLQ) is addressed; a route whose dead-letter exchange has its own
bindings keeps the reject.
The exception carries _rabbitkit_dead_letter_failed: an outer
ExceptionMiddleware re-raises it even with swallow_permanent=True or a
matching fallback handler, because recovering it would ack a message that
exists nowhere else. terminal_action="reject" always takes the broker
path.
Note for TestBroker users: on the default path the consumed message's
disposition is acked (the DLQ copy is in broker.published_messages), not
rejected.
The DLQ is a plain durable queue with no TTL. Messages accumulate there until replayed or discarded manually.
Validated against a real broker¶
Both halves of retry -- the delay-queue topology and the RetryMiddleware
that actually routes failures into it -- are exercised end-to-end against a
real RabbitMQ broker (via testcontainers,
not a mock) in
test_async_retry_exhaustion_to_dlq.
That test raises a transient error, asserts the handler is called exactly
max_retries + 1 times (proving the delay queue actually redelivers), and
consumes the .dlq queue to confirm the exhausted message lands there. It runs
on every PR and nightly. See the README's
Real-broker integration tests
section for how to run it locally.
Code Example¶
from rabbitkit import AsyncBroker, ConnectionConfig, RabbitConfig
from rabbitkit.core.config import RetryConfig
config = RabbitConfig(connection=ConnectionConfig.from_url("amqp://guest:guest@localhost/"))
broker = AsyncBroker(config)
retry = RetryConfig(
max_retries=3,
delays=(5, 30, 120), # seconds: attempt 1→5s, attempt 2→30s, attempt 3→120s
)
@broker.subscriber(
queue="orders.created",
retry=retry,
)
async def handle_order(order_id: str) -> None:
await process_order(order_id)
async def main() -> None:
await broker.run() # runs until SIGINT/SIGTERM, then drains and stops
With this configuration:
- First failure: message waits 5 seconds in
orders.created.retry.5sthen redelivered. - Second failure: message waits 30 seconds in
orders.created.retry.30s. - Third failure: message waits 120 seconds in
orders.created.retry.120s. - Fourth failure: message is published to
orders.created.dlq.
Error Classification¶
Every handler failure is classified into a category. The category's
severity decides retry versus DLQ; the category itself is written to
x-rabbitkit-error-category on every retry and DLQ copy, labels
rabbitkit_messages_retried_total / rabbitkit_messages_dead_lettered_total
(category="..."), and appears in the pipeline's failure log line.
| Category | Severity | Default examples | What happens (with RetryConfig) |
|---|---|---|---|
retryable |
transient | TimeoutError, ConnectionError / OSError, EOFError |
delayed retry |
infrastructure |
transient | aio-pika/aiormq AMQPConnectionError, ChannelClosed, ChannelInvalidStateError; pika's connection/channel errors; InfrastructureError |
delayed retry, labelled separately |
unknown_outcome |
transient | HandlerTimeoutError (a processing deadline expired) |
delayed retry; the handler must be idempotent |
permanent |
permanent | ValueError, KeyError, TypeError, ..., and any unclassified exception |
DLQ on the first attempt |
invalid_message |
permanent | body decode/validation failures, on_receive failures (bad gzip, bad signature), MissingDependencyError, InvalidMessageError |
DLQ on the first attempt |
shutdown |
— | asyncio.CancelledError, KeyboardInterrupt, SystemExit |
never retried: requeued, and the cancellation propagates |
Evaluation order: shutdown; a category the exception declares itself
(rabbitkit_error_category on the class or instance, which the typed
exceptions below and rabbitkit's decode/on_receive tagging set); your
predicates; the registered infrastructure errors; the transient and
permanent type tuples; unknown_policy (default PERMANENT). A declared
category wins over predicates, so a decode failure stays terminal even with
a catch-all predicate or unknown_policy=TRANSIENT.
core/ imports no transport, so the client-library errors are registered by
the brokers when they are constructed
(rabbitkit.core.errors.register_infrastructure_errors). aiormq's
ChannelClosed is the base of every channel-level close, so a 403/404/406
raised inside a handler is retried too, bounded by max_retries.
Classify your own errors with typed exceptions or predicates:
from rabbitkit import ErrorCategory, InfrastructureError, InvalidMessageError, RetryConfig
def handle_order(payload: dict) -> None:
if "order_id" not in payload:
raise InvalidMessageError("missing order_id") # DLQ, category invalid_message
try:
save(payload)
except DatabaseUnavailable as exc:
raise InfrastructureError("orders DB unavailable") from exc # retried
def by_http_status(exc: BaseException) -> bool | ErrorCategory | None:
status = getattr(exc, "status", None)
if status in (502, 503, 504):
return ErrorCategory.INFRASTRUCTURE
if status is not None and 400 <= status < 500:
return ErrorCategory.PERMANENT
return None # no opinion
retry = RetryConfig(max_retries=3, delays=(5, 30, 120))
# RetryMiddleware(retry, predicates=[by_http_status])
A predicate may still return True (transient) / False (permanent).
Inspecting the DLQ¶
Use the RabbitKit CLI to view messages sitting in a DLQ without consuming them:
This prints a summary of messages including headers, routing key, original exchange, and failure reason if recorded.
inspect fetches up to --limit messages with basic_get, holds them all
unacked until the fetch ends, then requeues them, so each message is shown
once. Before 0.19 it requeued each message straight away; the message went
back to the head and --limit 20 printed the first one 20 times.
Replaying from the DLQ¶
To re-publish messages from a DLQ back to the queue each one failed on:
or through an exchange, by each message's original routing key:
The second argument is the target exchange name ("" = default exchange).
Routing rule, shared by the CLI and DLQInspector.replay(): an explicit
routing key (--routing-key / target_queue) wins; to the default exchange
a message is routed by its original QUEUE (x-rabbitkit-original-queue, then
the broker's x-last-death-queue / x-death); to a named exchange by its
original ROUTING KEY (x-rabbitkit-original-routing-key, then x-death).
Headers are kept — including x-rabbitkit-retry-count, meaning a previously
max-retried message is terminal after ONE failed attempt and returns to the
DLQ. Pass --reset-retry-count to grant a fresh retry ladder: it strips the
count, the broker's x-death/x-first-death-*/x-last-death-* history and
x-rabbitkit-first-failed-at, keeping the old count as
x-rabbitkit-previous-retry-count.
Loop prevention. Each replayed copy carries x-rabbitkit-replay-count,
x-rabbitkit-replayed-at and x-rabbitkit-replayed-by. A message already
replayed --max-replays times (default 3; DLQInspector.replay(max_replays=3))
is refused and stays on the DLQ; --ignore-replay-limit /
max_replays=None lifts that, deliberately.
Audit. Every run logs a start record (actor, queue, target, predicate,
limit, reset flag, replay limit) and a finish record with the counts
(replayed, failed, requeued, skipped, refused) to the rabbitkit.dlq.audit
logger; route it to your audit sink. --actor / actor= (default: the OS
user) names who ran it.
Authorization. The hard boundary is RabbitMQ's: run replays as a
dedicated user with read on the DLQs and write on the replay targets
only. On top of that:
DLQInspector(transport, authorizer=fn)callsfn(ReplayRequest)before everyreplayandpurge. The request carries the operation, the actor, whether the actor was passed explicitly, the queue, the target and the options (max_replays=Noneshows a lifted loop limit).Falseor an exception raisesDLQOperationDeniedErrorbefore anything is read, and aDLQ replay DENIEDrecord goes to the audit logger.purge(queue, actor=...)is audited too.- The CLI's
--require-actor(orRABBITKIT_REPLAY_REQUIRE_ACTOR=1in a shared ops environment) refuses, audited, a replay that does not name its--actor.
from rabbitkit import DLQInspector, ReplayRequest
ALLOWED = {"alice", "ops-bot"}
def only_named_operators(request: ReplayRequest) -> bool:
return request.actor_explicit and request.actor in ALLOWED and request.operation == "replay"
def make_inspector(transport: object) -> DLQInspector:
return DLQInspector(transport, authorizer=only_named_operators)
To replay only a subset:
Replay uses publisher confirms and mandatory=True, and acks each DLQ
original only after its republish is confirmed. A message whose republish
fails is left on the DLQ (requeued once the fetch loop ends) and replay
continues with the rest, reporting the failure count (non-zero exit) at
the end — one poison target doesn't strand the remaining replayable messages.
A broker dead-lettered message's own routing key is the DLQ's name, so when
no origin can be resolved a message that would be published straight back
into the DLQ is skipped and left there (ReplayResult.skipped; the CLI
prints SKIPPED), instead of republishing it into the queue being drained,
which used to make replay(dlq) loop forever. Pass allow_self_replay=True
to opt out.
Replay copies every property the original carried (timestamp,
delivery_mode, expiration to the millisecond, priority, ...) and leaves
absent ones absent. One exception on the async transport: aiormq stamps a
random message_id on any publish that lacks one.
Quorum DLQs and delivery limits¶
AMQP 0-9-1 has no browse, so a peek is basic_get plus a requeue, and a
quorum queue counts every requeue as a delivery (closing the channel
counts too). Past the queue's delivery limit RabbitMQ dead-letters the
message, or drops it if the queue has no dead-letter exchange. rabbitkit's
DLQs have none. Measured on 3.13.7 and 4.1.8:
| RabbitMQ 3.x | RabbitMQ 4.x | |
|---|---|---|
| no limit configured | unlimited | 20 (the default) |
x-delivery-limit: -1 |
drops on the first return | unlimited |
| argument and policy both set | lowest wins | lowest non-negative wins |
So on 4.x, 20 peeks of a quorum DLQ deleted its messages.
What rabbitkit declares. A DLQ inherits quorum from a quorum source
queue (retry DLQs and the safety DLX alike). On 4.x a new quorum DLQ is
declared with x-delivery-limit: -1; on 3.x it gets no limit argument.
RabbitMQ refuses to add or remove that argument on an existing queue (406),
so a DLQ that already exists is not redeclared. Fix an existing 4.x quorum
DLQ with a policy, which needs no redeclare:
RabbitMQ applies only the highest-priority matching policy. If one already
matches the DLQ (for example a rabbitkit-<queue>-dlq policy rendered by
policy_templates(), priority 10), add "delivery-limit": -1 to it rather
than creating a second one that would be shadowed.
A route that consumes another route's quorum DLQ declares it the same way.
Note that a 4.x DLQ created by 0.19 carries the argument, so an older
rabbitkit that redeclares it without the argument gets a 406; set
SafetyConfig(on_topology_conflict="warn_continue") before rolling back.
In a mixed 3.x/4.x cluster mid-upgrade, the version is the node the
connection reached: let new quorum DLQs be created only once every node runs
4.x.
What the inspector checks. Give DLQInspector a management client and
peek / filtered replay read the queue's type, arguments and effective
policy first. They raise UnsafeToBrowseError for a quorum queue whose
limit isn't unlimited, and fail closed when they can't tell (unknown broker
version, or statistics not emitted yet):
from rabbitkit import DLQInspector, RabbitManagementClient
inspector = DLQInspector(transport, management=RabbitManagementClient())
Without a management client (or when the API answers 404, e.g. a wrong
vhost; vhost defaults to the transport's) the inspector stops as soon as a
fetched message carries x-delivery-count, a header only quorum queues set
(from a message's second delivery on), and refuses that queue from then on.
That caps the damage at about two deliveries per message. A replay()
without a predicate acks what it republishes, so it checks lazily, at the
first message it has to requeue (a failed or skipped one): a message that
keeps failing would otherwise lose a delivery every run.
check_delivery_limit=False turns both checks off.
The CLI does the same. rabbitkit dlq inspect and replay take
--management-url (or RABBITMQ_MANAGEMENT_URL), stop on
x-delivery-count without it, and exit 2 when they refuse.
Quorum dead-letter strategy and replica count¶
RabbitQueue(queue_type=QueueType.QUORUM, dead_letter_strategy="at-least-once",
overflow="reject-publish") makes the broker keep a dead-lettered message on
the source queue until the dead-letter target confirms it, instead of
dropping it when the DLQ is missing or full (RabbitMQ 3.10+;
at-least-once without overflow="reject-publish" is rejected at
construction, because RabbitMQ would silently fall back to at-most-once).
Verified on RabbitMQ 4.1: with the DLQ deleted, a terminally failed message
stays on the source (messages_dlx=1 in the management API); with
at-most-once it is gone. quorum_initial_group_size=3 sets
x-quorum-initial-group-size.
For the retry chain: RetryConfig(delay_queue_type="quorum",
delay_queue_dead_letter_strategy="at-least-once", quorum_initial_group_size=3)
applies the same to the delay queues (adding x-overflow=reject-publish) and
the replica count to a quorum DLQ. All default to None, so existing
topology is declared exactly as before.
Retry jitter: jitter_mode="sharded"¶
By default every message that fails at the same moment retries at the same moment — the delay-queue TTLs are exact, so a burst of correlated failures re-hammers the recovering dependency as a phase-locked wave. Per-message TTL jitter is NOT an option here: mixed TTLs in one classic queue reintroduce head-of-line blocking (a long-TTL message parks everything behind it).
RetryConfig(jitter_mode="sharded", jitter_shards=3, jitter_factor=0.1)
decorrelates the wave while keeping every queue's TTL uniform: each tier
becomes N sub-queues whose TTLs stagger across ±jitter_factor
(orders.retry.2 at 30s, .s1 at 27s, .s2 at 33s), and a message picks
its shard by a stable hash of its message_id — the same message keeps
the same cadence across redeliveries and processes. Shard 0 keeps the
legacy queue name and exact TTL, so enabling this on an existing topology
is purely additive: no 406s, no migration. The default "off" produces
byte-identical topology to previous releases.
jitter_factor has no effect with jitter_mode="off". This is deliberate:
the only way to jitter a single TTL delay queue is a per-message
expiration, and a classic queue expires messages only at its head, so one
long-TTL message would hold back every shorter one queued behind it.