DLQ growth¶
Alert: DeadLetterQueueGrowing — rabbitkit_messages_dead_lettered_total
rising, or a *.dlq queue's ready count growing.
What it means¶
Messages failed terminally (permanent, invalid_message, retries
exhausted, retry age exceeded) and were copied to the dead-letter queue
(<queue>.dlq, or rabbitkit.dlq in shared mode) with failure metadata
headers (x-rabbitkit-error-category, x-rabbitkit-failure-category, ...).
Nothing is lost, but every message needs a decision.
Triage¶
sum by (queue, category) (increase(rabbitkit_messages_dead_lettered_total[1h])).- Inspect without consuming:
rabbitkit dlq inspect <queue>.dlq(orDLQInspector), group byx-rabbitkit-error-categoryand error code. - One category dominating after a deploy: a contract or code regression.
Mitigation¶
- Fix the cause, then replay:
rabbitkit dlq replay <queue>.dlq --actor <you>(replays are audited;--require-actorenforces a named actor). - Messages that can never succeed: export them for the data owner, then purge with an explicit actor.
- Never replay before fixing the cause: it only refills the DLQ.