Retry storm¶
Alert: RetryStormDetected — more than half of the handler runs on a
queue are retried, or rabbitkit_retry_handoff_paused == 1.
What it means¶
- A high retry ratio means most messages fail transiently: a dependency is down or overloaded. Retries add load to it.
retry_handoff_paused == 1: the retry publish to the delay queue fails (rabbitkit_retry_handoff_failures_total), so the handoff backs off and messages are requeued on the source queue instead of moving forward.
Triage¶
sum by (queue, category) (rate(rabbitkit_messages_retried_total[5m])):infrastructure/retryable/unknown_outcome.- Handoff failures:
rate(rabbitkit_retry_handoff_failures_total[5m])andrabbitkit_publish_routing_failures_totalfor the retry exchanges; check the delay queues exist (rabbitmqctl list_queues name | grep retry). - The retry delay queues' depth (
rabbitkit_queue_messages_ready).
Mitigation¶
- Fix or shed load from the failing dependency; consider pausing consumers.
- Retry topology missing: re-deploy the owning service so it re-declares it.
- Do not raise
max_retriesduring a storm: it multiplies the load.