Consumer failures¶
Alerts:
ConsumerProcessingFailures— more than 5% of handler runs raise.ConsumerNotConsuming— a route's consumer is not subscribed while the broker is connected.
What it means¶
rabbitkit_messages_consumed_total{status="error"} counts handler runs that
raised. rabbitkit_messages_failed_total{queue,category} says why, using the
failure taxonomy:
| category | meaning | what rabbitkit does |
|---|---|---|
retryable |
transient (timeouts, connection errors) | retried |
infrastructure |
a dependency or the AMQP client is unavailable | retried |
unknown_outcome |
the processing deadline expired; side effects may have happened | retried (handlers must be idempotent) |
permanent |
a retry cannot succeed (bug, business rule) | dead-lettered |
invalid_message |
undecodable or invalid message | dead-lettered |
shutdown |
cancelled by shutdown | requeued |
ConsumerNotConsuming: rabbitkit_route_consuming == 0. Usually RabbitMQ
cancelled the consumer (basic.cancel: queue deleted, quorum leader moved,
operator action); rabbitkit_consumer_cancelled_total{queue} counts those.
Both brokers re-subscribe once the queue exists again; they never re-create
the queue.
Triage¶
sum by (queue, category) (rate(rabbitkit_messages_failed_total[5m])).- Logs:
Handler failed ... category=... reason=...(no payloads). permanent/invalid_messagespike after a deploy: a producer or a consumer changed the message contract.infrastructure/retryable: a downstream dependency is failing.- Not consuming: does the queue exist (
rabbitmqctl list_queues name)? Was it deleted, or is the quorum queue without a leader (quorum unavailable)?
Mitigation¶
- Roll back a bad deploy; dead-lettered messages can be replayed after the fix (DLQ growth).
- Dependency outage: retries are bounded by
max_retries; consider pausing the consumer instead of burning retries. - Deleted queue: re-create it (re-deploy the owner); the consumer resumes on its own.