Publisher confirm failures and unknown outcomes¶
Alerts:
PublisherConfirmFailures—messages_published_total{status=~"nacked|error"}above 0, sustained for 5 minutes.PublisherUnknownOutcomes— more than 1% of publishes endunknownortimeout, for 10 minutes (a reconnect or a quorum election alone stays well below that).PublisherConfirmLatencyHigh— the p99 ofpublish_confirm_latency_secondsabove 250 ms for 10 minutes (10× the measured quorum p99), with at least 0.1 confirms a second.
What it means¶
| Status | Meaning | Safe to publish again? |
|---|---|---|
nacked |
RabbitMQ refused the message (internal error, overflow reject-publish) |
yes |
error |
refused (exchange_not_found, access_refused, precondition_failed), invalid envelope, or unclassified |
after fixing the cause |
timeout |
no confirm within confirm_timeout / the per-call timeout |
only with the same message_id |
unknown |
the connection or channel was lost after the frame left | only with the same message_id |
reason on the same counter is the bounded PublishReason code. A TIMEOUT
or UNKNOWN message may be on the broker: republishing without the same
message_id can duplicate it; not republishing can lose it.
Triage¶
sum by (exchange, status, reason) (rate(rabbitkit_messages_published_total[5m])).- Confirm latency:
histogram_quantile(0.99, sum by (le, exchange) (rate(rabbitkit_publish_confirm_latency_seconds_bucket[5m]))). It is timed from handing the frame to the client library to the broker's answer, so a rise is the broker (disk fsync on quorum/durable queues, overload) or the network, not rabbitkit's admission. - Correlate with
rabbitkit_connection_blocked,rabbitkit_connection_failures_totaland the broker's queue length limits (x-overflow: reject-publishnacks). nackedwithreject-publish: the target queue is full; see consumer backlog.
Mitigation¶
- Make sure producers republish
UNKNOWN/TIMEOUToutcomes with the samemessage_idand consumers deduplicate (seedocs/production/idempotency.md). - Raise
PublisherConfig.confirm_timeoutonly if measured confirm latency legitimately exceeds it. - Persistent
errorwithaccess_refused: fix the user's permissions;precondition_failed: the publish properties conflict with the queue.