Flow control and resource alarms¶
Alerts (all client side, from rabbitkit's own metrics, source: rabbitkit):
RabbitMQDiskAlarm—rabbitkit_connection_blocked_reason{reason="disk"}: the broker blocked the service's connections with "low on disk".RabbitMQMemoryAlarm—rabbitkit_connection_blocked_reason{reason="memory"}: "low on memory".RabbitMQConnectionBlocked—rabbitkit_connection_blocked, for any other reason (or a client that cannot read it). It does not fire alongside the disk or memory alert.
Each fires when the connection was blocked for most of the last 2 minutes,
for 1 minute. The rabbitmq-broker-plugin group has broker-side
RabbitMQDiskAlarm / RabbitMQMemoryAlarm (source: rabbitmq_prometheus,
from rabbitmq_alarms_free_disk_space_watermark /
rabbitmq_alarms_memory_used_watermark) for clusters that run the
rabbitmq_prometheus plugin: they see an alarm before any client publishes.
What it means¶
When a node crosses its memory high watermark or its free-disk limit,
RabbitMQ raises an alarm and blocks every publishing connection in the
cluster (connection.blocked, with the reason "low on disk" or "low on
memory"). rabbitkit sets connection_blocked to 1 and
connection_blocked_reason to 1 for the named alarm until
connection.unblocked; transport.blocked_reason has the broker's text.
RabbitMQ blocks a connection when it next publishes, so a service that is
not publishing does not see the alarm.
While blocked: publishes wait and then time out (TIMEOUT / NOT_SENT with
an opt-in FlowController), consumers keep consuming (which is what clears a
memory alarm). With ConnectionConfig.blocked_connection_timeout set (60 s
by default), a publisher connection blocked longer than that is closed and
reconnected; the gauges briefly read 0 after each such reconnect, which is
why the alerts average over 2 minutes.
Triage¶
- Which alarm: the alert name (or
rabbitkit_connection_blocked_reason), thenrabbitmq-diagnostics alarms,rabbitmq-diagnostics -q memory_breakdown,dfon the node's data directory. - Which queues hold the memory or disk: large backlogs (consumer backlog), DLQs nobody drains (DLQ growth), classic queues with large message bodies.
Mitigation¶
- Let consumers drain; stop or slow the producers that filled the queues.
- Free disk / add capacity; raise the watermark only with a capacity plan.
- Never restart consumers to "unblock": they are what drains the backlog.