Benchmarking methodology¶
rabbitkit ships two benchmark tiers with different jobs. Knowing which question each can answer — and which it cannot — matters more than any single number.
Tier 1: the classic suite (python -m benchmarks)¶
Single-pass smoke benchmarks: throughput drain, closed-loop latency, failure-path overhead, lifecycle timings, resource tracking. Runs in CI's best-effort step and the nightly workflow.
What it is for: catching order-of-magnitude regressions ("consume got 10x slower") and verifying every path works under load.
What it is NOT for: detecting small regressions or comparing serializers. Its payloads are ~9 bytes, so every scenario is AMQP round-trip-bound — raw / JSON / Pydantic / msgspec all measure the same thing (which is why their numbers agree). Its numbers are single runs on shared CI vCPUs; a ±10% swing between runs is noise, not signal.
Tier 2: the advanced suite (python -m benchmarks.advanced)¶
Repeated, statistically-reported measurements designed for real
performance questions. ~10 minutes; runs nightly and on demand
(--quick for a 2-rep functional pass).
Overhead A/B (bench_overhead)¶
The toolkit's headline cost: identical work through bare aio-pika and
through rabbitkit's full pipeline (middleware chain, DI, AUTO ack),
interleaved raw/kit/raw/kit so runner drift biases both sides
equally, 5 reps, reported as median ± CV with an instability flag when
CV exceeds 5%.
Dimension sweeps (bench_matrix)¶
- Payload sizes 100 B / 4 KB / 64 KB — shows where the workload flips from per-message overhead-bound to bandwidth-bound (msg/s falls while MB/s climbs).
- Classic vs quorum queues — the production checklist mandates quorum; this measures what that choice costs on the consume path.
Open-loop paced latency (bench_matrix)¶
The classic latency bench is closed-loop: the publisher paces itself by
its own progress on the same event loop as the consumer, which hides
queueing delay (coordinated omission). The paced bench publishes on an
absolute schedule (t0 + i/rate) and measures latency from the
intended send time, so scheduler lag and queue buildup are charged to
the system under test. It also reports the publisher's own schedule lag
(p99) — if the load generator couldn't hold the pace, the report says so
instead of publishing flattering numbers.
Tier 2b: bulk vs single (python -m benchmarks.bench_bulk)¶
Answers "what do publish_many / ack_many / CoalescingAcker actually buy
over one call per message?" — with the correctness accounting every
throughput number must carry (every input has a reported state; the queue is
verified drained through the management API).
| Scenario | What it does | Median msg/s | Wire frames / msg | Accounting |
|---|---|---|---|---|
| publish single / async | await broker.publish(e) one after another |
932 | 1 publish + 1 confirm | .ok per call |
| publish gather / async ×10 | hand-rolled asyncio.gather under a 10-slot semaphore |
4,685 | same | none (you write it) |
publish_many ×10 (default) |
default options → in-flight capped to the 10-channel pool | 4,548 | same | one BulkPublishItem per input |
publish_many ×64 |
PoolConfig(channel_pool_size=64), max_in_flight=64 |
8,951 | same | same |
publish_many + AsyncBatchPublisher |
batch config, in-flight 256 (pipelined confirms) | 9,862 | same | same |
| publish single / sync | SyncBroker.publish one after another |
833 | same | .ok per call |
publish_many / sync |
SyncBroker.publish_many (sequential by design) |
1,015 | same | one item per input |
| ack each | await msg.ack_async() per delivery |
15,308 | 1.000 | — |
ack_many ×100 |
park 100, one ack_many call |
11,808 | 1.000 (20 API calls) | one SettlementItem per delivery |
CoalescingAcker ×100 |
register/complete, in-order completion | 15,670 | 0.010 (20 frames for 2,000) | coordinator ledger |
Environment: Apple Silicon (arm64, 12 cores), Python 3.12.2, RabbitMQ 3.13
in Docker on the same machine, 2,000 messages × 1 KiB, persistent,
confirms on, mandatory=True, durable classic queue, median of 3 runs,
git ed2e8bc. Every scenario reported 2,000/2,000 CONFIRMED (or settled)
and the management API showed the queue drained to 0 ready / 0 unacked.
Absolute numbers are this machine's; the ratios are what travel.
How to read it
- Bulk publish is a concurrency story, not a wire story. RabbitMQ has no
multi-message publish;
publish_manysends one message per envelope and pipelines confirms. Sequentialpublishis RTT-bound (~0.9k/s). The same concurrency by hand (gather×10) gets ~4.7k/s, andpublish_many×10 matches it — the accounting layer costs nothing measurable. Raising the channel pool to 64 doubles it again; the batch publisher (confirms pipelined on shared channels) reaches ~9.9k/s, about 10× sequential. - Sync bulk is honestly not faster. pika cannot pipeline confirms, so
SyncBroker.publish_manyruns at the same ~1k/s as a loop. What you gain is the per-item outcome contract, not throughput. UseAsyncBroker(orSyncBatchPublisher) for volume. ack_manyis not a throughput feature. It still emits onemultiple=Falseframe per delivery — by design, so it can never settle a sibling that is still running — and it awaits them sequentially inside one call, which is why it measures slightly below per-handler acks that overlap with consumption. Its value is one validation pass, one report and one API call per batch commit (STALE/DUPLICATE/ALREADY_SETTLEDper item), i.e. correctness after a database transaction.- Coalescing is the wire optimisation.
CoalescingAckersettled 2,000 deliveries with 20 cumulative frames (0.01 frames/msg, 100× fewer) at the highest settle rate — because the coordinator knows every outstanding delivery and only acks through a completed prefix. Out-of-order completion lowers the ratio (stranded tags fall back to individual acks aftermax_holdrounds); in-order completion is the ceiling shown here. - Handler cost was zero in the ack scenarios, so those rates are a settlement-overhead ceiling, not an end-to-end consume rate.
# throwaway testcontainers broker (Docker required)
python -m benchmarks.bench_bulk --n 2000 --size 1024 --reps 3
# or an existing broker
python -m benchmarks.bench_bulk --url amqp://guest:guest@localhost:5672/ --mgmt-url http://localhost:15672
Results (with an environment fingerprint) land in
benchmarks/results/bulk_<timestamp>.json.
Tier 2c: the reliability profile (python -m benchmarks.bench_reliability)¶
Measures the workload a reliable deployment actually runs, rather than a
toy one. Every message is 1 KiB, persistent, published with confirms and
mandatory=True into a durable quorum queue (--queue-type classic
to compare), and consumed with manual acks after the handler. Publishers
and consumers run in separate processes. The end-to-end runs are open-loop:
latency is measured from each message's intended send time, so publisher
lag and broker queueing count against the system. Every run accounts for
every message: loss, duplicates and non-CONFIRMED publishes are reported,
never dropped from the numbers.
| Scenario | What it does |
|---|---|
publish |
publish_many, 256 in flight, 50,000 messages; confirm accounting |
consume |
drain a preloaded queue of 50,000 messages, prefetch 256 |
sustained |
open-loop at 2,778 msg/s for 60 s (about 10M messages an hour), 1 publisher and 1 consumer process |
burst |
open-loop at 20,000 msg/s for 10 s, 4 publisher and 2 consumer processes; peak backlog and drain time |
Results for 0.20.0¶
Environment: Apple Silicon (arm64, 12 cores), Python 3.12.2, aio-pika 10.2.1, RabbitMQ 3.13 in Docker on the same machine. The broker is a single node, so the quorum queue has a replication factor of 1 and no Raft network round trip. Other workloads were running on the host (load average 5–9). Single runs unless noted, so treat the numbers as indicative.
| Scenario | Result |
|---|---|
publish (quorum) |
7,535–7,751 msg/s (3 runs), 50,000/50,000 CONFIRMED, about 127 µs client CPU per message |
consume (quorum) |
11,664–12,148 msg/s (3 runs), about 74 µs client CPU per message |
sustained 2,778 msg/s × 60 s (quorum) |
Rate held. 166,680/166,680 CONFIRMED, 0 lost, 0 duplicated, peak backlog 0. End-to-end p50 1.9 ms, p95 9.3 ms, p99 27.5 ms, p99.9 94 ms. Peak RSS 65 MiB (publisher), 49 MiB (consumer) |
sustained 2,778 msg/s × 60 s (classic) |
Same accounting; end-to-end p50 1.1 ms, p95 5.6 ms, p99 12.5 ms, p99.9 32 ms |
burst 20,000 msg/s × 10 s (quorum) |
Publishers achieved 14,688–19,745 msg/s (3 runs). 200,000/200,000 CONFIRMED, 0 lost, 0 duplicated each time. The two consumers fell behind: peak backlog 72k–97k, drained 5.0–7.5 s after the last send, so end-to-end p50 was 4.9–6.1 s |
On a 3-node cluster (quorum replication factor 3)¶
Same host and versions; three RabbitMQ 3.13 nodes clustered on a Docker
network (--cluster-nodes 3), every quorum queue with 3 members online,
clients connected to one node. All three brokers and the clients share one
laptop (load average about 6.5), so the replication traffic competes with the
clients for CPU: this bounds the latency from above rather than predicting a
production cluster.
| Scenario | Result |
|---|---|
publish (quorum, 3 replicas) |
6,752–7,395 msg/s (3 runs), 50,000/50,000 CONFIRMED, about 131 µs client CPU per message |
consume (quorum, 3 replicas) |
9,906–10,780 msg/s (3 runs), about 80 µs client CPU per message |
sustained 2,778 msg/s × 60 s |
Rate held. 166,680/166,680 CONFIRMED, 0 lost, 0 duplicated, peak backlog 735. End-to-end p50 5.7 ms, p95 380 ms, p99 800 ms, p99.9 1.06 s; confirm p99 185 ms |
burst 20,000 msg/s × 10 s |
Publishers achieved 12,623–15,423 msg/s (3 runs). 200,000/200,000 CONFIRMED, 0 lost, 0 duplicated each time; peak backlog 105k–109k, drained 9.6–11.6 s after the last send |
What this shows, and what it doesn't:
- The 2,778 msg/s sustained target is met with headroom on one publisher and one consumer process, with exact accounting and no loss.
- 20,000 msg/s bursts are absorbed without loss, but the publish rate reached 74–99% of the target on this shared host, and the consumers need seconds to drain the backlog. Size consumers for the burst you need to drain, not just for the sustained rate.
- Replication keeps the guarantees but costs tail latency. On the
single node, quorum queues roughly doubled tail latency compared with
classic queues. With three replicas sharing one laptop, the sustained rate
still held with exact accounting and no loss, but p99 rose to 800 ms and
throughput fell by about 5–15%. On separate machines the CPU contention
goes away, but network round trips are added. Measure on your own cluster
before relying on these figures; the default
PublisherConfirmLatencyHighalert (confirm p99 above 250 ms) was not reached in these runs. - One earlier
sustainedrun on the same host had a 433 ms p99, which came from publisher schedule lag (all 256 in-flight slots waiting on slow confirms). It did not reproduce. A tail like that means the broker stalled confirms long enough to exhaust the in-flight window.
Cost of the 0.20.0 reliability work on the publish hot path¶
Client CPU per message, 0.19.2 against 0.20.0, both run under the same virtualenv (aio-pika 10.2.1), 20,000 × 1 KiB confirmed publishes per run, 6 interleaved rounds, median:
| Path | 0.19.2 | 0.20.0 | Change |
|---|---|---|---|
publish_many, 64 in flight (direct confirmed path) |
87.7 µs | 93.5 µs | +5.9 µs (+6.7%); throughput −4.0% |
publish_many + AsyncBatchPublisher |
95.3 µs | 102.2 µs | +6.9 µs (+7.2%); throughput −2.3% |
Before the hot-path fixes in this release these were +8 µs and +18 µs. The
remaining cost comes from features 0.20.0 adds to every publish: bounded
waits, publisher drain on stop(), return detection on every path, outcome
classification with reasons, connection-liveness checks and the metrics
hooks. There is no single hotspot left to remove. Moving from aio-pika 9.6
to 10.x costs about +7 µs/message on its own (0.19.2 code measured under
both), so compare benchmarks only across runs with the same aio-pika version.
Tier 3: the soak harness (python -m benchmarks.soak)¶
Sustained-runtime evidence for the two questions point-in-time tests cannot answer — the top risks for a long-running Kubernetes consumer:
- Connection recovery under sustained abuse. The broker container is
killed every
--restart-everyseconds for the entire run; the verdict requires the consumer to make progress within 60 s of every bounce, and every broker-confirmed publish to be consumed at least once by the end (duplicates are counted and reported — that's the at-least-once contract — but loss fails the run). - Leak detection. RSS, open file descriptors, and asyncio task count are sampled every 15 s. The verdict fits a least-squares slope to the post-warmup RSS series (fails above 256 KB/min sustained) and requires FD/task counts to stay flat across all the reconnect cycles — leaked channels, consumers, or timers show up here.
Runs weekly (.github/workflows/soak.yml, 30 min with a kill every 3
minutes ≈ 10 recovery cycles; dispatchable with custom duration) and
locally: python -m benchmarks.soak --duration 600 --restart-every 120.
The exit code is the verdict; the JSON report (with full sample series
and environment fingerprint) uploads as a workflow artifact.
Reading the numbers¶
- Prefer medians; CI runners have heavy right tails.
- A result with
⚠(CV > 5%) is noise-dominated — rerun or compare only against results from the same machine. - Every advanced result JSON (
benchmarks/results/advanced_*.json) embeds an environment fingerprint (python, platform, CPU count, git SHA, CI flag). Numbers without their machine are not comparable. - Absolute msg/s across different machines is meaningless; ratios (overhead %, quorum/classic, size-to-size) travel well.
Known limitations (deliberate, documented)¶
- No CPU pinning / turbo control — impossible on hosted runners; the interleaved A/B design compensates where it matters most.
- Steady-state producer/consumer equilibrium is not measured (drain bursts are a consume ceiling, labeled as such).
- No flamegraph capture; profile locally with
py-spy record -- python -m benchmarks.advanced --quick --url ...when a regression needs explaining.