asyncio backpressure: bound the queue before it eats your memory
Contents
Contents
TL;DR
- An unbounded queue lets a fast producer run ahead of its consumers forever. In the benchmark (2,500 requests at 500/s, 1.5× the capacity of 4 workers doing ~12 ms of work each), the naive queue peaks at 917 deep and 1.27 MB, with an end-to-end p99 of 2.9 s.
- Bounding the queue with backpressure (blocking
put) fixes the memory, not the latency. The server-side p99 drops to ~236 ms, but callers still wait ~2.9 s end-to-end at p99, because the wait moved upstream into the producer. That is coordinated omission. - Load shedding (rejecting the overflow) is the only option that keeps the caller’s wait low: 34% of requests were refused, and the accepted ones answered with a 233 ms end-to-end p99. Measure latency from the intended send time, not from when the request reached the queue.
You bound the queue, the dashboard shows a 236 ms p99, and callers still wait almost 3 s. The metric starts its clock at the queue and can’t see the time spent waiting to get in. This post breaks an asyncio service on purpose, measures both latencies, and shows which fix lowers the one your callers feel.
The numbers come from a small benchmark you can run yourself, and they will differ on your machine. The shape of the result won’t.
The problem, and where it shows up
An asyncio service usually has a fast side and a slow side. A web handler accepts a request and hands it to a queue. A scraper finds URLs and hands them to a fetcher. An ingestion loop reads messages and hands them to a processor. The handoff is an asyncio.Queue.
The slow side is slower on purpose. It waits on I/O: a database, an upstream API, a file. If the fast side produces faster than the slow side consumes, the difference has to go somewhere. With an unbounded queue, it goes into memory, and the time each request waits in line goes up.
queue: asyncio.Queue = asyncio.Queue() # no maxsize -> unbounded
async def producer() -> None:
for i in range(TOTAL):
await queue.put(make_request(i)) # never blocks
async def worker() -> None:
while True:
req = await queue.get()
await handle(req) # slow-ish work
queue.task_done()
put on an unbounded queue always succeeds immediately. It never awaits a slot, so the producer never feels the consumers lagging. The producer just keeps appending.
The sibling anti-pattern is fire-and-forget tasks: asyncio.create_task(handle(req)) on every request, with no cap. The queue item becomes a task, but the failure is the same. Under load the number of live tasks grows without limit.
You won’t see this in the happy path. It needs a producer that outruns the consumers, and that only happens under load: a retry storm, a batch job that starts dumping, a thundering herd after a cache flush.
The naive version, and how to make it fail on demand
The benchmark is bench.py, in this post’s bundle. Standard library only, Python 3.12+:
uv run --python 3.12 bench.py
It runs three versions of the same service under the same load and reports latency percentiles, peak queue depth, peak memory, and dropped requests. The load is fixed and open-loop:
- 2,500 requests, intended at a fixed rate of 500/s, 1.5× the service’s capacity
- 4 worker tasks, each doing ~12 ms of simulated I/O (
await asyncio.sleep, jittered 6 to 18 ms by a seeded RNG), so capacity is4 / 12 ms ≈ 333/s - each request carries a 1 KB body, so memory shows up in the numbers
seed=0, so reruns reproduce the same sequence
“Open-loop” means the producer fires on a fixed schedule regardless of what the service is doing. Real clients do this: a load balancer probing on a timer, a batch job streaming records, an external system that doesn’t know or care how busy you are. The alternative, a closed-loop client that waits for a response before sending the next request, throttles itself against your slowness and hides exactly the failure we want to see.
The naive version is the snippet above: an unbounded queue, workers that drain it, a producer that never blocks.
What the failure looks like in numbers
On my laptop (Fedora Linux 44, Intel Core i7-1065G7, Python 3.13.15), one run prints:
python 3.13.15 seed=0
requests=2,500 workers=4 work~12ms payload=1024B queue_bound=64
intended rate=500/s (capacity~333/s) -> 1.5x overload
naive
accepted 2,500
completed 2,500
dropped 0
p50 service 1442.12 ms
p99 service 2923.72 ms
p50 end-to-end 1442.92 ms
p99 end-to-end 2924.60 ms
peak depth 917
peak mem 1.27 MB (tracemalloc)
wall time 7.95 s
backpressure
accepted 2,500
completed 2,500
dropped 0
p50 service 214.75 ms
p99 service 236.47 ms
p50 end-to-end 1488.25 ms
p99 end-to-end 2942.30 ms
peak depth 64
peak mem 0.25 MB (tracemalloc)
wall time 7.97 s
shed
accepted 1,640
completed 1,640
dropped 860
p50 service 211.44 ms
p99 service 232.44 ms
p50 end-to-end 212.08 ms
p99 end-to-end 232.94 ms
peak depth 64
peak mem 0.20 MB (tracemalloc)
wall time 5.21 s
scenario acc drop p50 srv p99 srv p50 e2e p99 e2e depth MB wall
-------------------------------------------------------------------------------------
naive 2,500 0 1442.12 2923.72 1442.92 2924.60 917 1.27 7.95
backpressure 2,500 0 214.75 236.47 1488.25 2942.30 64 0.25 7.97
shed 1,640 860 211.44 232.44 212.08 232.94 64 0.20 5.21
Across three runs the numbers moved by under 3%. Every figure below comes from the run above.
The naive queue accepted everything and dropped nothing, and paid for it. The queue peaked at 917 requests and 1.27 MB of Python allocations, and the end-to-end p99 was 2.9 s for ~12 ms of actual work. The queueing is the entire cost.
Coordinated omission: why there are two latency columns
Each request gets two timestamps. end-to-end is completion minus the intended send time. service is completion minus when the request actually reached the queue.
The two columns are the point of the benchmark, because they disagree in the backpressure row. Its service p99 is 236 ms, twelve times better than the naive queue’s 2.9 s. Its end-to-end p99 is 2.9 s, the same as naive’s. Nothing got faster. The wait moved upstream into the producer, and a metric that starts the clock at the queue can’t see it.
Gil Tene calls this coordinated omission in How NOT to Measure Latency (2015). A load generator that only records responses it actually sent omits the time a request spent waiting to be sent. The busier the service, the more requests wait to be sent, and the more of the real latency the metric silently drops. The result looks flat while callers time out.
In the benchmark, end-to-end measures from the intended time so the wait to be sent can’t disappear. service is what a server-side histogram would report. For the naive queue the two agree, because the unbounded put never delays the producer. For backpressure they diverge, because the bounded put does.
The fix, and the event loop mechanics that make it work
The fix has two parts. Bound the queue, and decide what happens when it’s full.
Bound it
queue: asyncio.Queue = asyncio.Queue(maxsize=64)
Now the queue can’t hold more than 64 requests. That fixes the memory. It does not, by itself, fix the latency, and the two policies below show the difference.
Backpressure: block the producer
async def producer() -> None:
for i in range(TOTAL):
await queue.put(make_request(i)) # blocks while the queue is full
With a bounded queue, await queue.put on a full queue suspends the producer until a worker frees a slot. The producer can’t run ahead, so the queue stays at 64 and memory stays at 0.25 MB.
Look at the backpressure row again: memory fixed, nothing dropped, but end-to-end p99 still 2.9 s. Backpressure protects the process. It does not make callers wait less when you’re over capacity, because the caller is now the one waiting to send. The ~12 ms of work per request is unchanged, and the queue still can’t serve faster than 333/s.
Load shedding: reject the overflow
dropped = [0] # a mutable counter, as in bench.py (a bare `dropped += 1` would raise UnboundLocalError)
async def producer() -> None:
for i in range(TOTAL):
try:
queue.put_nowait(make_request(i))
except asyncio.QueueFull:
dropped[0] += 1 # tell the caller it was refused
put_nowait never blocks. On a full queue it raises asyncio.QueueFull, and the producer refuses the request instead of queueing it.
This is the only row where the end-to-end latency stays low: 233 ms p99, with 34.4% of requests refused. The arithmetic predicts that number. Capacity is 333/s and the intended rate is 500/s, so the excess is 1 - 333/500 ≈ 33%. The measured 860 drops out of 2,500 is 34.4%, right next to it.
Shedding doesn’t speed up the work. It keeps the queue short by turning the overflow into immediate rejections, so the requests it accepts don’t wait behind a backlog.
A semaphore for fire-and-forget
If your design is fire-and-forget tasks rather than a queue, a semaphore is the same bound:
sem = asyncio.Semaphore(10) # at most 10 tasks in flight
async def on_request(req) -> None:
async with sem:
await handle(req)
async with sem waits when the counter hits zero and resumes when a task calls release at the end of the block. That’s the same mechanism as a bounded queue, and it has the same caveat: the async with blocks, so it’s backpressure, not shedding. To shed with a semaphore you’d check sem.locked() and reject before acquiring.
Why the bound works, at the event loop level
The event loop runs one coroutine at a time. A coroutine gives up the CPU only at an await, and it resumes only when whatever it awaited is ready. Backpressure is just a coroutine awaiting something that a consumer unblocks.
An unbounded queue.put has nothing to await. When the queue has no maxsize, put calls put_nowait and returns immediately. The producer coroutine never suspends, so the consumers’ slowness has no channel back to it. The only place the imbalance can accumulate is the queue, and the queue has no ceiling.
A bounded queue wraps its fullness in a condition. When the queue is full, put awaits that condition, and the producer coroutine suspends. When a worker calls get and frees a slot, it wakes the waiter, and the producer resumes. The worker’s completion is now the event that lets the producer continue. That is the entire mechanism: give the producer an await that the consumer controls.
put_nowait is the same idea, minus the wait. On a full queue it raises QueueFull instead of suspending, and the producer turns that exception into a rejection.
The practical rule that falls out of this: in asyncio, the only way to slow a producer down is to make it await something a consumer unblocks. A queue with a bound gives you that. An unbounded queue, a bare create_task, and an unthrottled loop give you nothing.
Trade-offs, and when not to bound
- Backpressure protects the process; shedding protects the caller. Blocking keeps every request but moves the wait upstream, so the end-to-end latency is unchanged. Shedding refuses the overflow so the accepted requests stay fast. Pick per endpoint: a health check or a search can shed; a payment or a write probably should not.
- Shedding only works if the client can cope. A
QueueFullbecomes a 503 or a retryable error, and the client must be able to retry or move on. If it can’t, you need backpressure, a durable queue, or both. - Don’t bound a queue that never fills. If the producer is genuinely slower than the consumers, the queue stays near empty and the bound is dead weight. The catch is that “genuinely slower” is the assumption that bursts and traffic spikes break. The bound is cheap insurance against the overload you didn’t predict.
- Pick the bound from your targets, not a guess. A queue is a buffer, and a steady-state queue holds
L = λ × Witems (arrival rate times wait time). The bound is also the maximum queueing delay: withBslots and capacityμ, a request can wait at mostB / μbefore it reaches the front. In the run above that’s64 / 333/s ≈ 190 ms, most of the 236 ms service p99. If you can accept at most 100 ms of queueing at a capacity of 1,000/s, size the queue around 100. The arrival rate doesn’t enter the formula. It only decides how often the queue is full. - A bound is not a timeout. A bounded queue caps how many requests wait, not how long any one of them waits. One slow request can still sit at the head and stall everyone behind it. Timeouts and bounds are separate, and you want both.
Gotchas
These are the ways a bounded queue still falls over, in roughly the order they bite.
Blocking calls freeze the loop.
time.sleep,requests.get, a sync database driver: any of these stops the event loop for everyone, including the workers that are supposed to drain the queue. The producer keeps filling, the workers stop running, and the bound doesn’t help because the consumers are frozen. This includes a sync library called inside a background task: it holds the loop for the whole call. Useawait asyncio.sleep, an async client, orasyncio.to_thread(orloop.run_in_executor) so the call parks on a thread instead of on the loop.CPU-bound work stalls too.
asynciois cooperative, not parallel. A tight CPU loop between awaits doesn’t release the loop, so it starves every other coroutine. If the work is CPU-bound, put it in aProcessPoolExecutor.to_threadhelps only for work that releases the GIL (C extensions, most I/O).One failing task cancels its siblings in a
TaskGroup.asyncio.TaskGroupowns its children: if one raises,__aexit__cancels the remaining tasks and re-raises as anExceptionGroup. It is always a group, even for a single failure, soexcept ValueErrorwon’t catch it. Useexcept*.async def main() -> None: async with asyncio.TaskGroup() as tg: for _ in range(4): tg.create_task(worker(queue)) # one bad request kills all four try: asyncio.run(main()) except* ValueError as eg: print(f"{len(eg.exceptions)} worker(s) failed:", eg.exceptions)So a single unhandled exception in one worker tears down the whole pool, and the queue stops draining. Catch per task inside the child, or decide deliberately that one failure should stop everything. When a sibling is cancelled, its
awaitraisesCancelledError, so don’t swallow it: anexcept CancelledErrororexcept BaseExceptionwithout araise, or areturninsidefinally, leaves the task looking healthy while it was cancelled. A plainfinallyre-raises on its own.Fire-and-forget tasks can be garbage collected.
asyncio.create_taskreturns a task the event loop only holds a weak reference to. Drop the reference and the task can be collected before it runs. Keep a strong reference (a set) or own the tasks in aTaskGroup.
Production checklist
- Every queue and semaphore is bounded. No
asyncio.Queue()without amaxsize. - The full-queue policy is explicit per endpoint: block or shed, and why. Block protects memory; shed protects latency.
- Shedding returns a clear, client-visible error (a 503, a retryable status) and a metric. Silent drops are the worst of both worlds.
- Timeouts on every external call. A bound doesn’t bound one slow item.
- No blocking or sync calls on the loop.
to_threadfor blocking I/O,ProcessPoolExecutorfor CPU-bound work. - Tasks are owned by a
TaskGroupor a kept reference, with per-task error handling. - Measure end-to-end latency from the intended send time, not just server-side service time. A server histogram alone will miss the wait that moved upstream.
- Meter queue depth, drops, and latency percentiles. Alert when depth approaches the bound, not when it’s already past it.
- Run the naive version under load first. You want to see the failure before you ship the fix, so you know what the fix is buying.
Run uv run --python 3.12 bench.py, read the three columns, then go find the unbounded queue in your own code.