Putting a queue between an API and the work it triggers is one of the most useful moves in system design. The API answers in milliseconds, workers process at their own pace, and a burst of requests turns into a backlog instead of an outage. It’s also one of the most misunderstood. A queue doesn’t create processing capacity. It moves the waiting from the request into the queue, and that waiting needs limits, an owner and an alert, or the queue becomes a place where work quietly goes to age.
What a queue actually buys you
Take an export service. Users request CSV exports of their transactions; building one takes a few seconds of database and CPU time. Without a queue, the API builds the export inside the request, and a month-end rush of requests saturates the database and times out everything else.
With a queue, the API writes an export record, enqueues a message and returns 202 Accepted (see API contracts and client recovery for that side). A pool of workers pulls messages and builds files. The queue gives you three things:
- Burst absorption. Short spikes above worker capacity become a temporary backlog.
- Isolation. The API stays fast even when workers are slow or down.
- Retry. If a worker crashes mid-export, the message becomes visible again and another worker picks it up.
What it doesn’t give you is throughput. If requests arrive faster than workers finish them for long enough, the backlog grows until something breaks: the queue’s retention limit, users’ patience, or the data’s usefulness.
A month-end burst, minute by minute
Model the export service with workers that finish 1,000 exports a minute. Normal traffic is 400 a minute. On the first working day of the month, from 09:00 to 11:00, traffic jumps to 1,200 a minute. A short Python simulation tracks the queue:
for minute in range(8 * 60, 13 * 60):
arrivals = 1200 if 9 * 60 <= minute < 11 * 60 else 400
q.extend([minute] * arrivals)
for _ in range(min(CAP, len(q))): q.popleft()
oldest = minute - q[0] if q else 008:00 arriving 400/min depth 0 oldest waiting 0 min
08:30 arriving 400/min depth 0 oldest waiting 0 min
09:00 arriving 1200/min depth 200 oldest waiting 0 min
09:30 arriving 1200/min depth 6,200 oldest waiting 5 min
10:00 arriving 1200/min depth 12,200 oldest waiting 10 min
10:30 arriving 1200/min depth 18,200 oldest waiting 15 min
10:59 arriving 1200/min depth 24,000 oldest waiting 19 min
11:00 arriving 400/min depth 23,400 oldest waiting 20 min
11:30 arriving 400/min depth 5,400 oldest waiting 13 min
11:39 arriving 400/min depth 0 oldest waiting 0 min
11:40 arriving 400/min depth 0 oldest waiting 0 min
12:00 arriving 400/min depth 0 oldest waiting 0 min
12:30 arriving 400/min depth 0 oldest waiting 0 minThe arithmetic behind it is short enough to do in an interview. During the burst, the backlog grows at arrivals minus capacity: 1,200 − 1,000 = 200 a minute, so two hours adds 24,000. After the burst, it drains at capacity minus arrivals: 1,000 − 400 = 600 a minute, so 24,000 takes 40 minutes to clear, which is exactly when depth hits zero at 11:39.
Little’s law turns depth into waiting time: average time in the queue equals the number waiting divided by the throughput. 24,000 waiting at 1,000 a minute means a new export waits about 24 minutes before a worker touches it.
Two lessons come out of this table. First, the burst is fine if a 20-minute wait at month end is acceptable, and that’s a product question, not a technical one. Second, the spare capacity during normal hours (600 a minute) decides how fast you recover. A system running its workers at 95% of capacity every day has almost no drain rate, and any burst leaves a backlog that lasts for hours.
Depth versus age
Most dashboards graph queue depth. Users experience age: how long their export has been waiting. They’re related but not interchangeable. 24,000 messages is a 24-minute wait for exports and a 2-second wait for a queue whose workers handle 10,000 messages a second. Age is also what catches a stuck consumer, where depth may look modest but nothing has moved in an hour.
So alert on the age of the oldest message against a target derived from the product promise. If exports are promised within 30 minutes, page when the oldest message is 20 minutes old and still rising. SQS exposes this as ApproximateAgeOfOldestMessage; Kafka’s equivalent is consumer lag, best converted to time rather than an offset count.
Poison messages and the dead-letter queue
Some messages fail every time: an export for an account that was deleted, a malformed payload, a date range the builder can’t handle. With SQS-style queues, a failed message reappears after its visibility timeout and gets retried. Without a limit, it’s retried forever.
That sounds harmless until you do the arithmetic. Suppose 1% of exports are poison and each reappears every 5 minutes. A simulation over 24 hours:
poison messages, no limit, retried forever : 1,130 failing attempts/min after 24 h (113% of worker capacity)
poison messages, maxReceiveCount 5, then DLQ : 20 failing attempts/min after 24 h (2% of worker capacity)Poison messages accumulate: four new ones a minute, none ever leaving. After a day there are about 5,700 of them, each attempted every five minutes, which is more failing attempts than the entire fleet can process. The queue fills with work that will never succeed, and real exports starve. With a receive limit of 5 (SQS calls it maxReceiveCount in the redrive policy), each poison message gets five tries and moves to a dead-letter queue, and the cost stays at 2% of capacity.
A dead-letter queue needs an owner. Alert when anything lands in it, record why each message failed, and have a way to redrive messages back to the main queue once the bug is fixed. A DLQ nobody looks at is just a slower way to lose work.
Also separate permanent failures from temporary ones. A deleted account will never succeed and can go straight to the DLQ (or be marked failed on the export record). A database timeout might succeed on the next try, ideally after a growing delay.
Consumers must be idempotent
Most queues deliver at least once. A worker that builds an export, uploads the file, and crashes before acknowledging the message will see that message again, and so will a worker whose processing outlives the visibility timeout. Design consumers so a repeat is harmless: check the export record’s status before starting, write the output to a path derived from the export id so a rerun overwrites rather than duplicates, and make the final status update conditional (UPDATE exports SET status = 'succeeded' WHERE id = $1 AND status = 'running'). Set the visibility timeout comfortably above the worst-case processing time, or extend it from the worker while it runs.
Interview practice: the growing backlog
“A video transcoding queue normally sits near zero. Since a marketing campaign started three days ago, its depth has grown every day and the oldest message is now nine hours old. Workers are healthy and busy. What do you look at and what do you do?”
Answer
Healthy, busy workers with a backlog that grows every day means arrival rate is above processing capacity on average, not just during spikes. A queue can only absorb a temporary excess, so this won’t fix itself. First I’d confirm the numbers: arrivals per hour versus completions per hour over the last few days, and whether the mix changed. The campaign might bring longer videos, so each job takes more worker time even if the count is similar.
Then I’d act on both sides of the equation. On capacity: scale workers, and if scaling is limited, find what bounds throughput (CPU, a shared encoder licence, the storage write rate). On arrivals: prioritise. Separate short videos from long ones, or paying customers from free, into different queues with their own workers, so a long backlog of one kind doesn’t delay everything. For the existing nine-hour backlog, compute drain time from spare capacity after scaling, and tell users their realistic wait. I’d also check the DLQ and retry counts to make sure some of that busy time isn’t poison messages being retried, then add an alert on oldest-message age so the next slow build-up is caught in hours, not days.