Producer-Consumer Queues and Backpressure

The producer-consumer pattern puts a queue between code that creates work and code that does it: request handlers and a log writer, a crawler and its parsers, a web server and a pool of thumbnail workers. The queue lets each side run at its own pace and absorbs bursts. That part is easy. The part interviewers care about is what happens when producers are faster than consumers for longer than a burst, because then the queue grows, and every design has to answer the question “and then what?”

A slow consumer, three ways

To make it concrete, here’s one producer offering items as fast as it can (with a 1 ms pause, which on this Mac works out to about 650 items per second) and one consumer that takes 10 ms per item (about 75 per second in practice). Each item carries its creation time so the consumer can measure how long it sat in the queue. After two seconds the script records the backlog and shuts the queue down:

import queue, threading, time

def run(maxsize, policy, seconds=2.0):
    q = queue.Queue(maxsize=maxsize)
    offered = dropped = 0
    waits = []
    def producer():
        nonlocal offered, dropped
        while True:
            item = time.perf_counter()
            try:
                if policy == "drop":
                    q.put_nowait(item)
                else:
                    q.put(item)                   # blocks while the queue is full
            except queue.Full:
                dropped += 1
            except queue.ShutDown:
                return
            offered += 1
            time.sleep(0.001)                     # producer offers up to ~1000 items/s
    def consumer():
        while True:
            try:
                item = q.get()
            except queue.ShutDown:
                return
            waits.append(time.perf_counter() - item)
            time.sleep(0.01)                      # consumer handles ~100 items/s
    p = threading.Thread(target=producer, daemon=True); c = threading.Thread(target=consumer, daemon=True)
    p.start(); c.start()
    time.sleep(seconds)
    backlog = q.qsize()
    q.shutdown(immediate=True)                    # Python 3.13+: wakes blocked put() and get() callers
    p.join(2); c.join(2)
    print(f"maxsize={maxsize:<3} {policy:5}: offered {offered:4}, dropped {dropped:4}, consumed {len(waits):3}, "
          f"backlog {backlog:4}, worst wait {max(waits)*1000:5.0f} ms", flush=True)

run(0, "block")      # unbounded
run(50, "block")
run(50, "drop")

Python 3.14 on an M1:

maxsize=0   block: offered 1331, dropped    0, consumed 156, backlog 1175, worst wait  1767 ms
maxsize=50  block: offered  201, dropped    0, consumed 151, backlog   50, worst wait   707 ms
maxsize=50  drop : offered 1312, dropped 1108, consumed 154, backlog   50, worst wait   670 ms

The consumer processed about 150 items in every case. Nothing about the queue made it faster. What changed is where the excess went.

The unbounded queue only postpones the problem

maxsize=0 means no limit. The producer never waited, so it looked healthy, and after two seconds 1,175 items were waiting. The worst wait was 1.8 seconds and climbing, because every new item joins the back of a line that’s growing by roughly 575 items a second. Leave it for an hour and the backlog is two million items, each waiting about eight hours, and the process has probably been killed for running out of memory long before that.

This is the most common production failure of the pattern, and it’s sneaky because nothing errors. Latency drifts up, memory climbs, and the first visible symptom is the out-of-memory kill or a timeout far downstream. Defaulting to unbounded queues (in Java, new LinkedBlockingQueue<>() with no capacity; in many thread pool constructors, the default work queue) is a frequent interview red flag when the candidate can’t say what bounds it.

A bound turns overload into a decision

With maxsize=50, the queue fills in well under a second, and then the producer has to do something with each new item. The two runs show the two basic answers.

Block the producer. This is backpressure: the slow stage pushes back on the fast one. The producer spent most of its time inside put(), its throughput fell to match the consumer (201 offered, 151 consumed, 50 waiting), and the worst wait stopped growing at about 0.7 seconds. If the producer is itself fed by something upstream, the slowdown propagates: a blocked request handler stops reading its socket, TCP’s window fills, and the client slows down. That’s backpressure working as intended, all the way to the edge.

Drop. put_nowait() raised queue.Full 1,108 times, and those items were discarded. The producer kept its pace, and memory and latency stayed bounded. Dropping suits data where fresher is better or loss is tolerable: metrics samples, UI progress updates, debug logs. You can drop the newest item (what this code does), drop the oldest to make room, or keep a count of drops so the loss is visible.

A third answer is reject: tell the caller “busy, try later” with an HTTP 429 or 503, or a gRPC RESOURCE_EXHAUSTED. It’s dropping with the decision pushed back to the client, which can retry with backoff or go elsewhere. Load balancers and API gateways do this so a server overloaded today doesn’t queue requests whose callers gave up long ago.

Which one is right depends on whether the work can wait, whether it can be lost, and whether the producer can afford to stall. Blocking a request thread so it can write a debug log is usually wrong. Dropping payment events is always wrong.

Sizing the bound

Little’s law ties the numbers together: the average number of items in a stable queue equals the arrival rate times the average time each spends there. Turn it round and a full queue of capacity C, drained at rate μ, means a new item waits about C / μ. Here that’s 50 / 75 per second ≈ 0.67 seconds, very close to the measured 0.67 to 0.71.

So size the queue from latency, not memory. If your consumer handles 75 items a second and an item older than 200 ms is useless to the user, a capacity of about 15 is the most that makes sense. A capacity of 10,000 would just let the system hold onto two minutes of stale work during overload. The bound should cover expected bursts; sustained overload needs more consumers or less work, not a bigger buffer. Networking people call the result of oversized buffers bufferbloat.

Building and stopping it correctly

Under the hood, a blocking bounded queue is a mutex plus two condition variables: producers wait on “not full”, consumers wait on “not empty”, and each side notifies the other after changing the count. Python’s queue.Queue, Java’s ArrayBlockingQueue and Go’s buffered channels all give you this, so you rarely build it yourself. The classic textbook version uses two counting semaphores (empty slots and filled slots) plus a mutex instead.

Shutdown deserves as much thought as the happy path. If the consumer exits while the producer is blocked in put() on a full queue, nothing will ever make room, and the producer hangs forever. An earlier draft of the experiment above did exactly that: the consumer returned on a timer, the producer sat in put(), and the script never exited. There are two standard fixes:

• Sentinels (poison pills). The producer puts one special value per consumer, such as None, after its last real item. Each consumer exits when it reads one. Everything queued before the sentinel still gets processed.
• Queue.shutdown() (Python 3.13 and later). Further put() calls raise ShutDown, and so do any already blocked. With the default immediate=False, consumers drain what’s left and then get ShutDown; with immediate=True the remaining items are discarded and every waiter wakes at once, which is what the experiment uses.

Go’s equivalent is closing the channel, after which receivers drain the buffer and then see the zero value with ok == false.

Interview exercise: sizing a log-shipping queue

Your application threads produce log lines in bursts of up to 50,000 lines per second lasting about 10 seconds, with a normal rate of 2,000 per second. A background shipper sends at most 20,000 lines per second to a remote collector. Design the in-process queue between them.

Answer

Start with the rates. During a burst the excess is 30,000 lines per second for 10 seconds, so absorbing a whole burst without loss needs about 300,000 slots. At a few hundred bytes per line that’s on the order of 100 MB, which may or may not be acceptable. After the burst the shipper drains the backlog at 18,000 lines per second spare capacity, so it clears in under 20 seconds, and the worst line waits around 15 seconds. For logs that’s fine.

The policy when full matters more than the size. Request threads must never block on logging, because then a slow collector becomes an application outage. So the queue should be bounded and the producer side should never wait: put_nowait, and on Full, drop the line and increment a counter that is itself reported, so operators can see that logs were lost. Two refinements are worth proposing. Keep error-level lines in a small separate queue, or let them evict debug lines, so the important ones survive a flood. And if no loss is acceptable, spill to a local file when the memory queue is full and ship from disk later, which trades loss for disk I/O.

Finally, expose queue depth, drop count and shipper latency as metrics, and on shutdown give the shipper a short deadline to flush before exiting.

Related reading

• Operating Systems interview questions
• Operating Systems MCQs
• Condition Variables and Predicate-Based Waiting
• Operating Systems: Thread Pools
• Python queue module documentation, including Queue.shutdown() and the ShutDown exception