A thread pool keeps a fixed number of worker threads and a queue of tasks, reusing threads instead of creating one per task. It bounds concurrency, which protects the system from overload and removes the cost of thread creation and teardown.
Before you start
You should understand threads and blocking I/O. This article covers pool sizing and queue behavior.
Step-by-step walkthrough
Step 1: Keep the pool bounded
A fixed pool of N threads limits simultaneous work to N, which caps resource use. Threads are reused across tasks, so the creation cost is paid once. The bound is the point: it prevents a spike from creating thousands of threads.
Step 2: Treat the queue as part of the design
If the queue is unbounded, tasks pile up until memory runs out, so the bound is defeated. Set a maximum queue size and a maximum wait, and reject when full so overload is visible instead of silent. The queue policy is as important as the pool size.
Step 3: Size by the workload
For CPU-bound work, size the pool near the number of cores so threads do not contend. For I/O-bound work, threads block on I/O, so a larger pool keeps cores busy — but each blocked thread still consumes a stack, so very large pools are costly. Match the size to whether the work waits or computes.
Worked scenario
The pool runs tasks with a bounded queue.
pool = FixedPool(threads=8, queue=100)
submit(task):
if queue.full: reject (backpressure)
else: enqueue; a worker picks it up when freeWalk through the example
At most eight tasks run at once, and up to a hundred wait in the queue; beyond that, submission is rejected so the caller learns the system is saturated. Workers are reused, so steady-state load does not create threads. The two bounds together keep memory predictable.
Common mistake
An unbounded queue, which converts overload into an out-of-memory crash instead of a fast rejection. Another is a pool far too small for blocking I/O, which leaves cores idle while tasks queue.
Verify the behavior
Submit more tasks than the queue holds and confirm rejections occur rather than memory growth. Measure CPU utilization with a CPU-bound workload at different pool sizes and find the point where more threads stop helping. Confirm threads are reused under steady load.
Interview exercise
Why is a pool of cores + 1 sometimes recommended for blocking work?
Answer and reasoning
When a task occasionally blocks, its thread idles and leaves the core unused; one extra thread can keep the core busy during that gap. It is a heuristic, not a rule — the right size depends on the blocking fraction and must be measured. For pure CPU work, more threads than cores only adds context-switch overhead.
Continue learning
Compare worker models in Producer-consumer and CPU scheduling. Read the Linux pthreads documentation and try the Operating systems interview questions.