Every I/O call answers two questions: does the caller wait, and who moves the data? The answers give you the classic I/O models. In blocking I/O, the thread waits inside the system call until the data is there. In non-blocking I/O, the call returns immediately with “not yet” if nothing is ready. In multiplexed I/O, one call waits on many descriptors and reports which are ready. In asynchronous, or completion-based, I/O you submit an operation and are told later that it has finished, with the data already in your buffer.
Interviewers ask about these because they decide how a server behaves under load. A design that’s fine with 50 connections can fall over at 5,000, and the reason is usually which model it used to wait.
Blocking I/O and the thread it costs
A blocking read() on a socket with no data puts the calling thread to sleep. The kernel wakes it when data arrives and copies the bytes into the buffer before returning. While asleep, the thread uses no CPU. That’s efficient, and it’s the easiest model to program: code reads top to bottom, and errors come back where the call was made.
The cost is that each concurrent wait needs its own thread. To measure what that means, I opened 2,000 socket pairs on an M1 Mac and started one Python thread per socket, each blocked in recv(), with the thread stack size set to 256 KiB. Then I did the same wait with 2,000 asyncio tasks on a single thread:
2000 blocked threads: +70.3 MiB RSS, 2001 threads
2000 waiting asyncio tasks: +4.6 MiB RSS, 1 threadAbout 35 KiB of resident memory per blocked thread, mostly stack pages and per-thread bookkeeping, against roughly 2 KiB per waiting task. The real numbers depend on the language and stack size: Java threads reserve about 1 MiB of stack each by default, though only touched pages become resident. Memory isn’t the only cost either. Thousands of runnable threads mean scheduler work and context switches, and each thread holds whatever per-request state it had when it blocked.
For a few hundred concurrent connections, thread-per-connection is often the right answer and is far easier to debug. It stops being right when most connections are idle most of the time, such as WebSockets, long polling or chat, because you pay for every waiting connection.
Non-blocking I/O on its own is a spin
Setting O_NONBLOCK on a descriptor makes read() return immediately with EAGAIN (or EWOULDBLOCK) when there’s nothing to read. That sounds like it frees the thread, but by itself it only changes where the waiting happens. To show that, I compared waiting one second for data with a blocking recv and with a loop that retries a non-blocking recv until it succeeds:
a.setblocking(False); spins = 0
while True:
try: a.recv(10); break
except BlockingIOError: spins += 1blocking recv : waited 1.01 s, CPU used 0.001 s
non-blocking spin: waited 1.00 s, CPU used 1.001 s, 1,314,536 EAGAIN returnsBoth waited one second. The blocking version used a millisecond of CPU; the spinning version used a full core for the whole second and made 1.3 million useless system calls. Adding a sleep between attempts lowers the CPU but adds latency, and you have to guess the interval.
Non-blocking descriptors become useful when you combine them with something that tells you when to try. That’s the next model.
Multiplexing: wait for readiness, then act
A readiness API (select, poll, epoll, kqueue) lets one thread wait for any of many descriptors to become readable or writable. When it returns, you perform the non-blocking reads and writes on the ready ones, handle whatever you can, and go back to waiting. This is the event loop inside nginx, Redis, Node.js and Python’s asyncio, and it’s why the single asyncio thread above could hold 2,000 waiting connections in under 5 MiB.
Two properties of this model are easy to miss. The I/O calls are still synchronous: when read() runs, the thread copies data from the kernel and waits for that copy. Readiness only guarantees the call won’t block waiting for data to arrive. And the event loop is one thread, so any CPU-heavy work or accidental blocking call (a synchronous DNS lookup, a slow JSON parse, a disk read) stalls every connection at once. Production event-loop servers either keep handlers short, push heavy work to a thread pool, or run one loop per core.
Readiness also has a blind spot: regular files. A disk file is always reported “ready” because a read won’t wait for a remote peer, yet the read can still block on the disk. That’s why Node.js runs file system operations on libuv’s thread pool rather than through its event loop.
Completion-based I/O
The fourth model flips the contract. Instead of asking “can I read now?”, you submit “read 64 KiB from this descriptor into this buffer” and get a notification when the read has completed. The kernel does the waiting and the copying. Windows has used this model for decades with I/O completion ports (IOCP). On Linux, io_uring (since kernel 5.1) provides it through two ring buffers shared between the process and the kernel: a submission queue you write requests into and a completion queue the kernel writes results into. Batching many submissions into one system call, or none with polling modes, also cuts the per-operation syscall cost.
The older POSIX AIO interface (aio_read and friends) exists too, but on Linux glibc implements it with user-space threads, which is part of why it never caught on. I didn’t run io_uring or IOCP for this note; the macOS machine used for the measurements has neither.
Completion APIs handle regular files properly, which readiness APIs can’t. The trade-off is complexity: buffers must stay valid until the operation completes, cancellation is a real protocol, and error handling moves away from the call site.
Note
“Asynchronous” in a language (async/await) describes how your code is written, not which kernel model sits underneath. asyncio and Node.js use readiness on sockets and threads for files; some runtimes use io_uring or IOCP. Ask which one when the distinction matters.
Choosing a model
A rough guide that holds up in practice:
- Few hundred concurrent requests, mostly busy: blocking I/O with a thread pool. Simplest, and the kernel scheduler does the work.
- Thousands of mostly idle connections: readiness-based event loop, usually through a framework (Netty, libuv, asyncio, Go’s runtime, which uses epoll or kqueue under goroutines).
- High-rate disk or mixed disk and network I/O: completion-based APIs where available, or a thread pool dedicated to blocking file calls.
- Anything: measure. Context switches, syscalls per request and memory per connection are all observable.
Virtual threads in Java 21 and goroutines in Go deserve a mention because they blur the line: you write blocking-style code, and the runtime parks the lightweight thread and multiplexes the waits onto epoll or kqueue for you. You get the programming model of the first section with most of the scalability of the third.
Exercise: the gateway that ran out of threads
An API gateway uses one platform thread per inbound request and calls a downstream service with a blocking HTTP client. Normally the downstream answers in 20 ms. One afternoon it slows to 2 seconds, and the gateway stops accepting requests even for routes that don’t use that service. Explain what happened and how the I/O model contributes.
Answer
With blocking I/O, every request waiting on the slow service holds a thread for the full 2 seconds instead of 20 ms, so the number of threads tied up grows a hundredfold at the same request rate. The thread pool fills with requests blocked in read() on downstream sockets, and new requests for any route queue behind them or get rejected, which is why unrelated routes fail too. The model isn’t wrong in itself, but it turns slow I/O into thread exhaustion. Fixes in order of effort: set a client timeout well below 2 seconds, and give each downstream its own bounded pool (a bulkhead) so one slow dependency can only consume its share. Longer term, move the gateway to an event-loop or virtual-thread model, where a waiting request costs a few kilobytes rather than a thread, so slow downstreams cause latency rather than an outage. Even then, keep the timeouts and limits, because an event loop can still accumulate unbounded pending requests.