Interrupts and Device Completion Notifications

Devices are slow and they finish work whenever they finish it. A disk read completes some microseconds or milliseconds after it was issued; a network packet arrives whenever the other side sends it. The CPU needs a way to find out. It can keep asking (polling), or the device can tell it (an interrupt). Interviewers use this topic to check whether you can trace what happens between “the packet hit the network card” and “my recv() returned”, and whether you know when the textbook answer, interrupts, stops being the right one.

Following one packet from wire to recv()

Take a TCP packet arriving at a Linux server. The steps, in order:

1. DMA. The network card writes the packet straight into a ring buffer in RAM that the driver set up earlier. The CPU isn’t involved yet.
2. The interrupt. The card signals that it has new data. Modern PCIe devices send a message-signalled interrupt (MSI-X), which can be aimed at a specific core. That core finishes its current instruction, saves enough state to resume, switches to kernel mode if it wasn’t already, and jumps to the handler registered for that interrupt vector.
3. The top half. The handler does as little as possible: acknowledge the device, perhaps mask further interrupts from that queue, and schedule the real work for later. Interrupt handlers run with at least that interrupt disabled and can’t sleep, so long handlers delay everything else.
4. Deferred work. On Linux this is a softirq, and for networking specifically NAPI: the kernel polls the card’s ring buffer and processes a batch of packets through the IP and TCP stacks. The payload ends up in the socket’s receive buffer.
5. The wake-up. If a thread is blocked in recv() or epoll_wait() on that socket, the kernel marks it runnable. That’s the moment the device’s completion becomes the application’s business.
6. Scheduling. The thread runs when the scheduler next picks it, on whatever core, and its recv() copies the bytes from the kernel buffer into user memory.

The thing interviewers listen for is the gap between steps 4 and 6. The device finished long ago; the data is safely in the kernel. If the application thread is busy, descheduled, or not reading that socket, the data waits. A full receive buffer then shrinks the TCP window and slows the sender. “The interrupt notifies the application” is the common misconception. It notifies the kernel, and the kernel decides when the application hears about it.

Polling versus blocking, measured

The trade-off between being told and asking can be measured without special hardware. In this experiment a child process plays the device: every 50 ms it writes the current monotonic clock time into a pipe. The parent waits for each message either by blocking in read() or by spinning on a non-blocking read(), then records how long the message took to reach it and how much CPU it used:

import os, time, resource, struct

def cpu():
    r = resource.getrusage(resource.RUSAGE_SELF); return r.ru_utime + r.ru_stime

def trial(mode, rounds=20):
    rfd, wfd = os.pipe()
    pid = os.fork()
    if pid == 0:                                  # child: the "device"
        os.close(rfd)
        for _ in range(rounds):
            time.sleep(0.05)                      # data shows up every 50 ms
            os.write(wfd, struct.pack("q", time.monotonic_ns()))
        os._exit(0)
    os.close(wfd)
    if mode == "poll":
        os.set_blocking(rfd, False)
    lat = []; c0 = cpu(); t0 = time.perf_counter()
    for _ in range(rounds):
        while True:
            try:
                data = os.read(rfd, 8)            # blocking: sleeps in the kernel until data arrives
                break
            except BlockingIOError:               # polling: nothing yet, ask again immediately
                continue
        lat.append(time.monotonic_ns() - struct.unpack("q", data)[0])
    wall = time.perf_counter() - t0; used = cpu() - c0
    os.waitpid(pid, 0)
    lat.sort()
    print(f"{mode:5}: wall {wall:.2f}s  CPU used {used:.2f}s  "
          f"wake-up latency median {lat[len(lat)//2]/1000:5.1f} us  max {lat[-1]/1000:6.1f} us")

Two rounds of each on an M1 Mac:

block: wall 1.04s  CPU used 0.00s  wake-up latency median  48.5 us  max   94.5 us
poll : wall 1.04s  CPU used 1.03s  wake-up latency median  10.8 us  max   61.4 us
block: wall 1.03s  CPU used 0.00s  wake-up latency median  48.0 us  max  274.5 us
poll : wall 1.04s  CPU used 1.03s  wake-up latency median  12.8 us  max   52.0 us

To be clear about what this measures: a pipe involves no hardware interrupt. But the blocking path is the same mechanism a driver uses at the end of step 5: the thread sleeps on a kernel wait queue and costs nothing until someone wakes it. Over one second, the blocking reader used no measurable CPU. The polling reader used 1.03 seconds of CPU in 1.04 seconds of wall time, a whole core, to get its data about 35 µs sooner. Some of the blocking latency is the scheduler and the idle core waking up; some is Python’s own overhead, which inflates both numbers equally.

That’s the trade-off in miniature. Interrupt-driven, blocking I/O is efficient when events are rare relative to the cost of handling one. Polling wins when events are so frequent, or latency so precious, that a dedicated core is worth spending.

When interrupts become the problem

Each interrupt costs a few microseconds of handler time plus disrupted caches on the core it lands on. At a few thousand events per second that’s nothing. A 10 GbE link carrying minimum-size packets can deliver over 14 million packets a second, and one interrupt per packet would consume the machine. Worse, a system can spend all its time taking interrupts and never get to the work they announced, a failure called receive livelock, described by Mogul and Ramakrishnan in 1997.

Drivers and kernels respond in a few ways:

• Interrupt coalescing. The card waits for a few packets or a few microseconds before interrupting, trading a little latency for far fewer interrupts. On Linux, ethtool -C tunes it.
• NAPI. After the first interrupt, Linux disables further interrupts from that queue and polls the ring buffer until it’s empty, then re-enables them. It’s interrupt-driven when idle and polling when busy.
• Busy polling and kernel bypass. SO_BUSY_POLL lets a socket spin on the device queue for lower latency. Frameworks like DPDK go further: a user-space driver polls the card from dedicated cores, with no interrupts at all, used in trading systems and packet processing where those cores are cheap compared with the latency.
• Fast storage. NVMe drives have per-core submission and completion queues, and with devices answering in a few microseconds, Linux can poll for completions instead of waiting for an interrupt. io_uring’s submission-queue polling mode applies the same idea to the system call side.

Where interrupts land, and other kinds

On Linux, /proc/interrupts shows a row per interrupt source with a count per CPU, which is the quickest way to spot all of a card’s interrupts landing on one core. Affinity is set per interrupt (/proc/irq/<n>/smp_affinity) or managed by irqbalance, and multi-queue cards spread flows across cores with receive-side scaling. macOS has no equivalent file, and its powermetrics tool, which reports interrupt rates, needs root, so I didn’t run either here.

Not every interrupt comes from a device. The timer interrupt drives preemption; “tickless” kernels stop it on idle cores to save power. Cores interrupt each other with inter-processor interrupts, for example to ask another core to reschedule or to flush stale TLB entries after a page table change. And traps and exceptions (a system call instruction, a page fault) enter the kernel through the same vector mechanism, which is why older x86 Linux made system calls with the int 0x80 instruction.

Interview practice: one core drowning in softirq

A Linux server receives a high rate of small UDP packets on a 25 GbE card. top shows one core at 100% in si (softirq time) while the other 31 cores are mostly idle, and the application reports missing packets. What’s happening, and how would you fix it?

A strong answer

All the receive processing is landing on one core. Either the card is using a single receive queue, or all its queues’ interrupts are routed to the same CPU, so that core takes every interrupt and runs every NAPI poll and softirq for the whole packet stream. Once it’s saturated, the ring buffer fills and the card drops packets before the kernel even sees them.

To confirm, I’d look at /proc/interrupts for the card’s queues (counts piling up in one CPU’s column), at ethtool -S for receive drop or missed counters, and at /proc/net/softnet_stat, where the second column counts packets dropped because the backlog was full and the third counts how often the poll ran out of budget.

Fixes: enable more hardware queues with ethtool -L and let receive-side scaling hash flows across them; spread the queue interrupts across cores with irqbalance or explicit smp_affinity; if the card can’t do multi-queue, use RPS to steer packets to other cores in software. Tune coalescing so each interrupt carries more packets, and increase the ring size with ethtool -G to absorb bursts. Finally, make sure the application has enough receiving threads or sockets (SO_REUSEPORT) on different cores, or the bottleneck moves up a layer.

More on I/O

• Operating Systems interview questions
• Operating Systems MCQs
• Operating Systems: I/O Models
• Operating Systems: epoll and select
• OSTEP, I/O Devices, on polling, interrupts, DMA and device drivers