“Too many open files” is one of the most common production errors that has nothing to do with files. It shows up as failed database connections, refused HTTP requests and crashed log rotation, usually days after a deploy, and it goes away after a restart, which is why it comes back. Interviewers ask about file descriptors to see whether you understand what the kernel is actually tracking, which limits apply, and how you’d find a leak instead of just raising the limit.
One small integer, three layers behind it
When a process opens a file, the kernel returns a small non-negative integer, the file descriptor. It’s an index into that process’s descriptor table. Each slot points to an open file description, a kernel object that holds the current offset, the access mode and status flags like O_APPEND or O_NONBLOCK. That in turn points to the underlying object: an inode for a file, a socket, a pipe, a device.
The middle layer matters more than people expect. After fork(), or after dup(), two descriptors point to the same open file description, so they share one offset. That’s how a shell’s child processes can all append to the same log in order. Opening the same file twice with two open() calls gives two descriptions with independent offsets.
Two rules explain most of what you see in practice. Descriptors 0, 1 and 2 are standard input, output and error, inherited from whoever started the process. And open() always returns the lowest free number. This Python script opens a file and a listening socket, lists its own descriptors with lsof, then closes the file and opens another:
import os, socket, subprocess
f = open("notes.txt", "w")
s = socket.socket(); s.bind(("127.0.0.1", 0)); s.listen()
print("file fd:", f.fileno(), " socket fd:", s.fileno(), " inheritable:", os.get_inheritable(f.fileno()))
out = subprocess.run(["lsof", "-a", "-p", str(os.getpid()), "-d", "0-9"], capture_output=True, text=True).stdout
for line in out.splitlines():
cols = line.split()
print(f"{cols[3]:>4} {cols[4]:6} {os.path.basename(cols[-1]) if cols[-1].startswith('/') else ' '.join(cols[8:])}")
f.close()
g = open("other.txt", "w")
print("after closing fd 3, the next open got fd", g.fileno())Run on macOS with stdin from /dev/null and output piped into cat:
file fd: 3 socket fd: 4 inheritable: False
FD TYPE NAME
0r CHR null
1 PIPE
2 PIPE
3w REG notes.txt
4u IPv4 localhost:52974 (LISTEN)
5 PIPE
7 PIPE
after closing fd 3, the next open got fd 3The file got 3, the socket got 4, and after closing 3 the next open reused it. Descriptors 5 and 7 are the pipes subprocess created to capture lsof’s output, a reminder that libraries open descriptors on your behalf. The inheritable: False is covered below.
Everything draws from the same pool
Regular files are a minority of most servers’ descriptors. Each TCP connection, listening socket, pipe, epoll or kqueue instance, eventfd, timer descriptor and open directory takes a slot. That’s why a leak of one kind breaks something else: leaked file handles make accept() fail on new connections.
Two TCP states are worth knowing because they show up in leak hunts. Connections in TIME_WAIT don’t hold a descriptor at all; the application already closed them and the kernel keeps a little state for a while. Connections in CLOSE_WAIT are the opposite: the peer closed its end, and your process still holds the descriptor because it never called close(). A growing pile of CLOSE_WAIT sockets is close to a confession.
Limits at several levels
The per-process limit is RLIMIT_NOFILE, with a soft value (what’s enforced) and a hard value (the ceiling an unprivileged process may raise its soft limit to). ulimit -n shows the soft one. Above it sit system-wide limits: on Linux, fs.file-max for the whole system and fs.nr_open for the largest per-process value; on this Mac, sysctl reports kern.maxfiles: 30720 and kern.maxfilesperproc: 10240.
The limit a process gets depends on how it was started, which surprises people. On this Mac, launchctl limit maxfiles shows a default soft limit of 256 for things launched by launchd, while the shell these experiments ran in had inherited a soft limit of 1,048,576 from its parent process. Same binary, different launcher, four thousand times the headroom. On Linux, a service under systemd gets its limit from LimitNOFILE= in the unit file, not from your login shell’s ulimit, so “it works when I run it by hand” proves little.
Reproducing exhaustion safely
You don’t need to exhaust a real server to see the failure. A throwaway process can lower its own soft limit to 64 and leak until it hits it:
import resource, socket
resource.setrlimit(resource.RLIMIT_NOFILE, (64, 64)) # only for this throwaway process
leaked = []
def handle_request(i):
f = open("notes.txt") # opened, never closed: the leak
leaked.append(f) # a cache, a list or a closure keeps it alive
return f.readline()
try:
for i in range(1000):
handle_request(i)
except OSError as e:
print(f"request {i} failed: {e!r}")
try:
socket.socket()
except OSError as e:
print(f"new socket also fails: {e!r}")
print("highest fd in use:", max(f.fileno() for f in leaked))
for f in leaked: f.close()
s = socket.socket(); print("after closing the leaked files, socket() got fd", s.fileno())request 61 failed: OSError(24, 'Too many open files')
new socket also fails: OSError(24, 'Too many open files')
highest fd in use: 63
after closing the leaked files, socket() got fd 3Descriptors 0 to 2 plus 61 leaked files fill all 64 slots, and request 61 gets error 24, EMFILE. The socket fails too, even though sockets were never the problem; in a real service, that’s the error you’d see in the logs. (ENFILE, error 23, is the system-wide table filling up instead.) Closing the leaked files frees the slots immediately.
The script keeps the files in a list deliberately. In CPython, a file object that becomes unreachable is usually closed at once by reference counting, and with warnings enabled you see ResourceWarning: unclosed file <_io.TextIOWrapper name='notes.txt' mode='r' encoding='utf-8'>. But anything that keeps a reference (a cache, a list of pending work, an exception traceback) keeps the descriptor open, and runtimes with tracing garbage collectors (the JVM, Go, PyPy) may not collect for a long time. Garbage collection manages memory. Descriptors need explicit lifetimes: with in Python, try-with-resources in Java, defer f.Close() in Go, using in C#.
Descriptors cross fork and exec
By default, a child created with fork() and exec() inherits every descriptor that isn’t marked close-on-exec. That causes two classic bugs. A server spawns a helper process, the helper inherits the listening socket, and after the server restarts the port is still held. Or a parent creates a pipe, starts a child, and waits for end-of-file on the read end, but EOF never comes because a second child also inherited the write end.
Python has made descriptors non-inheritable by default since 3.4 (PEP 446), which is the inheritable: False in the output above. In C, pass O_CLOEXEC to open() and SOCK_CLOEXEC to socket(), and only clear it on descriptors you mean to pass down.
Interview exercise: too many open files after three days
A Linux service runs fine for about three days, then starts logging Too many open files on incoming connections. A restart fixes it. A teammate proposes raising LimitNOFILE from 65,536 to 1,048,576. What do you do?
Answer
A failure that takes days and clears on restart is a leak, so raising the limit only changes three days into a few weeks and makes the eventual failure bigger. I’d find what’s accumulating first.
Measure the count over time: ls /proc/<pid>/fd | wc -l, or the process_open_fds metric many Prometheus clients export, and confirm it climbs steadily rather than spiking. Then group the open descriptors by type and target with lsof -p <pid>. If they’re sockets, ss -tanp shows their states: many CLOSE_WAIT connections to one upstream means the code doesn’t close connections when the peer hangs up, often an HTTP client response body that’s never closed or a connection pool that drops broken connections without closing them. If they’re regular files, the paths usually point straight at the code that opens them.
The fix is in the code: put the resource in a with or try-with-resources block, or fix the pool. Then add an alert at, say, 70% of the limit so the next leak is caught early. Also check how the accept loop handles EMFILE: the pending connection stays in the queue, so an event loop can spin at 100% CPU retrying it. Some servers keep one spare descriptor open, and on EMFILE they close it, accept and immediately close the connection, then reopen the spare, so clients get a clean refusal instead.
Related reading on I/O
- Operating Systems interview questions
- Operating Systems MCQs
- Operating Systems: epoll and select
- Processes Versus Threads and Shared State
- getrlimit(2) on man7.org, including RLIMIT_NOFILE and the soft and hard limit rules