Filesystem Writes, Buffers and Durability

Few return values are misread as often as a successful write(). It means the kernel accepted your bytes. It doesn’t mean they’re on the disk, and if the power goes out a second later they may be gone, along with the confirmation you already sent the user. Interviewers working on storage, databases or payment systems ask about this because the gap between “written” and “durable” is where acknowledged data gets lost.

Name the failure first

“Is this durable?” has no answer until you say durable against what. Data passes through several layers on its way down, and each failure wipes out different ones:

1. The application’s buffer. Python’s buffered file objects, C’s stdio, a logging library’s queue. Lost if the process crashes.
2. The kernel’s page cache. After write() returns, the data sits in memory as dirty pages. It survives a process crash, since the kernel still has it, and is lost if the kernel crashes or the machine loses power before writeback.
3. The drive’s own cache. Most drives acknowledge writes once they’re in a volatile on-device cache. Lost on power failure unless the drive has power-loss protection or was told to flush.
4. The media. Survives power loss. Doesn’t survive the drive dying, which is what replication and backups are for.

So “survives a process crash” needs only a write() (or a flush() of user-space buffers). “Survives power loss” needs the data pushed through layers 2 and 3. Asking which one the requirement means is often the most valuable thing you can do in this interview.

What each call actually promises

• flush() on a buffered file object moves data from layer 1 to layer 2. Nothing more.
• write() copies into the page cache and returns.
• fsync(fd) asks the kernel to write the file’s dirty data and metadata to the device and wait. On Linux, with the default mount options, it also asks the device to flush its cache. fdatasync(fd) skips metadata that isn’t needed to read the data back, like the modification time, and can save a write.
• O_SYNC and O_DSYNC make every write() behave as if followed by fsync or fdatasync.
• On macOS, fsync is weaker. Apple’s man page says it moves data to the drive but doesn’t make the drive write its cache to permanent storage. For that you need fcntl(fd, F_FULLFSYNC).

What the promises cost

This appends 500 records of 121 bytes to a log, first with only write(), then with fsync after each, then with F_FULLFSYNC after each:

import os, fcntl, time

def append_records(mode, n=500):
    fd = os.open("journal.log", os.O_CREAT | os.O_WRONLY | os.O_TRUNC | os.O_APPEND, 0o644)
    record = b"x" * 120 + b"\n"
    t0 = time.perf_counter()
    for _ in range(n):
        os.write(fd, record)
        if mode == "fsync":
            os.fsync(fd)
        elif mode == "F_FULLFSYNC":
            fcntl.fcntl(fd, fcntl.F_FULLFSYNC)     # macOS: also flush the drive's own cache
    dt = time.perf_counter() - t0
    os.close(fd)
    print(f"{mode:12} {n} appends: {dt*1000:8.1f} ms total, {dt/n*1e6:8.1f} us per record")

Two runs on an M1 MacBook Pro’s internal SSD:

write only   500 appends:      1.0 ms total,      1.9 us per record
fsync        500 appends:     11.4 ms total,     22.8 us per record
F_FULLFSYNC  500 appends:   1896.2 ms total,   3792.5 us per record
write only   500 appends:      1.5 ms total,      2.9 us per record
fsync        500 appends:     15.0 ms total,     30.0 us per record
F_FULLFSYNC  500 appends:   1908.3 ms total,   3816.6 us per record

A plain write() costs 2 to 3 µs because it’s a memory copy. macOS’s fsync at 23 to 30 µs is suspiciously cheap for a storage round trip, and that’s the tell: it isn’t waiting for the media. F_FULLFSYNC at about 3.8 ms is the real price of getting a record onto stable storage here, more than a thousand times a plain write. This is also why SQLite has a fullfsync pragma for macOS, off by default: the safe option is that much slower.

Writing all 500 records and then calling F_FULLFSYNC once took 4.5 to 14 ms in total, against 1.9 seconds for one flush per record. A flush covers everything written before it, so you pay for flushes, not records. Databases call this group commit: hold incoming commits for a moment, write them together, flush once, then acknowledge them all.

Replacing a file without exposing half of it

Overwriting a config file in place is dangerous: a crash midway leaves a mix of old and new bytes. The standard pattern writes a complete new file and swaps names, because rename() within one filesystem atomically replaces the destination:

import os, fcntl, json

def full_sync(fd):
    # fsync on Linux; on macOS fsync leaves data in the drive cache, F_FULLFSYNC flushes it
    if hasattr(fcntl, "F_FULLFSYNC"):
        fcntl.fcntl(fd, fcntl.F_FULLFSYNC)
    else:
        os.fsync(fd)

def atomic_write_json(path, obj):
    directory = os.path.dirname(os.path.abspath(path))
    tmp = f"{path}.tmp.{os.getpid()}"
    fd = os.open(tmp, os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o644)
    try:
        os.write(fd, json.dumps(obj).encode())
        full_sync(fd)                       # 1. new contents are on stable storage
    finally:
        os.close(fd)
    os.rename(tmp, path)                    # 2. atomically swap the name to the new inode
    dfd = os.open(directory, os.O_RDONLY)
    try:
        full_sync(dfd)                      # 3. the rename itself is on stable storage
    finally:
        os.close(dfd)

atomic_write_json: 6.9 ms -> {"version": 2, "replicas": 3}

Each step guards against a specific failure. Skip the first sync and the rename can reach the disk before the data does; after a crash, the name points at an empty or partial file. ext4 users met exactly this around 2009 with delayed allocation, and ext4 added a heuristic for the rename case, but portable code shouldn’t depend on it. Skip the directory sync and the rename itself may not survive a crash, so the old file comes back after you told someone the new one was saved. And keep the temporary file in the same directory: rename() across filesystems fails with EXDEV.

When fsync itself fails

In 2018 PostgreSQL developers found that on Linux, when background writeback hit an I/O error, the kernel could mark the failed pages clean and report the error once. A later fsync retry would then return success even though the data never reached the disk. The episode, nicknamed “fsyncgate”, changed PostgreSQL’s behaviour: it now treats an fsync failure as fatal, crashes, and recovers from its write-ahead log rather than trusting a retry. The general lesson is that an fsync error means you no longer know what’s on disk, and retrying doesn’t tell you.

Appending safely to a log

Logs and write-ahead logs avoid in-place overwrites entirely: they only append, flush, then acknowledge. Two more details make them robust. A crash can leave a torn final record, so each record carries its length and a checksum, and recovery stops at the first record that doesn’t verify. And a newly created log file needs its directory synced once too, or the file itself might not exist after a crash. Preallocating the log file lets later appends use fdatasync, since the file size and block mapping don’t change.

Interview practice: durable writes at 5,000 per second

You’re designing the write path for a key-value store that must never lose an acknowledged write on power loss. It runs on SSDs where a cache flush costs about 4 ms, and it must handle 5,000 writes per second. How do you meet both requirements?

Answer

One flush per write caps the store at about 250 writes per second (1 / 0.004 s), so per-write flushing can’t hit 5,000. But the requirement is that acknowledged writes survive, not that each write gets its own flush, which allows group commit.

Incoming writes are appended to a write-ahead log in memory order. A single log writer collects whatever has arrived, writes it to the log, issues one flush (fdatasync on a preallocated file on Linux, F_FULLFSYNC on macOS), and only then acknowledges every write in that batch. At 5,000 writes per second and 4 ms per flush, each batch averages about 20 writes, and the worst-case added latency is roughly one flush interval plus waiting for the batch in progress, around 8 ms. Under light load batches are small and latency is just the flush.

Each log record gets a length prefix and checksum so recovery can find a torn tail, and the data structure (a memtable or B-tree) is rebuilt by replaying the log from the last checkpoint. I’d confirm the drives really honour flushes, or have power-loss protection, because a drive that acknowledges a flush early breaks every guarantee above. And power loss isn’t the only failure: losing the whole machine needs the log replicated to another node before acknowledging, which often costs less latency than a local flush.

More on storage and durability

• Operating Systems interview questions
• Operating Systems MCQs
• Operating Systems: Journaling and Durability
• Operating-System Page Cache and I/O Measurements
• fsync(2) on man7.org, including the note that the directory needs its own fsync