At 12:00:00 the cache entry for the homepage’s featured product expires. That key gets 5,000 requests a second. Every request that arrives after expiry misses, and every miss runs the same database query. If the query takes 300 ms, about 1,500 copies of it start before the first one finishes and puts the value back. The database sees a wall of identical work, slows down, and the rebuild takes longer, which lets even more requests miss.
That’s a cache stampede, also called a thundering herd or dog-piling. It’s a favourite interview follow-up because the naive cache-aside code has no defence against it, and the obvious fix (more cache) doesn’t help at all.
Why a bigger cache doesn’t help
A stampede isn’t caused by the cache being too small. The key was in the cache, it expired on schedule, and the problem is what happens during the few hundred milliseconds it takes to rebuild. Adding memory, nodes or replicas to Redis changes nothing about that window. Interviewers listen for this distinction: hit ratio problems are about capacity and eviction, stampedes are about coordination.
Stampedes come in two shapes:
- One hot key expires. A single popular entry, such as a homepage config or a celebrity’s profile, misses under heavy concurrent load.
- Many keys expire together. After a deploy, a cache flush or a cold start, thousands of keys get loaded within a few seconds, all with the same TTL. Ten minutes later they all expire in the same second.
The first needs refill coordination. The second needs expiry spread.
Counting the herd
The back-of-envelope formula is short: origin queries at expiry are roughly the request rate times the rebuild time. 5,000 requests per second times 0.3 seconds is 1,500.
That number assumes the query stays at 300 ms while 1,500 copies of it run, which it won’t. Connection pools fill, queries queue, and the rebuild might take two seconds instead, which means 10,000 misses. The feedback loop is the dangerous part. A database that comfortably handles the steady load can fall over at every expiry.
Four ways to coordinate the refill
Single-flight inside each process. The first request for a missing key starts the load; concurrent requests in the same process wait on the same promise instead of starting their own. In Node it’s a map of in-flight promises:
const inflight = new Map();
function loadOnce(key, loader) {
if (!inflight.has(key)) inflight.set(key, loader().finally(() => inflight.delete(key)));
return inflight.get(key);
}It’s cheap and has no network coordination, but it only deduplicates within one process. With 20 app servers you still get up to 20 queries per expiry.
A shared lock, with the old value served while one request rebuilds. Keep the value in Redis a little longer than its logical freshness (store a “fresh until” timestamp inside it). When a request sees the value is past its fresh time, it tries to take a short lock with SET ... NX PX, which succeeds only if the key doesn’t exist:
const got = await redis.set(`lock:${key}`, token, { NX: true, PX: 5000 });
if (got === 'OK') {
refreshInBackground(key).finally(() => releaseIfOwner(`lock:${key}`, token));
}
return cachedValue; // everyone, including the lock holder, answers with the old value right nowExactly one request rebuilds; the rest keep serving the slightly old value. The PX 5000 expiry matters: if the lock holder crashes, the lock frees itself after five seconds instead of blocking refills forever. Release only if you still own the lock (compare the token in a small Lua script), or a slow holder can delete a lock that someone else has since taken.
Early probabilistic refresh. Instead of everyone noticing the expiry at the same instant, each request rolls a die that gets more likely to trigger a refresh as expiry approaches. The XFetch rule (from Vattani, Chierichetti and Lowenstein’s paper on optimal stampede prevention) refreshes when:
const shouldRefresh = now - rebuildSeconds * beta * Math.log(Math.random()) >= expiresAt;Math.log(Math.random()) is negative, so the term pushes now forward by a random amount scaled by how long a rebuild takes. With beta = 1, one request typically refreshes the value a little before it expires, and nobody ever sees a miss. No lock is needed, though in practice you’d still guard against two refreshes overlapping.
Background refresh for known hot keys. If you know which keys are hot (the homepage, the top 100 products), a scheduled job can rewrite them before they expire, and request paths never rebuild them at all.
What the coordination buys
To compare these, a simulation runs one key at 5,000 requests per second for ten minutes, with a 60-second TTL, a 300 ms rebuild and 20 app servers:
hot key: 5000 req/s, TTL 60s, rebuild 300 ms, 10 minutes, 20 app servers
strategy origin queries peak concurrent requests that waited served stale
no protection 13453 1524 13453 0
single-flight per server 180 20 13492 0
shared lock, serve stale 9 1 0 13308
early refresh (XFetch) 10 1 0 0Nine expiries happened in the window. Without protection each one sent about 1,500 queries to the database, peaking at 1,524 running at once, and every one of those requests waited for its own query. Per-server single-flight cut the queries to 20 per expiry (one per server), but the waiting didn’t change: 13,492 requests still sat through a 300 ms rebuild. The shared lock got it down to one query per expiry, and nobody waited, at the cost of 13,308 responses using a value up to 300 ms past its freshness time. Early refresh did one query per minute, slightly ahead of expiry, with no waits and no stale reads.
Remember that this model keeps the rebuild fixed at 300 ms. In the unprotected case the real database would slow down under 1,500 concurrent copies, so the real gap is wider than the table shows.
Spreading expirations with jitter
The second shape needs a different fix. Here’s the peak number of keys expiring in any one second, when 50,000 keys are loaded at the same moment with a 600-second TTL:
50,000 keys warmed at once, TTL 600s no jitter : peak reloads in one second = 50000
50,000 keys warmed at once, TTL 600s ±10% jitter: peak reloads in one second = 457
50,000 keys warmed at once, TTL 600s ±20% jitter: peak reloads in one second = 248Without jitter, all 50,000 expire together and the database gets 50,000 reloads in one second. Adding a random ±10% to each TTL (so 540 to 660 seconds) spreads them over two minutes and the peak drops to 457. It’s one line of code (ttl * (0.9 + Math.random() * 0.2)) and it’s worth adding to every cache write by default.
Serving stale is a product decision
The lock-and-serve-stale approach is the most effective fix, and it rests on a question only the product can answer: is a value that’s a few seconds old acceptable here? For a product description or a recommendations list, yes. For the remaining stock in a flash sale, maybe not; there you’d rather have requests wait briefly for the one rebuild (with a timeout) than show “in stock” for an item that sold out.
Two details often get skipped. Waiters need a timeout and a fallback, or a stuck rebuild turns into a pile of hung requests. And the stale copy has to physically exist, which means the Redis TTL must be longer than the logical freshness window. If Redis deletes the key at expiry, there’s nothing stale to serve.
Interview practice: the ten-minute CPU spike
“Database CPU jumps to 100% for about two seconds every ten minutes. The spikes started after the team added a cache for the site configuration, which every page request reads. Traffic is 8,000 requests per second. What’s happening and what would you do?”
Answer
The ten-minute period almost certainly matches the config key’s TTL. Every page reads the key, so when it expires all in-flight requests miss together and each one runs the config query. At 8,000 requests per second, even a 250 ms query means about 2,000 concurrent copies, which explains the CPU spike, and the spike lasts two seconds because the database slows down under that load.
I’d stop the herd first. The config changes rarely, so serving a slightly old copy is fine: store it with a long Redis TTL, keep a fresh-until timestamp inside the value, and let one request take a SET NX PX lock to refresh it while everyone else keeps using the old copy. Since it’s one known hot key, a background job that refreshes it every few minutes is even simpler, and then no request ever rebuilds it. I’d also cache it in process memory for a few seconds, which removes most of the Redis traffic for a value every request needs. Finally, I’d add jitter to all other TTLs so warm-ups after deploys don’t line up, and confirm the fix by watching database queries per second at the expiry boundary.