Adding a cache is a one-line change and a long-term commitment. The line is if (cache.has(key)) return cache.get(key). The commitment is that, from now on, some users will see data that’s no longer true, and you’ve accepted responsibility for how old it can be and how it gets corrected. Interviewers asking about caching are usually checking whether you think about that second part.
This guide walks through the layers you can cache at in a Node.js service, then builds a small in-process cache and fixes the two bugs almost every first version has. The code ran on Node 22.
Where a cache can sit
Picture a product page request for an online shop. On its way to the database it passes several places where a copy could be kept:
- The browser and CDN, controlled by HTTP headers such as
Cache-Control: public, max-age=60. Cheapest of all, since the request never reaches Node. It only suits responses that are identical for everyone, or that vary by a header you declare withVary. - In the Node process, a
Mapor an LRU library. Lookups take well under a microsecond and need no network hop. Each process has its own copy, though, and it’s lost on every deploy. - A shared cache such as Redis or Memcached. One copy for all instances, survives deploys, and can be invalidated in one place. Each lookup is now a network round trip, typically a fraction of a millisecond within a data centre, and the cache itself can fail.
- The database, which has its own buffer cache. You don’t manage that one, but it’s why a repeated query is often faster than the first.
The choice follows from the data. A product’s description changes a few times a year, everyone sees the same text, and a minute of staleness is harmless: CDN plus a long TTL. Its price changes during a sale and must not lag for long: a short TTL in a shared cache, invalidated when the price is written. A user’s basket is private and changes constantly: usually not cached at all, or cached only for the duration of a request.
So the first interview question back to the interviewer is “how stale can this be?” Without that number, any TTL is a guess.
A first in-process cache, and the stampede it allows
Here’s the version most people write first:
const cache = new Map();
async function getProduct(id) {
if (cache.has(id)) return cache.get(id);
const product = await loadProduct(id); // database query, ~50 ms
cache.set(id, product);
return product;
}It looks right, and with requests arriving one at a time it is. Now fire five concurrent requests for the same product at a cold cache:
await Promise.all(Array.from({ length: 5 }, () => getProduct('sku-42')));
console.log('db calls:', dbCalls);db calls: 5All five calls checked the Map before any of them finished loading, so all five went to the database. With five requests that’s a shrug. When a cache entry for a popular key expires under real traffic, it’s hundreds of identical queries landing at once, which is the moment a database is least able to absorb them. This is called a cache stampede (or thundering herd), and it’s why a cache can make an outage worse: the system has been sized for the hit rate, and a mass expiry removes it all at once.
Single-flight: cache the promise
The fix is to store the promise on the first miss, before awaiting it. Every later caller in the same process gets the same in-flight promise and waits on that instead of starting its own load.
class TtlCache {
#entries = new Map();
constructor({ ttlMs, maxEntries }) {
this.ttlMs = ttlMs;
this.maxEntries = maxEntries;
}
get(key, load) {
const hit = this.#entries.get(key);
if (hit && hit.expiresAt > Date.now()) return hit.value;
const value = load(key);
this.#entries.set(key, { value, expiresAt: Date.now() + this.ttlMs });
value.catch(() => {
if (this.#entries.get(key)?.value === value) this.#entries.delete(key);
});
if (this.#entries.size > this.maxEntries) {
this.#entries.delete(this.#entries.keys().next().value);
}
return value;
}
delete(key) {
this.#entries.delete(key);
}
}Note that get isn’t async. It returns the stored promise directly, so there’s no await between checking the map and writing to it, and nothing else can run in that gap. Node runs your JavaScript on one thread, which is what makes this check-then-set safe inside a process without a lock.
Running the same experiment, plus two more:
concurrent gets: 5 db calls: 1
first sku-7: db timeout
second sku-7: 1999 db calls: 3
after invalidation, db calls: 4Five concurrent gets, one database call. That’s single-flight.
Don’t cache failures
The catch handler in get deals with the second classic bug. Because we cache the promise, a rejected promise would be cached too. If the database times out once, every request for the next 30 seconds gets the same rejection instantly, without ever trying again. One blip becomes a 30-second outage for that key, and it looks bizarre in the logs because the database is healthy.
The handler removes the entry when the load fails, and the identity check (=== value) makes sure it removes only its own entry, not a newer one that another request stored after an explicit delete. In the output above, sku-7’s first load failed, the entry was dropped, and the second call loaded again successfully (database calls went from 1 to 3).
Two related judgements come up in interviews. Should you cache “not found”? Often yes, briefly: caching a null for a nonexistent product for 10 seconds stops someone from hammering your database with random ids. That’s a deliberate negative cache with its own TTL, not an accident. And should you serve a stale value when the refresh fails? For a product description, yes; that’s the idea behind stale-while-revalidate in HTTP. For an account balance, no.
Expiry, size and invalidation
TTL bounds staleness when nobody tells the cache anything. If thousands of keys are written at the same moment, say at startup, they also expire at the same moment and stampede together. Adding jitter (ttlMs * (0.9 + Math.random() * 0.2)) spreads them out.
Size matters because an in-process cache lives on the V8 heap. Without maxEntries, a cache keyed by user id or search term grows until the process dies with JavaScript heap out of memory. The eviction above drops the oldest-inserted key, because a Map iterates in insertion order. That’s FIFO, not LRU. To make it least-recently-used, delete and re-insert the key on each hit, or use a library such as lru-cache that does this and also handles TTLs.
Invalidation on write is what keeps short-lived staleness short. When an admin changes the price, the write path calls cache.delete('sku-42'), and the next read loads fresh data (the fourth database call above). Delete rather than set the new value: two concurrent writers that each set can finish in the wrong order and leave the older value cached, while two deletes can’t disagree.
The catch is “which cache?” If you run four instances behind a load balancer, each has its own Map. The instance that handled the admin’s request deletes its copy; the other three keep serving the old price until their TTLs expire. Your options are a shared cache with one copy to delete, a broadcast (Redis pub/sub, a message on your event bus) telling every instance to drop the key, or accepting staleness up to the TTL and keeping the TTL short. Saying that out loud is usually the point at which an interviewer stops probing.
Interview exercise: the flash sale
“An e-commerce API runs eight Node instances. Product reads go through a per-process cache with a 5-minute TTL. During a flash sale, prices are cut at 12:00, but customers keep seeing old prices until about 12:05, and at 12:05 the database CPU spikes to 100% for a few seconds. Explain both symptoms and what you’d change.”
Answer. Both come from the per-process cache. The admin’s price update reached one instance at most, so the other seven kept the old entry until its 5-minute TTL ran out, which is the delay customers saw. And because the sale products were probably all loaded just before noon (traffic was building), their entries expired together at about 12:05. Each of the eight instances then missed on the same hot keys at once. Even with single-flight, that’s eight loads per key per wave, multiplied by every sale product, all in the same second.
I’d change three things. First, invalidate on write across instances: publish the changed product id on Redis pub/sub or the existing event bus, and have every instance delete that key, or move prices into a shared Redis cache that the write path deletes from. Second, keep single-flight inside each process and add TTL jitter so entries loaded at the same time don’t expire at the same time. Third, split the data by how stale it may be: descriptions and images can keep a long TTL (and CDN caching), while prices get a short TTL, maybe 30 seconds, as a safety net behind explicit invalidation. To verify, I’d load-test with concurrent requests at expiry time and assert that database queries per key stay at one per instance, and that a price change is visible on every instance within a second.