“Design a notification system” sounds like plumbing, and most of it is. What makes it a good interview question is that the plumbing has to respect rules a product manager cares about (don’t spam people, don’t wake them at 3 a.m., don’t send the same “your order shipped” message twice) while third-party providers fail underneath you. A strong answer separates deciding whether to notify from delivering the notification, and puts queues between the parts so each can fail independently.
Requirements worth agreeing up front
Start by asking what kind of notifications these are. The answer changes the design:
- Transactional: order shipped, password reset, payment failed. Low volume per user, must arrive quickly, users can’t opt out of some (security codes).
- Marketing: sales, recommendations, re-engagement. Huge bursts when a campaign starts, subject to opt-out laws and frequency caps, and nobody minds if they’re an hour late.
Channels are usually push (APNs for iOS, Firebase Cloud Messaging for Android), email and SMS, plus an in-app inbox. Delivery guarantee: at least once to the provider, with deduplication so the user sees each notification once in the common case. Exactly once all the way to a phone screen isn’t achievable, because once a provider accepts a message you can’t take it back.
Then size it. Say you have 50 million users, transactional traffic averages a few hundred per second, and marketing wants to send a campaign to 20 million users within ten minutes:
campaign: 33333 sends/s, 67 workers at 500/s eachThat’s from a two-line Node script, and the per-worker rate of 500 sends a second is an assumption you’d state, not a measured figure. The point it makes is the useful one: campaign traffic is roughly a hundred times the transactional baseline. If both share a queue, a campaign launch puts every password-reset code behind 20 million sale announcements.
The pipeline, piece by piece
1. Ingestion. Other services publish events like OrderShipped { orderId, userId } to a topic, or call an internal POST /notifications API. Prefer events written through a transactional outbox, so the “order shipped” notification can’t be lost if the order service crashes after committing. Ingestion validates the payload and writes it to a queue; it does no other work, so it stays fast.
2. The decision step. A worker consumes each event and decides what, if anything, to send. This is where the product rules live: preferences, deduplication, frequency caps, quiet hours, and channel choice. Here’s a small version of that logic in Node 22, run against a handful of events at 17:30 UTC:
const prefs = { u1: { push: true, email: true, marketing: true, tz: 'Europe/London' },
u2: { push: false, email: true, marketing: false, tz: 'Asia/Kolkata' },
u3: { push: true, email: false, marketing: true, tz: 'Asia/Kolkata' } };
const seen = new Set(); const sentToday = new Map();
function decide(evt, nowUtc) {
const p = prefs[evt.user];
if (evt.category === 'marketing' && !p.marketing) return 'drop: opted out of marketing';
const dedupeKey = `${evt.user}:${evt.type}:${evt.refId}`;
if (seen.has(dedupeKey)) return 'drop: duplicate ' + dedupeKey;
seen.add(dedupeKey);
if (evt.category === 'marketing') {
const n = sentToday.get(evt.user) ?? 0;
if (n >= 2) return 'drop: frequency cap (2 marketing/day)';
sentToday.set(evt.user, n + 1);
}
const hour = Number(new Intl.DateTimeFormat('en-GB', { hour: 'numeric', hourCycle: 'h23', timeZone: p.tz }).format(nowUtc));
if (evt.category === 'marketing' && (hour >= 22 || hour < 8)) return `defer: quiet hours (local ${hour}:00)`;
const channel = p.push ? 'push' : p.email ? 'email' : null;
return channel ? 'send via ' + channel : 'drop: no channel enabled';
}u1 order_shipped o-981 => send via push
u1 order_shipped o-981 => drop: duplicate u1:order_shipped:o-981
u2 order_shipped o-982 => send via email
u2 sale c-55 => drop: opted out of marketing
u1 sale c-55 => send via push
u1 sale c-56 => send via push
u1 sale c-57 => drop: frequency cap (2 marketing/day)
u3 sale c-55 => defer: quiet hours (local 23:00)A few decisions in there deserve explanation in an interview.
The dedupe key is user:type:refId, built from the business meaning of the notification, not from the event’s message id. If the order service publishes OrderShipped twice (a retry after a timeout, say), the two messages have different ids but describe the same shipment, and the key catches them. In production the seen set is a Redis SET key NX EX 86400 or a unique row in a table, with an expiry long enough to cover redeliveries.
Transactional messages ignore marketing opt-out and quiet hours. u2 has marketing turned off but still gets the shipping email. u3 is in Kolkata, where 17:30 UTC is 23:00, so the sale is deferred rather than dropped: push it onto a delay queue scheduled for 08:00 local time. Time zones come from the user’s profile, not the server, which is an easy detail to forget.
Look at the order of the checks, too. The frequency cap is counted before the quiet-hours check, so u3’s deferred sale has already used one of their two daily slots. That might be what product wants or it might be a bug, and pointing out that ordering of rules changes behaviour is exactly the sort of thing that impresses interviewers.
3. Per-channel queues and workers. The decision step writes a concrete message (push to device token X, title, body) onto a queue for that channel, and transactional and marketing traffic go on separate queues with separate worker pools. Workers render templates, call the provider, and record the result. Channels fail independently: an email provider outage shouldn’t stop push.
4. Status and the inbox. Store each notification with its status (queued, sent, delivered, failed, opened) for support queries and the in-app inbox. Email and SMS providers send delivery and bounce webhooks; feed those back into the status table, and mark hard-bounced addresses so you stop sending to them.
When providers fail
Providers time out, rate-limit you (429), and occasionally go down for an hour. Plan for each:
- Retries with exponential backoff and jitter, capped at a few attempts and a maximum age. A password-reset code that arrives 40 minutes late is worse than useless; a sale email that arrives late is fine. Give each category its own time-to-live.
- Dead-letter queue for messages that exhaust their retries, with an alert and a way to replay them.
- Permanent errors aren’t retried. APNs telling you a device token is no longer valid (
410 Unregistered) means delete the token, not try again tomorrow. - Failover for critical channels. Keep two SMS or email providers and route around one that’s failing, using a circuit breaker on error rate. Failover can create duplicates if the first provider accepted the message but timed out on the response, so it’s reserved for channels where a duplicate is better than nothing (a login code, yes; a marketing email, no).
- Watch the backlog. If a campaign queue builds a backlog of millions, newer messages queue behind stale ones. The AWS Builders’ Library piece on queue backlogs covers patterns like dropping messages past their TTL and serving newest-first when recency matters.
Practice question: the duplicate shipping texts
After a deploy, some customers received the same “Your order has shipped” SMS three times. Logs show the order service published the event once. The SMS worker’s logs show three successful provider calls for each affected message. What’s the likely cause, and how do you make it impossible to recur?
Answer. If the event was published once but sent three times, the repetition happened between the SMS queue and the provider. The likely story is that the provider call succeeded but the worker didn’t acknowledge the queue message before its visibility timeout or ack deadline expired. Maybe the deploy made workers slower, or reduced the timeout. The queue redelivered the message, another worker sent it again, and so on.
Fix the immediate cause by making the visibility timeout comfortably longer than a worst-case provider call, and by acknowledging promptly. But don’t rely on timing. Make the delivery step idempotent: before calling the provider, claim a per-message send key (sms:<notificationId>) with an atomic insert, and skip the call if it’s already claimed and marked sent. Pass the same id to the provider as its idempotency or client reference key if the provider supports one, so a retry after a timeout is deduplicated on their side too. Finally, add a metric on “sends per notification id greater than one” so a regression shows up on a dashboard before it shows up in customer complaints.