Why not have each service call the push/email/SMS provider directly?

Because then every service that wants to notify a user has to know about every channel's provider API, and a slow provider blocks whatever service is trying to send. Centralizing notifications behind one service — and making it async and queue-based — decouples callers from provider details and isolates a slow or failing channel to its own queue instead of degrading everything upstream.

How do you stop a marketing campaign to 10 million users from delaying a one-time password text?

Separate queues by priority, not just by channel. A transactional send (OTP, security alert) and a bulk marketing send both eventually reach the same channel worker and provider, but they shouldn't compete in the same queue — the bulk campaign can and should be rate-limited independently so it never crowds out low-latency, time-sensitive sends.

How do you avoid sending the same notification twice on a retry?

The request carries a client-generated idempotency key, and the notification service deduplicates on (request, user, channel) before a send actually goes out. This is the same idempotency-key pattern used anywhere a caller can't be sure whether its last request actually succeeded — retries need to be safe by construction, not by hoping they don't happen.

What's the detail that separates a senior answer from a mid-level one here?

Recognizing that 'centralize the sends in one service' isn't enough on its own — the service also has to be asynchronous and priority-aware internally, with per-channel queues, or a single burst of bulk traffic (or one slow provider) degrades every other notification flowing through the same system.

Notification System Design

A notification system looks simple from the outside — send a push, send an email, send a text — until it has to do all three, for hundreds of millions of users, without a marketing blast delaying someone's one-time password. This post walks through that design the way an interview conversation actually unfolds.

Requirements and Goals of the System

Functional Requirements

Send a notification to a user across one or more channels: push, email, SMS
Respect each user's channel preferences and quiet hours
Support both time-sensitive transactional sends (a security code, a delivery update) and bulk/marketing sends
Deliver reliably — a notification is eventually delivered or explicitly marked failed, never silently dropped

Below the line (out of scope)

Notification template/content authoring tooling — assume a template already exists by ID
Device push-token registration itself — assume tokens/addresses are already on file
Engagement analytics (open rates, click-through) — a downstream consumer of delivery events, not this system's job

Non-Functional Requirements

Scale: ~500M users generating a few notifications each per day — call it 2-3 billion notifications/day
Latency: transactional sends should go out within seconds; bulk sends can reasonably take minutes, spread out deliberately
Reliability: every request results in an explicit outcome — delivered or failed — never a silent drop
Idempotency: a retried request must never result in a duplicate delivery to the user

Capacity Estimation and Constraints

Traffic: ~3 billion notifications/day averages to roughly 35,000/second, with sharp bursts around bulk campaigns or widespread events (an outage notice going to every active user at once, say).

Payload volume: each notification (template reference + a small amount of per-user data) is roughly 1KB once expanded per channel. A single bulk campaign to 10 million users, expanded across channels, is on the order of 10-30GB of fanned-out work queued at once — small in absolute storage terms, but a real burst in queue depth that has to be handled without starving transactional traffic sharing the same infrastructure.

Dedup window: deduplication only needs to look back far enough to catch realistic retries — a rolling 24-hour window of recently-sent (request, user, channel) keys, held in a fast, TTL'd store, comfortably covers this without keeping years of history around.

System APIs

http

POST /v1/notifications
{ "user_id" | "user_segment", "template_id", "data", "channels": ["push", "email", "sms"], "priority": "transactional" | "bulk" }
→ { "request_id" }

GET /v1/notifications/{request_id}/status
→ { "status": "queued" | "sent" | "delivered" | "failed", "per_channel": {...} }

High-Level Design

The core requirement to design around: reliably deliver a notification through the right channel(s), respecting preferences, without one type of traffic degrading another.

Bad solution: the calling service (say, the ride-sharing dispatch service from an earlier post) calls each channel's provider directly — APNs for push, an SMTP relay for email, Twilio for SMS — synchronously, right when the triggering event happens.

This tightly couples every service that wants to notify a user to the details of every channel's provider API, and it means a slow provider (an SMS gateway having a bad day) directly slows down the calling service's own request handling. There's also no single place to apply preferences, quiet hours, or deduplication consistently — every caller would have to reimplement that logic.

Good solution: introduce a centralized Notification Service. Calling services submit a request describing intent (which user, which template, what data, which channels) and the Notification Service handles preference lookup, channel selection, and the actual provider calls.

This decouples callers from channel details, but if the whole path is still synchronous end-to-end, a slow downstream provider still blocks whoever called in, and a large bulk campaign (10 million users) sent through the same synchronous path can overwhelm the system for everyone, transactional traffic included.

Great solution: make it fully asynchronous and priority-aware. A caller submits a request and gets an immediate acknowledgment; the request lands on a queue and a fanout worker expands it into one Notification per (user, channel) pair after applying preferences and quiet hours. Each of those lands on a per-channel queue, consumed by channel-specific workers that call the actual provider. Separately, requests are tagged transactional or bulk at submission time and routed to different queues with different priority — so a marketing campaign's fanout work never competes with a security alert for the same worker capacity.

Database Design

NotificationRequest — what a calling service asked for: template, data, target user or segment, priority
UserPreference — per-user channel opt-ins/opt-outs and quiet-hours window
Notification — one concrete instance: a specific user, a specific channel, with status (pending/sent/delivered/failed)
DeviceEndpoint — where a channel actually delivers to for a given user: a push token, an email address, a phone number

The modeling choice worth explaining if asked: NotificationRequest and Notification are deliberately two different entities, not one. A single request ("notify this user their order shipped") can fan out into two or three Notification rows (push, email, SMS), each tracked independently with its own delivery status — which is what makes per-channel retries, per-channel failure handling, and per-channel dedup possible without conflating "did the request get processed" with "did this specific channel actually deliver."

Detailed Component Design

How do we avoid sending the same notification twice on a retry?

The request carries a client-generated idempotency key. Before a fanout worker expands a request, it checks that key against a short-lived (24-hour) store of already-processed requests; a duplicate submission — the caller retried because it wasn't sure the first attempt succeeded — is recognized and short-circuited rather than fanned out again. The same idempotency applies one level down: a channel worker checks (request, user, channel) before actually calling the provider, since queue-based delivery is typically at-least-once, meaning a message can be redelivered even without the caller retrying anything.

How do we respect quiet hours and per-user preferences?

Each user's preference row carries their timezone, a quiet-hours window, and per-channel opt-outs, checked at fanout time before a Notification row is even created for a channel the user has opted out of or is currently in quiet hours for. The one wrinkle worth naming: not everything should honor quiet hours uniformly — a security alert (unusual login, fraud warning) is typically allowed to override quiet hours, while a marketing notification is not, so priority/category needs to be an input into the quiet-hours check, not just the clock.

How do we stop bulk traffic from delaying transactional sends?

This is why priority is tagged at the request level and carried through to separate queues, not bolted on later. A 10-million-user marketing campaign and a single OTP text both eventually reach the same channel worker pool and the same provider, but they never share a queue — bulk queues can be explicitly rate-limited and drained more slowly, while transactional queues stay short and are processed with priority, so a large campaign in flight never becomes the reason someone's login code arrives late.

What happens when a channel provider is down or rate-limiting us?

Channels with more than one viable provider (SMS especially often has two) should support failover — if the primary provider's error rate or latency crosses a threshold, new sends route to the secondary provider automatically, based on live provider health rather than a static configuration. Where there's no secondary provider, the send is retried with backoff and, if it eventually fails permanently, marked failed explicitly and visibly rather than left in an ambiguous pending state.

Fault Tolerance and Replication

Failure	Impact	Response
Queue broker node loss	Risk of losing queued requests	Queues run with replication (e.g. multiple broker replicas) so a single node loss doesn't lose in-flight messages
Channel worker crash	Messages it was processing need reprocessing	At-least-once redelivery from the queue; idempotency checks at the channel-worker level absorb the resulting duplicate delivery attempt safely
Provider outage (e.g. SMS gateway down)	Sends on that channel can't complete	Failover to a secondary provider where one exists; otherwise retry with backoff, and surface an explicit `failed` status rather than a silent drop
Fanout worker backlog	Notifications delayed, transactional and bulk alike, if not separated	Because transactional and bulk already run on separate queues, a backlog in the bulk queue doesn't propagate into transactional latency — this is the main payoff of separating them upstream, not just downstream

Extended Requirements

Observability & Monitoring

Signal	Why it matters	Alert on
Transactional send latency (P99)	The whole point of the priority split is protecting this number	Sustained P99 above a few seconds
Per-channel delivery failure rate	Flags a struggling or down provider fast	Sharp rise vs. per-provider baseline
Bulk queue depth / drain rate	Confirms large campaigns are draining at a sane, rate-limited pace	Depth growing rather than draining over a sustained window
Dedup hit rate	An unexpectedly high rate can indicate a caller retry-storm bug upstream	Sharp spike vs. baseline

Security & Compliance

Unsubscribe and opt-out requests have real regulatory teeth (CAN-SPAM, TCPA-style rules depending on jurisdiction and channel) — an opt-out has to take effect immediately and durably, checked at fanout time on every subsequent request, not eventually consistent with some lag. Contact information itself (email, phone number) is personal data that deserves the same access-scoping discipline as any other PII in the system — the notification service should be one of the few systems allowed to read it directly, rather than every calling service holding its own copy.

Leveled Expectations

Mid-level: Proposes centralizing sends behind one notification service, and can be prompted toward making it asynchronous rather than synchronous end-to-end.

Senior: Independently proposes per-channel queues and an async fanout worker; gets idempotency and dedup right at both the request and channel-send level without prompting.

Staff+: Separates transactional from bulk traffic at the queue level as a first-class design decision, not an afterthought; designs multi-provider failover based on live health rather than static config; reasons about quiet-hours overrides (security vs. marketing) as a real product/compliance distinction the system has to encode.

Notification System Design at Scale