Why not have each service call the push/email/SMS provider directly?

Because then every service that wants to notify a user has to know about every channel's provider API, and a slow provider blocks whatever service is trying to send. Centralizing notifications behind one service — and making it async and queue-based — decouples callers from provider details and isolates a slow or failing channel to its own queue instead of degrading everything upstream.

How do you stop a marketing campaign to 10 million users from delaying a one-time password text?

Separate queues by priority, not just by channel. A transactional send (OTP, security alert) and a bulk marketing send both eventually reach the same channel worker and provider, but they shouldn't compete in the same queue — the bulk campaign can and should be rate-limited independently so it never crowds out low-latency, time-sensitive sends.

How do you avoid sending the same notification twice on a retry?

The request carries a client-generated idempotency key, and the notification service deduplicates on (request, user, channel) before a send actually goes out. This is the same idempotency-key pattern used anywhere a caller can't be sure whether its last request actually succeeded — retries need to be safe by construction, not by hoping they don't happen.

What's the detail that separates a senior answer from a mid-level one here?

Recognizing that 'centralize the sends in one service' isn't enough on its own — the service also has to be asynchronous and priority-aware internally, with per-channel queues, or a single burst of bulk traffic (or one slow provider) degrades every other notification flowing through the same system.

Notification System Design

A notification system looks simple from the outside — send a push, send an email, send a text — until it has to do all three, for hundreds of millions of users, without a marketing blast delaying someone's one-time password. Every design decision below is really an answer to one question: which traffic gets to slow down which other traffic, and who decides.

What Has to Be True

Send a notification across one or more channels (push, email, SMS); respect each user's channel preferences and quiet hours; support both time-sensitive transactional sends (a security code, a delivery update) and bulk/marketing sends; deliver reliably, so a notification is eventually delivered or explicitly marked failed, never silently dropped. Out of scope: template-authoring tooling, device push-token registration (assume it's already on file), and engagement analytics (a downstream consumer of delivery events).

At ~500M users generating a few notifications each per day (2-3 billion/day, ~35,000/second average, with sharp bursts around bulk campaigns or an outage notice going to everyone at once), the interesting constraint isn't the average — it's that a single 10-million-user campaign, expanded per-channel, is 10-30GB of fanned-out work queued nearly at once, sharing infrastructure with traffic that needs to move in seconds, not minutes.

The rest of this design is a sequence of forks — five real decisions, each with a wrong-by-default answer, each of which the diagram at the end assembles into one architecture.

Fork One: Who Talks to the Providers?

Each calling service integrates with APNs, an SMTP relay, and Twilio directly, right when the triggering event happens — versus one centralized Notification Service that every caller submits intent to (which user, which template, which channels), leaving provider details behind a single boundary.

Direct integration is the version everyone reaches for first, and it tightly couples every service that wants to notify a user to every channel's provider API — a slow provider (an SMS gateway having a bad day) directly slows down whichever unrelated service happened to be calling it, and there's no single place to apply preferences, quiet hours, or dedup consistently. Centralizing wins here, decisively: it's the only version where preference logic gets written once.

Fork Two: Synchronous or Queued?

Centralizing alone doesn't fix the coupling problem — it just moves it. If the Notification Service still handles a request end-to-end synchronously, a slow downstream provider still blocks whoever called in, and a large bulk campaign sent through the same synchronous path can overwhelm the system for everyone, transactional traffic included.

The fix is full asynchrony: a caller submits a request and gets an immediate acknowledgment; the request lands on a queue, and a fanout worker expands it into one Notification per (user, channel) pair after applying preferences and quiet hours.

Fork Three: One Queue, or Priority-Split?

This is the fork the whole design exists to get right. Requests are tagged transactional or bulk at submission time. Sharing one queue means a 10-million-user marketing campaign and a single OTP text both wait in the same line — the campaign's sheer volume determines how long the OTP waits, which is exactly the failure mode this system is being designed to avoid. Splitting into separate queues per priority means bulk traffic can be explicitly rate-limited and drained slowly while transactional traffic stays short and is processed first, so a large campaign in flight is never the reason someone's login code arrives late.

Fork Four: One Provider per Channel, or Failover?

Channels with more than one viable provider (SMS especially often has two) can either commit to a single provider per channel, or support live failover — if the primary's error rate or latency crosses a threshold, new sends route to the secondary automatically, based on live provider health rather than static configuration. Where there's no secondary, the answer defaults to retry-with-backoff and an explicit failed status rather than a silent drop — never leaving a send in an ambiguous pending state.

Fork Five: Do Quiet Hours Apply to Everything?

Each user's preference row carries a quiet-hours window, checked at fanout time before a Notification row is created for a channel they're currently in quiet hours for. The wrinkle: quiet hours can't apply uniformly. A security alert (unusual login, fraud warning) is typically allowed to override quiet hours; a marketing notification is not — so priority/category has to be an input into the quiet-hours check, not just the clock.

The Architecture These Forks Add Up To

Requests vs. Notifications: Why They're Two Different Rows

NotificationRequest — what a calling service asked for: template, data, target user or segment, priority
UserPreference — per-user channel opt-ins/opt-outs and quiet-hours window
Notification — one concrete instance: a specific user, a specific channel, with status (pending/sent/delivered/failed)
DeviceEndpoint — where a channel actually delivers to for a given user: a push token, an email address, a phone number

NotificationRequest and Notification are deliberately two different entities. A single request ("notify this user their order shipped") can fan out into two or three Notification rows (push, email, SMS), each tracked independently with its own delivery status — which is what makes per-channel retries, per-channel failure handling, and per-channel dedup possible without conflating "did the request get processed" with "did this specific channel actually deliver."

http

POST /v1/notifications
{ "user_id" | "user_segment", "template_id", "data", "channels": ["push", "email", "sms"], "priority": "transactional" | "bulk" }
→ { "request_id" }

GET /v1/notifications/{request_id}/status
→ { "status": "queued" | "sent" | "delivered" | "failed", "per_channel": {...} }

Making Retries Safe: Idempotency at Two Levels

The request carries a client-generated idempotency key. Before a fanout worker expands a request, it checks that key against a short-lived (24-hour) store of already-processed requests; a duplicate submission — the caller retried because it wasn't sure the first attempt succeeded — is recognized and short-circuited rather than fanned out again. The same idempotency applies one level down: a channel worker checks (request, user, channel) before actually calling the provider, since queue-based delivery is typically at-least-once, meaning a message can be redelivered even without the caller retrying anything. A rolling 24-hour dedup window, held in a fast TTL'd store, comfortably covers realistic retry patterns without keeping years of history around.

What Happens When a Piece Goes Down

Failure	Impact	Response
Queue broker node loss	Risk of losing queued requests	Queues run with replication (e.g. multiple broker replicas) so a single node loss doesn't lose in-flight messages
Channel worker crash	Messages it was processing need reprocessing	At-least-once redelivery from the queue; idempotency checks at the channel-worker level absorb the resulting duplicate delivery attempt safely
Provider outage (e.g. SMS gateway down)	Sends on that channel can't complete	Failover to a secondary provider where one exists; otherwise retry with backoff, and surface an explicit `failed` status rather than a silent drop
Fanout worker backlog	Notifications delayed, transactional and bulk alike, if not separated	Because transactional and bulk already run on separate queues, a backlog in the bulk queue doesn't propagate into transactional latency — this is the main payoff of Fork Three, not just a downstream detail

Keeping Transactional Traffic Protected

Signal	Why it matters	Alert on
Transactional send latency (P99)	The whole point of Fork Three is protecting this number	Sustained P99 above a few seconds
Per-channel delivery failure rate	Flags a struggling or down provider fast	Sharp rise vs. per-provider baseline
Bulk queue depth / drain rate	Confirms large campaigns are draining at a sane, rate-limited pace	Depth growing rather than draining over a sustained window
Dedup hit rate	An unexpectedly high rate can indicate a caller retry-storm bug upstream	Sharp spike vs. baseline

Unsubscribe and opt-out requests have real regulatory teeth (CAN-SPAM, TCPA-style rules depending on jurisdiction and channel) — an opt-out has to take effect immediately and durably, checked at fanout time on every subsequent request, not eventually consistent with some lag. Contact information itself (email, phone number) is personal data that deserves the same access-scoping discipline as any other PII in the system — the notification service should be one of the few systems allowed to read it directly, rather than every calling service holding its own copy.

Reading the Depth of an Answer

Mid-level: Proposes centralizing sends behind one notification service, and can be prompted toward making it asynchronous rather than synchronous end-to-end.

Senior: Independently proposes per-channel queues and an async fanout worker; gets idempotency and dedup right at both the request and channel-send level without prompting.

Staff+: Separates transactional from bulk traffic at the queue level as a first-class design decision, not an afterthought (Fork Three, unprompted); designs multi-provider failover based on live health rather than static config; reasons about quiet-hours overrides (security vs. marketing) as a real product/compliance distinction the system has to encode.

Notification System Design at Scale