Real-Time Bank Fraud Detection System Design
A system design interview walkthrough for a real-time payments fraud detection system: scoring 8,000 transactions per second within a 200ms deadline, the rules-plus-ML hybrid, and the label pipeline that keeps it honest.
Every time a card gets swiped, a bank has about 200 milliseconds to decide: approve it, block it, or flag it for a human to look at. Get it wrong in one direction and a legitimate customer's payment fails at the register — on Black Friday, that's a customer who may never come back. Get it wrong in the other direction and the bank eats a fraudulent charge outright, no recourse. This post walks through designing that system the way you'd actually talk through it in an interview — starting simple, finding where it breaks, drawing the diagram again, and building up from there.
Requirements and Goals of the System
Functional Requirements
- Score every transaction as APPROVE, BLOCK, or FLAG_REVIEW
- Take into account the user's recent behavior (spend velocity, typical device, typical location)
- Let the fraud team add or change rules without an engineering deploy
- Feed analyst decisions and chargebacks back into the system so it keeps improving
Below the line (out of scope)
- Building the analyst review UI itself — we'll assume it exists and just talk about the API it calls
- The card network / chargeback dispute process — we only care about chargebacks as a label source
- KYC/identity verification at account creation — this system scores transactions, not onboarding
Non-Functional Requirements
- Scale: 8,000 transactions/second at peak (roughly a mid-size national card network)
- Latency: a hard 200ms deadline — if we don't answer in time, the payment gateway auto-approves, so slow is functionally the same as wrong
- Availability: 99.99% — an outage here means every transaction either auto-approves (fraud gets through) or auto-blocks (every customer's card stops working)
- Durability: every decision needs to be reconstructable later, for both audits and for retraining the model
Capacity Estimation and Constraints
Traffic: at an average of ~3,000 TPS with an 8,000 TPS peak, that's roughly 3,000 × 86,400 ≈ 260 million transactions/day.
Storage: each scored decision needs to keep the exact feature snapshot used, not just the transaction itself — call that ~2KB per record once rules-evaluated metadata is included. 260M records/day × 2KB ≈ 520GB/day of audit data. Regulators typically require 5 years of retention, which is roughly 950TB over that window before any compression — in practice, columnar storage (ClickHouse hot tier, Parquet cold tier) gets the effective footprint down substantially, but the raw number is why this data doesn't live in a row-oriented transactional database long-term.
Feature store (Redis) size: for, say, 500 million active users with a ~1KB behavioral profile each, that's ~500GB — comfortably shardable across a Redis cluster and small enough to keep entirely in memory, which is the point.
Bandwidth: 260M transactions/day at a couple KB per request+response is on the order of tens of MB/second sustained — trivial for modern networking. This confirms something worth saying out loud in an interview: this system's hard constraint is latency, not throughput or bandwidth. Nothing here is a "big data" problem in the volume sense; it's a "do this specific sequence of operations inside 200ms, every time" problem.
System APIs
Two endpoints, deliberately. Everything else — rules management, the analyst dashboard, alerting — is a client of the scoring and label APIs, not a new API surface.
decision_id doubles as an idempotency key. If the payment gateway's own client times out waiting for a response and retries — even though the first request actually succeeded — the scoring service recognizes the duplicate transaction_id against a short-lived cache of recent decisions and returns the original result immediately, rather than scoring the same transaction twice or, worse, letting a retried request race the original through the rules engine.
High-Level Design
The core requirement to design around: score a transaction accurately, inside 200ms.
Bad solution: score synchronously — on each request, query the database for the user's transaction history, compute their spend velocity and device history on the fly, then run the model.
This is the obvious first answer, and it falls apart immediately: computing 30-day spend velocity means scanning weeks of transaction rows inside the 200ms budget, for every single request, at 8,000 requests per second. No database survives that query pattern at that concurrency — this diagram is really just one arrow (scan 30 days of rows) away from a full outage.
Good solution: precompute a behavioral profile for each user ahead of time — spend velocity, typical devices, typical countries — and store it in a fast key-value store like Redis. The live request just reads one row instead of aggregating history.
This gets the read down to single-digit milliseconds, but it leaves one arrow deliberately unanswered — the dangling ? node above. If the scoring service updates the profile itself, synchronously, we're right back to doing expensive work inside the request path; the read got fast, but the write didn't.
Great solution: decouple scoring from learning entirely. The live request reads the precomputed profile from Redis (fast, cheap) and evaluates a rules engine plus an ML model against it. Meanwhile, every transaction is also published to a message queue (Kafka), and a separate stream processor consumes that queue asynchronously to update the behavioral profiles for next time. The scoring path never waits on this update — the dangling question from the "Good" diagram gets answered by an entirely separate, slower pipeline.
This is the shape almost every real-time scoring system converges on: a fast, read-only path for the live decision, and a slower, async path that keeps the data the live path reads fresh. The same pattern shows up in ad-click fraud detection and recommendation systems — the decision and the learning are always two different systems with two different latency budgets.
Within the scoring service itself, rules run first (cheap, explainable, instantly updatable) and can short-circuit straight to BLOCK or APPROVE; anything left ambiguous falls through to the ML model, which is the more expensive step (tens of milliseconds) and the main reason the 200ms budget matters at all.
Database Design
- User — whose transaction this is, plus a rolling behavioral profile (recent spend, typical devices, typical countries)
- Transaction — the thing being scored: amount, merchant, device, timestamp — written once and never modified afterward, since it's the record everything else refers back to
- PaymentInstrument — the card or account used; stored as a hash, never the raw card number, so the same instrument can be matched across users (a fraud-ring signal) without the system ever holding a real PAN
- DeviceFingerprint — a fuzzy identifier for the phone/browser the transaction came from; a transaction from a brand-new device on an established account is itself a signal
- Merchant — where the money is going; merchant category and recent chargeback rate are two of the strongest fraud features that exist
- ScoringDecision — the output: APPROVE/BLOCK/FLAG_REVIEW, which rules fired, what the model scored, and — critically — a snapshot of the exact features used, since the user's live profile will have moved on by the time anyone looks at this decision again
- FraudRule — a fraud-team-authored condition (e.g. "block if amount > $2,000 and device is new"), versioned and toggleable between shadow and active
- FraudLabel — ground truth arriving after the fact, from an analyst or a chargeback, referencing the ScoringDecision it corresponds to
Two things worth calling out explicitly if asked: the Transaction and ScoringDecision records are write-once — corrections and later labels are new rows referencing them, never in-place edits, because an auditor or a future training run needs to see exactly what the system knew at the time. And PaymentInstrument stores a hash rather than a real card number for the same reason a password is hashed rather than stored — it lets you match "this card was used by 40 different accounts this week" without ever holding the sensitive value itself.
Storage tiers follow from access pattern, not just entity type: User/Merchant/PaymentInstrument are low-write, ACID-appropriate rows in a relational database; the live UserBehaviorProfile read path lives in Redis, rebuildable from Kafka if lost; and ScoringDecision is append-only, write-heavy audit data suited to a columnar store (ClickHouse hot, S3 Parquet cold) rather than the same relational database as everything else.
Detailed Component Design
How do we handle a user who's already known to be bad, instantly?
Not every block decision needs a rules engine or a model at all. A user or device already confirmed fraudulent — from a prior chargeback, a prior analyst decision — should never have to wait on either; that's needless latency for a decision the system already knows the answer to. This is a distinct, even cheaper mechanism than the rules engine: a blocklist check against an in-memory Bloom filter, evaluated before rules and before the model.
A Bloom filter is a probabilistic set membership structure — it can say "definitely not in the blocklist" with certainty, or "possibly in the blocklist" with a small, tunable false-positive rate (well under 1%), in exchange for using a fraction of the memory a real hash set would need at this scale. A "possibly blocked" result falls through to a confirming lookup against the real blocklist store; a "definitely not blocked" result — the overwhelming majority of traffic — skips straight past this check in microseconds. The one property that matters most here: zero false negatives. A Bloom filter tuned this way will occasionally do a small amount of unnecessary confirming work, but it will never let an actually-blocked user slip through as a false "not blocked."
How do we actually keep this under 200ms?
Break the budget down: a Redis read for the profile costs a few milliseconds, the rules engine (running in-process, not as a network call — a network hop alone can burn 10-30ms) costs another few, which leaves the bulk of the budget — call it 60-100ms — for the ML model itself. The model runs as a co-located sidecar reached over a loopback call rather than a separate networked service, specifically to avoid paying network latency twice.
If the model is slow to respond and the deadline is close, the safest move is to abort early and fall back to rules-only — a late result that arrives after the gateway has already auto-approved is wasted work, not a correct answer. Notice in the sequence above that the rules engine sits before the model specifically because it's cheaper: a rule firing skips the most expensive step in the whole budget entirely.
The labels for training arrive 90 days late — how does that change the design?
This is the detail that separates a fraud system from a generic streaming pipeline, and it's worth raising even if the interviewer doesn't ask directly. A chargeback can take up to 90 days to arrive. Until then, a transaction's true label is unknown — not "legitimate."
If the training pipeline treats "no chargeback yet" as a negative label, the model learns that recent fraud looks safe, and its accuracy quietly degrades on exactly the transactions closest to the present — a $50 grocery charge from a compromised card looks identical to a real one for the first few weeks, and a model trained to say "no news is good news" will happily learn the wrong lesson from it. The fix is an explicit observation window: only transactions older than 90 days (or with an earlier confirmed label) are eligible for training, and the training join has to be point-in-time correct — using the feature snapshot stored with the original decision, not the user's current (since-updated) profile.
There's a second, subtler issue here: the training set only contains labels for transactions that were flagged or blocked in the first place. Fraud that was approved and never charged back is invisible to the model — it can't learn from mistakes it never got labeled feedback on. This is survivorship bias, and it's usually handled by occasionally, deliberately approving a small, low-value sample of borderline transactions just to get a label on what the model would otherwise never see.
How do we stop one bad rule from blocking too much legitimate traffic?
New rules should launch in a "shadow" mode — evaluated on live traffic, logged, but not actually affecting decisions — so the fraud team can see what percentage of traffic a rule would block before it goes live.
Once active, an automatic alert should fire if any single rule's block rate spikes well above its recent baseline, with a kill switch that can disable just that rule within seconds, without touching the rest of the system — the same shadow-then-promote pattern, in miniature, as the model rollout below.
How do we roll out a new model version without gambling on it blindly?
A brand-new model has the exact same trust problem as a brand-new rule, and the same shadow-mode answer applies: the new model runs alongside the live one, scoring every transaction, but its output is only ever logged — never used to make a real decision — until there's enough evidence to trust it.
Because real labels take 90 days to fully arrive, "enough evidence" at first has to mean proxy signals — comparing the shadow model's score against the live model's on transactions that have already charged back, or on the small slice of early, high-confidence labels — before a gradual, weighted traffic rollout replaces a hard cutover. A model promoted on day one with zero shadow traffic is a model the team is trusting on faith, not evidence.
Fault Tolerance and Replication
The system should degrade, not fail, and it should survive the loss of any single component without losing data.
| Failure | Impact | Response |
|---|---|---|
| Feature store (Redis) unreachable | Scoring would lose behavioral context | Fall back to conservative population-average features and a stricter threshold; flag the decision as degraded. Circuit breaker trips after a few consecutive slow responses |
| ML model down | No model score available | Fall back to rules-only scoring rather than blocking everything or approving everything |
| Kafka broker loss | Profile updates and audit writes would stall | Kafka runs with a replication factor of 3 — a single broker loss doesn't lose data, just briefly reduces headroom |
| Scoring service pod crash | Requests in flight to that pod fail | Pods run stateless across multiple availability zones behind a load balancer; a crashed pod is simply removed from rotation, no session or state is lost |
| Redis primary loss | Feature store briefly unavailable | Redis replicas promote automatically; because the feature store is a rebuildable cache (from Kafka replay), a short gap is recoverable rather than catastrophic |
| Model registry unreachable at pod startup | A new scoring pod can't load the current model on boot | The pod fails its startup probe, but the previously-downloaded model binary is cached on local disk as a fallback, so a registry outage doesn't stop new pods from coming up with a slightly older, still-valid model |
Two replication principles worth stating explicitly: the feature store is intentionally treated as disposable and rebuildable (from the Kafka event log), which is what makes losing a Redis node a minor incident instead of a data-loss incident — and the audit log (ScoringDecision) is the one place durability is non-negotiable, which is why it's written to two independent stores (ClickHouse and S3) before the write is considered complete.
Scalability, Availability, and Consistency
Every HLD interview eventually circles back to these three words. It's worth answering them explicitly rather than trusting that the diagrams above make the case on their own.
Scalability. The scoring service is stateless, so it scales the boring way — add more pods behind the load balancer, linearly, with no coordination needed between them. The more interesting scaling decision is the partition key: Kafka's raw-transactions topic and the Redis feature-store cluster are both partitioned by user_id. That single choice does two jobs at once — it spreads load evenly across many partitions/shards for parallelism, and it guarantees every event for a given user lands on the same partition, in order, which is exactly what the stream processor needs to update that user's behavioral profile correctly without coordinating with any other partition. Pick the wrong partition key (say, merchant_id, which is far more skewed — a handful of huge merchants dominate volume) and a few hot partitions end up doing most of the work while the rest sit idle.
Availability. A 99.99% target is a budget, not a slogan — it works out to roughly 52 minutes of allowed downtime a year. There's no single point of failure by design: scoring pods run across at least three availability zones behind a load balancer, Kafka and Redis are both replicated within a region, and the audit log is written to two independent stores. What's left deliberately open, and worth surfacing as an open question rather than glossing over: is cross-region failover active-active (every region can take live traffic at all times, which is more available but means the feature store has to reconcile writes from two regions) or active-passive (a warm standby region that takes over on failure, simpler but slightly slower to fail over)? Most real payment systems start active-passive precisely because reconciling a feature store across two actively-writing regions is a hard problem to take on before it's actually needed.
Consistency. Not every piece of data in this system carries the same consistency guarantee, and that's a deliberate choice, not an inconsistency to apologize for. The behavioral profile in Redis is eventually consistent — a few seconds of staleness after a transaction is an accepted, explicit trade-off for keeping the live read fast. Rule propagation across scoring pods is likewise eventually consistent, on a 5-10 second window. But the audit log (ScoringDecision) cannot be eventually consistent: once a decision is made, the record of it must be durable before the transaction is considered fully handled, because a regulator or a future training run has to see the exact truth, not a best-effort approximation that catches up later. Naming which parts of the system get which guarantee — and why — is a stronger answer than defaulting everything to either extreme.
| Layer | P50 | P99 | If this regresses |
|---|---|---|---|
| Feature store read | ~3ms | ~8ms | Circuit breaker trips above ~15ms; degrade to population-average features |
| Rules engine | ~2ms | ~5ms | Runs fully in-process, so regression here is rare; the same block-rate alerting that catches a bad rule would catch a rule that's also gotten slow |
| ML model | ~30ms | ~60ms | A spike toward 100-120ms is still inside the 200ms deadline; a sustained regression triggers autoscaling of model replicas rather than being absorbed silently |
| End-to-end | ~48ms | ~98ms | Comfortable headroom under 200ms with any single layer degraded — not enough headroom left for two layers regressing at once, which is why each layer gets its own alert rather than relying on the end-to-end number alone |
Extended Requirements
Observability & Monitoring
A system that silently gets worse is more dangerous than one that loudly fails, so the metrics worth watching split into two groups: is the system fast enough, and is it still accurate.
| Signal | Why it matters | Alert on |
|---|---|---|
| Scoring P99 latency | The whole design exists to hit 200ms | Sustained P99 > 160ms |
| Block rate, overall and per-rule | Catches a bad rule or model regression fast | Any rule's block rate spikes well above its 7-day baseline |
| Feature store hit rate | A drop means stale or missing behavioral profiles | Hit rate < 95% |
| Kafka consumer lag (profile updater) | If this falls behind, live scoring runs on stale behavior data | Lag > 5 minutes |
| Model score drift (e.g. population stability index) | Flags the model seeing a different distribution of transactions than it was trained on | Meaningful drift on any top feature |
| Audit log completeness | Every decision must be reconstructable later — a gap here is a compliance problem, not just an ops one | Any divergence between decisions made and decisions logged |
Distributed tracing across the request — gateway → feature read → rules → model → response — matters more here than a single latency dashboard, because when P99 does regress, the useful question is which hop, and a trace answers that in seconds where a dashboard alone doesn't.
Security & Compliance
A few of the modeling choices above aren't really engineering preferences — they're compliance requirements wearing an engineering hat. Never storing a raw card number (only a hash) is a PCI-DSS requirement, not a nice-to-have. Making ScoringDecision append-only, with no code path able to edit or delete a past decision, is what makes the audit trail actually trustworthy to a regulator. And a model with a meaningfully higher false-positive rate for one demographic or geography than another isn't a tuning knob to quietly accept — fair-lending rules mean that's a hard stop on shipping the model, checked automatically before any new version gets real traffic, not something a human reviews after the fact.
Access itself should be scoped as tightly as the data model implies: the scoring service gets insert-only access to the audit log and read-only access to the feature store, with no direct database write path at all; fraud analysts get read access to the audit log and can only modify rules or labels through the UI, never a direct database connection; ML engineers get read access to labeled training data but no access to the live scoring path. None of these roles need more than that, and granting more "to be safe" is itself the risk.
Leveled Expectations
Mid-level: Can design the basic synchronous scoring flow, identify that computing behavioral features on the fly is too slow, and land on a precomputed-profile-plus-cache approach with reasonable prompting.
Senior: Arrives at the decoupled real-time/async split independently, reasons clearly about the 200ms latency budget across each hop, and proactively discusses degraded-mode behavior when a dependency is slow or down.
Staff+: Goes beyond the scoring path to the feedback loop — raises the 90-day label delay and survivorship bias unprompted, discusses how a bad rule or a bad model gets caught and rolled back safely in production, treats the false-positive/false-negative trade-off as a business policy decision the system must stay configurable for, and explicitly names which parts of the system are eventually consistent versus strongly durable and why — including being willing to say that active-passive cross-region failover is the pragmatic starting point rather than reaching for active-active by default.
Open Questions
A real design doc doesn't end with every decision resolved, and saying so out loud in an interview reads as senior, not unfinished. A few worth naming rather than quietly assuming an answer to:
| # | Question | Owner | Priority |
|---|---|---|---|
| 1 | Beyond APPROVE/BLOCK/FLAG_REVIEW, should there be a fourth outcome — a step-up challenge (3D Secure) for borderline transactions instead of an outright block? | Risk + Engineering | High |
| 2 | Is the 5-10 second blocklist/rule propagation window fast enough, or does a fast-moving attack pattern need something closer to real-time? | Risk team | Medium |
| 3 | Is deliberately approving a small sample of borderline transactions (to get labels the model would otherwise never see) approved by the risk team, and at what transaction-value ceiling? | Risk team | High |
| 4 | Is the 90-day observation window right for every fraud type, or do slower-moving patterns (friendly fraud, disputed subscriptions) need a longer window? | Risk + Legal | Medium |
| 5 | Is active-passive cross-region failover sufficient for launch, or does the business case justify the added complexity of active-active? | Engineering + Leadership | High |
Further Reading
- Alex Xu — System Design Interview Vol. 2 (chapter on fraud/risk scoring systems)
