Design a Payment System: Idempotency Keys, a Double-Entry Ledger and the Timeout You Can't Trust
A system design walkthrough for taking card payments through an external processor without double charges or lost money: why a status column and a balance field fail, the payment state machine and its 'unknown' state, idempotency keys end to end, webhooks that arrive twice and out of order, a double-entry ledger, the outbox, and daily reconciliation.
A payment is not a row with a status column, and a balance is not a number you overwrite. A payment is a state machine with a state called "we don't know yet", and a balance is the sum of an append-only ledger. Put an idempotency key on every hop, treat every webhook as a possibly-duplicated, possibly-late hint, and reconcile against the processor's records every day, because something will drift and you want to find it before your customers or your auditors do.
The intuitive model of a payment system is a payments table with a status column and a balance field on each seller's account. A charge succeeds, you set status to succeeded and add the amount to the balance. It's how most first versions are built, and it fails in three predictable ways: a retry charges the customer twice, a crash between two updates loses money, and six months later the balances don't match the bank and nobody can say why.
The fix isn't a better status column. It's a different model, with two ideas borrowed from accounting and distributed systems: a state machine that admits uncertainty, and a ledger that never overwrites anything. This walkthrough designs a marketplace checkout on top of an external card processor (Stripe, Adyen, Razorpay or similar), the way most companies actually take payments.
The buyer clicks Pay. The browser posts to the payment service.
The service charges the card. If the connection drops after the processor charged but before the response arrived, the service sees an error. The buyer sees 'payment failed' and clicks Pay again: second charge.
On success the service updates the payment's status. A webhook from the processor may arrive and update the same row, in any order.
Then it adds the seller's share to their balance. If the process crashes between these two writes, the payment says succeeded and the seller is never paid. If a retry runs this step twice, they're paid twice. Either way the number gives no clue what went wrong.
≡
4 steps. Press Play to walk through them, or click any part of the diagram.
Checkout calls the payment service; it charges the processor, updates the payment's status, and adds to the seller's balance, as separate steps.
Each of those failures has a specific fix, and together they make up the design. I'll take them as decisions.
The most important state in a payment system is the one the naive design doesn't have. When the call to the processor times out, the charge may have gone through or not. You genuinely don't know, and every design choice downstream depends on not pretending otherwise.
So a payment is a state machine with explicit uncertainty:
The happy path: the charge is sent and the processor confirms it.
A decline is a definite answer. The buyer can try another card, as a new attempt under the same checkout.
A timeout or dropped connection is not a decline. The payment moves to unknown, and the buyer sees 'confirming your payment'.
A resolver asks the processor, by our idempotency key, what happened, and moves the payment to succeeded or failed on its answer.
Refunds are forward moves too. Nothing ever moves a payment backward.
≡
5 steps. Press Play to walk through them, or click any part of the diagram.
created → submitting → succeeded or failed; a timeout moves the payment to unknown, which only the processor's answer can resolve; succeeded can later move to refunded.
Two rules make it work:
01
Only forward transitions. A payment that is succeeded can't become submitting because a late webhook said so. The database enforces this with a conditional update: UPDATE ... SET state = 'succeeded' WHERE id = ? AND state IN ('submitting', 'unknown').
02
unknown is resolved by asking, never by guessing. A background resolver (a job on the scheduler) picks up payments in unknown, queries the processor by our idempotency key or payment reference, and moves them forward. The buyer's screen says "confirming your payment", not "failed".
Warning
The most expensive line of code in a payment system is the one that turns a timeout into "payment failed, please try again". If you can't confirm a charge, say you're confirming it.
A retry should be recognised as the same request at every hop: browser to us, us to the processor, and our own internal retries. That needs a key that identifies the purchase, not the HTTP request.
One idempotency key, end to end
POST /pay, Idempotency-Key1insert key (unique)2charge with derived key3retry, same key4Checkout pagekey = checkout_7f3aPayment serviceidempotency tableIdempotencyrecordskey → request hash,Card processorkey = pay_7f3a_attempt1
The checkout page creates a key when the checkout is created, not when Pay is clicked, so a double click, a refresh or a network retry all carry the same key.
The service inserts the key with a hash of the request body. A unique constraint means only one request per key can proceed; a second request with a different body for the same key is rejected as a client bug.
The service calls the processor with its own key derived from the payment and attempt number. If our call is retried, the processor sees the same key and returns the original charge instead of creating a new one.
When the browser retries, the service finds the key already stored with a result and returns that same result. If the first attempt is still in progress, it returns 'in progress' and the client polls.
≡
4 steps. Press Play to walk through them, or click any part of the diagram.
The checkout ID becomes the client's key; the payment service stores the first result under it and sends a derived key to the processor, so any retry at any hop maps to the same charge.
Processors already work this way, which is why this design leans on them. Stripe's API accepts an idempotency key on every write request, and saves the result of the first request made with a key, including failures, so repeating it returns the same response rather than doing the work again. Our service should behave identically toward its own clients. Stripe's engineering blog has a good long-form treatment of designing APIs around idempotency if you want the reasoning behind the pattern.
Keep idempotency records long enough to cover every realistic retry (days, not minutes), and store the response, not just "seen it". A retry deserves the same answer, not a 409.
Processors tell you about asynchronous outcomes (a 3-D Secure challenge completed, a charge captured, a dispute opened, a refund settled) through webhooks. Stripe's documentation is explicit about two properties: an endpoint might receive the same event more than once, and events aren't guaranteed to arrive in the order they were created.
So the answer is both, with clear roles:
Source
Role in this design
Webhook
A fast hint that something changed. Verify the signature, record the event ID (unique), acknowledge quickly, and process asynchronously.
Processor API
The source of truth for an object's current state. When a webhook doesn't fit the state machine, or a payment sits in unknown, fetch the object.
Our state machine
Decides whether a reported state is a legal forward move. Stale or duplicate events become no-ops.
That combination makes ordering irrelevant. A charge.refunded event arriving before charge.succeeded doesn't break anything: the handler sees a refund for a payment it still thinks is submitting, fetches the charge from the API, finds it succeeded then refunded, and applies both transitions in order.
This is the decision that separates payment systems that pass audits from ones that don't. The naive design stores a seller's balance and adds to it. The alternative is a double-entry ledger: every movement of money is a journal of two or more entries that debit some accounts and credit others, and each journal must sum to zero. Balances are never stored as the truth; they are the sum of entries.
Here's a $100 purchase with a 10% platform fee, as one journal:
Account
Debit
Credit
Processor receivable (money the processor owes us)
100.00
Seller payable: seller_42
90.00
Platform fee revenue
10.00
Total
100.00
100.00
A refund is another journal that reverses those entries. A payout to the seller is a journal that debits seller payable and credits the bank account. Nothing is ever updated or deleted. A correction is a new journal that says what it corrects.
Balance column
One row, one read
A double-applied update is invisible
No answer to 'why is it this number?'
Can't reconstruct a past balance
Drift shows up months later, at reconciliation
balance = balance + 90 on success
Double-entry ledger
Every cent traces to a journal and a payment
Errors surface as imbalance immediately
Point-in-time balances by summing to a date
More rows; balances need caching for fast reads
Append journals; each sums to zero
Balance = sum of entries
vs
Verdict: Use the ledger as the source of truth and keep a cached balance per account that is rebuilt from it, never edited by hand.
A balance column versus a double-entry ledger
Practical rules for the ledger:
Money is an integer of minor units plus a currency code: 10000 INR-paise, 10000 USD-cents. Never floats. Currency conversion is an explicit journal with an FX rate on it, not a multiplication hidden in a report.
Each journal is written in one database transaction, with a check that it sums to zero per currency. A journal that doesn't balance is a bug, and the database should refuse it.
Each journal references what caused it: a payment ID, a refund ID, a payout ID, so you can walk from any entry back to the event that created it.
Cache balances, don't trust them. A per-account running balance updated in the same transaction as each journal keeps reads fast. A nightly job recomputes it from entries and alerts on any difference.
When a payment succeeds, the order service needs to mark the order paid, email needs to send a receipt, and fraud needs to update its features. The tempting approach is to publish an event to Kafka after committing the ledger journal. If the process crashes between the commit and the publish, the payment succeeded and nothing else knows.
The outbox pattern closes the gap: write the ledger journal, the payment state change and an outbox row in the same transaction. A separate relay reads the outbox and publishes to the event bus, marking rows as sent. Consumers deduplicate by event ID, because the relay may publish a row twice after its own crash. This is the same coordination problem the saga pattern post works through for multi-service workflows: commit locally, publish reliably, compensate explicitly.
Checkout calls the payment service with an idempotency key; the service drives the state machine, calls the processor, and writes ledger journals plus outbox rows in one transaction; webhooks and a resolver keep states honest; reconciliation compares the ledger with the processor's reports.
Even with all of the above, money will occasionally disagree: a processor-side refund your webhook handler never processed, a dispute fee, a currency rounding difference, a settlement that arrived a day late. Reconciliation is the daily job that finds it.
It's a three-way match:
01
Our ledger says what we believe happened.
02
The processor's settlement report says what it charged, refunded, and the fees it took.
03
The bank statement says what actually arrived.
The job matches records by reference across all three and sorts the rest into a queue of breaks: in our ledger but not the processor's report, in the report but not our ledger, matched but with a different amount. Most breaks are timing (a settlement lands tomorrow) and clear themselves; the job ages them and alerts on any break older than a few days. The rest are bugs, and each one is a journal someone has to explain.
Note
Reconciliation is also your best fraud and bug detector for the payment system itself. The fraud detection design watches transactions; reconciliation watches your own code.