Design a Payment System: Idempotency Keys, a Double-Entry Ledger and the Timeout You Can't Trust

A system design walkthrough for taking card payments through an external processor without double charges or lost money: why a status column and a balance field fail, the payment state machine and its 'unknown' state, idempotency keys end to end, webhooks that arrive twice and out of order, a double-entry ledger, the outbox, and daily reconciliation.

Rahul Bisht

Founder, CrawlPilot

·
Dec 8, 2026
·
Verified Nov 15, 2026
·System Design·
17 min read
·
Design a Payment System: Idempotency Keys, a Double-Entry Ledger and the Timeout You Can't Trust
Key takeaway
A payment is not a row with a status column, and a balance is not a number you overwrite. A payment is a state machine with a state called "we don't know yet", and a balance is the sum of an append-only ledger. Put an idempotency key on every hop, treat every webhook as a possibly-duplicated, possibly-late hint, and reconcile against the processor's records every day, because something will drift and you want to find it before your customers or your auditors do.

The intuitive model of a payment system is a payments table with a status column and a balance field on each seller's account. A charge succeeds, you set status to succeeded and add the amount to the balance. It's how most first versions are built, and it fails in three predictable ways: a retry charges the customer twice, a crash between two updates loses money, and six months later the balances don't match the bank and nobody can say why.

The fix isn't a better status column. It's a different model, with two ideas borrowed from accounting and distributed systems: a state machine that admits uncertainty, and a ledger that never overwrites anything. This walkthrough designs a marketplace checkout on top of an external card processor (Stripe, Adyen, Razorpay or similar), the way most companies actually take payments.

The Marketplace We're Designing For

A marketplace where buyers pay, sellers receive the amount minus a platform fee, and refunds happen. Working requirements (illustrative numbers):

  • Peak of 1,000 payments a second on a sale day; most days, a fraction of that.
  • Never charge a customer twice for one purchase, whatever retries happen between the browser, our services and the processor.
  • Never lose or invent money. Every cent in a seller's balance traces to specific payments, fees and refunds.
  • Reconcile to the cent every day against the processor's settlement reports and the bank.
  • Card data never touches our servers. The processor's hosted fields or SDK collect it, which keeps us out of most PCI scope.

The Naive Design, and Where It Breaks

The status-column payment flow
POST /pay1charge $1002status = succeeded3balance += 904Buyerclicks PayPayment serviceone big handlerCard processorcharges the cardpaymentsstatus columnsellers.balanceoverwritten
  1. The buyer clicks Pay. The browser posts to the payment service.
  2. The service charges the card. If the connection drops after the processor charged but before the response arrived, the service sees an error. The buyer sees 'payment failed' and clicks Pay again: second charge.
  3. On success the service updates the payment's status. A webhook from the processor may arrive and update the same row, in any order.
  4. Then it adds the seller's share to their balance. If the process crashes between these two writes, the payment says succeeded and the seller is never paid. If a retry runs this step twice, they're paid twice. Either way the number gives no clue what went wrong.

4 steps. Press Play to walk through them, or click any part of the diagram.

Checkout calls the payment service; it charges the processor, updates the payment's status, and adds to the seller's balance, as separate steps.

Each of those failures has a specific fix, and together they make up the design. I'll take them as decisions.

Decision 1: What Happens When You Don't Know?

The most important state in a payment system is the one the naive design doesn't have. When the call to the processor times out, the charge may have gone through or not. You genuinely don't know, and every design choice downstream depends on not pretending otherwise.

So a payment is a state machine with explicit uncertainty:

The payment state machine
1confirmed1declined2timeout3processor says so4refund settles5createdcheckout existssubmittingcharge sentunknowntimeout, no answersucceededprocessor confirmedfailedprocessor declinedrefundedreversal journal
  1. The happy path: the charge is sent and the processor confirms it.
  2. A decline is a definite answer. The buyer can try another card, as a new attempt under the same checkout.
  3. A timeout or dropped connection is not a decline. The payment moves to unknown, and the buyer sees 'confirming your payment'.
  4. A resolver asks the processor, by our idempotency key, what happened, and moves the payment to succeeded or failed on its answer.
  5. Refunds are forward moves too. Nothing ever moves a payment backward.

5 steps. Press Play to walk through them, or click any part of the diagram.

created → submitting → succeeded or failed; a timeout moves the payment to unknown, which only the processor's answer can resolve; succeeded can later move to refunded.

Two rules make it work:

  1. 01
    Only forward transitions. A payment that is succeeded can't become submitting because a late webhook said so. The database enforces this with a conditional update: UPDATE ... SET state = 'succeeded' WHERE id = ? AND state IN ('submitting', 'unknown').
  2. 02
    unknown is resolved by asking, never by guessing. A background resolver (a job on the scheduler) picks up payments in unknown, queries the processor by our idempotency key or payment reference, and moves them forward. The buyer's screen says "confirming your payment", not "failed".
Warning
The most expensive line of code in a payment system is the one that turns a timeout into "payment failed, please try again". If you can't confirm a charge, say you're confirming it.

Decision 2: Where Does the Idempotency Key Come From?

A retry should be recognised as the same request at every hop: browser to us, us to the processor, and our own internal retries. That needs a key that identifies the purchase, not the HTTP request.

One idempotency key, end to end
POST /pay, Idempotency-Key1insert key (unique)2charge with derived key3retry, same key4Checkout pagekey = checkout_7f3aPayment serviceidempotency tableIdempotencyrecordskey → request hash,Card processorkey = pay_7f3a_attempt1
  1. The checkout page creates a key when the checkout is created, not when Pay is clicked, so a double click, a refresh or a network retry all carry the same key.
  2. The service inserts the key with a hash of the request body. A unique constraint means only one request per key can proceed; a second request with a different body for the same key is rejected as a client bug.
  3. The service calls the processor with its own key derived from the payment and attempt number. If our call is retried, the processor sees the same key and returns the original charge instead of creating a new one.
  4. When the browser retries, the service finds the key already stored with a result and returns that same result. If the first attempt is still in progress, it returns 'in progress' and the client polls.

4 steps. Press Play to walk through them, or click any part of the diagram.

The checkout ID becomes the client's key; the payment service stores the first result under it and sends a derived key to the processor, so any retry at any hop maps to the same charge.

Processors already work this way, which is why this design leans on them. Stripe's API accepts an idempotency key on every write request, and saves the result of the first request made with a key, including failures, so repeating it returns the same response rather than doing the work again. Our service should behave identically toward its own clients. Stripe's engineering blog has a good long-form treatment of designing APIs around idempotency if you want the reasoning behind the pattern.

Keep idempotency records long enough to cover every realistic retry (days, not minutes), and store the response, not just "seen it". A retry deserves the same answer, not a 409.

Decision 3: Trust the Webhook or the API?

Processors tell you about asynchronous outcomes (a 3-D Secure challenge completed, a charge captured, a dispute opened, a refund settled) through webhooks. Stripe's documentation is explicit about two properties: an endpoint might receive the same event more than once, and events aren't guaranteed to arrive in the order they were created.

So the answer is both, with clear roles:

SourceRole in this design
WebhookA fast hint that something changed. Verify the signature, record the event ID (unique), acknowledge quickly, and process asynchronously.
Processor APIThe source of truth for an object's current state. When a webhook doesn't fit the state machine, or a payment sits in unknown, fetch the object.
Our state machineDecides whether a reported state is a legal forward move. Stale or duplicate events become no-ops.

That combination makes ordering irrelevant. A charge.refunded event arriving before charge.succeeded doesn't break anything: the handler sees a refund for a payment it still thinks is submitting, fetches the charge from the API, finds it succeeded then refunded, and applies both transitions in order.

Decision 4: Store Balances, or Derive Them?

This is the decision that separates payment systems that pass audits from ones that don't. The naive design stores a seller's balance and adds to it. The alternative is a double-entry ledger: every movement of money is a journal of two or more entries that debit some accounts and credit others, and each journal must sum to zero. Balances are never stored as the truth; they are the sum of entries.

Here's a $100 purchase with a 10% platform fee, as one journal:

AccountDebitCredit
Processor receivable (money the processor owes us)100.00
Seller payable: seller_4290.00
Platform fee revenue10.00
Total100.00100.00

A refund is another journal that reverses those entries. A payout to the seller is a journal that debits seller payable and credits the bank account. Nothing is ever updated or deleted. A correction is a new journal that says what it corrects.

Balance column
  • One row, one read
  • A double-applied update is invisible
  • No answer to 'why is it this number?'
  • Can't reconstruct a past balance
  • Drift shows up months later, at reconciliation
  • balance = balance + 90 on success
Double-entry ledger
  • Every cent traces to a journal and a payment
  • Errors surface as imbalance immediately
  • Point-in-time balances by summing to a date
  • More rows; balances need caching for fast reads
  • Append journals; each sums to zero
  • Balance = sum of entries

Verdict: Use the ledger as the source of truth and keep a cached balance per account that is rebuilt from it, never edited by hand.

A balance column versus a double-entry ledger

Practical rules for the ledger:

  • Money is an integer of minor units plus a currency code: 10000 INR-paise, 10000 USD-cents. Never floats. Currency conversion is an explicit journal with an FX rate on it, not a multiplication hidden in a report.
  • Each journal is written in one database transaction, with a check that it sums to zero per currency. A journal that doesn't balance is a bug, and the database should refuse it.
  • Each journal references what caused it: a payment ID, a refund ID, a payout ID, so you can walk from any entry back to the event that created it.
  • Cache balances, don't trust them. A per-account running balance updated in the same transaction as each journal keeps reads fast. A nightly job recomputes it from entries and alerts on any difference.

Decision 5: How Do Other Systems Find Out?

When a payment succeeds, the order service needs to mark the order paid, email needs to send a receipt, and fraud needs to update its features. The tempting approach is to publish an event to Kafka after committing the ledger journal. If the process crashes between the commit and the publish, the payment succeeded and nothing else knows.

The outbox pattern closes the gap: write the ledger journal, the payment state change and an outbox row in the same transaction. A separate relay reads the outbox and publishes to the event bus, marking rows as sent. Consumers deduplicate by event ID, because the relay may publish a row twice after its own crash. This is the same coordination problem the saga pattern post works through for multi-service workflows: commit locally, publish reliably, compensate explicitly.

The Assembled Design

Payment system with a ledger
pay (key)charge (derived key)webhooks: hintsjournal + state + outboxrelay publishescompare with processor reportsCheckoutidempotency keyPayment servicestate machineCard processorcharges + webhooksLedger + outboxone transactionEvent busorders, email, fraudReconciliationdaily, three-way
Checkout calls the payment service with an idempotency key; the service drives the state machine, calls the processor, and writes ledger journals plus outbox rows in one transaction; webhooks and a resolver keep states honest; reconciliation compares the ledger with the processor's reports.

Reconciliation: Assume Something Drifted

Even with all of the above, money will occasionally disagree: a processor-side refund your webhook handler never processed, a dispute fee, a currency rounding difference, a settlement that arrived a day late. Reconciliation is the daily job that finds it.

It's a three-way match:

  1. 01
    Our ledger says what we believe happened.
  2. 02
    The processor's settlement report says what it charged, refunded, and the fees it took.
  3. 03
    The bank statement says what actually arrived.

The job matches records by reference across all three and sorts the rest into a queue of breaks: in our ledger but not the processor's report, in the report but not our ledger, matched but with a different amount. Most breaks are timing (a settlement lands tomorrow) and clear themselves; the job ages them and alerts on any break older than a few days. The rest are bugs, and each one is a journal someone has to explain.

Note
Reconciliation is also your best fraud and bug detector for the payment system itself. The fraud detection design watches transactions; reconciliation watches your own code.

Before You Ship Payments

Check yourself

Q1The call to the processor times out. What state should the payment move to?

Q2When should the checkout page generate its idempotency key?

Q3A refund webhook arrives for a payment your system still has in 'submitting'. What do you do?

Cheat sheet
Payment system design

State

  • State machine with an explicit 'unknown'; only forward transitions.
  • Timeouts move to unknown; a resolver asks the processor.
  • Never tell a buyer 'failed' unless the processor said so.

Idempotency

  • Client key identifies the purchase, created with the checkout.
  • Store key, request hash and the first response; replay it.
  • Send a deterministic key to the processor for each attempt.

Ledger

  • Double-entry journals that sum to zero, append-only.
  • Integer minor units plus currency; FX as its own journal.
  • Cached balances rebuilt from entries nightly.

Around it

  • Webhooks are deduplicated hints; the API is the truth.
  • Outbox in the same transaction as the journal.
  • Daily three-way reconciliation with an aging breaks queue.

Sources