Temporal Explained: Durable Workflows, When to Use Them, and a Saga You Can Crash

The worker process was killed 0.9 seconds into the checkout, after the card was charged and while the shipment was being created. About eleven seconds after the checkout began, a second worker process had created the shipment and sent the confirmation. It did not charge the card again, and nobody wrote recovery code for it.

That run, and the two below it, happened on a local Temporal server on October 25, 2026, using the code in this post. Each recorded history is linked from the walkthrough below. This is what Temporal is for: work that spans several services and has to finish even when the process running it doesn't.

This post builds on two others. The saga pattern explains choreography, orchestration and the pivot step. The durable agent runtime design builds the same machinery from scratch. Here we use the real thing.

Keeping a Checkout Alive by Hand

A checkout touches several systems: create the order, reserve stock, charge the card, book the shipment, email the customer. Any process running that sequence can die between two steps. The usual fix grows one piece at a time: a status column, a cron job that finds stuck orders, a retry loop, a flag that stops the cron job and the retry loop from charging twice.

What Temporal Is

Temporal started in 2019, founded by Samar Abbas and Maxim Fateev, who had built Cadence at Uber, which the company calls Temporal's direct predecessor (Temporal). It's open source, runs self-hosted or as Temporal Cloud, and has official SDKs for Go, Java, Python, TypeScript, .NET, Ruby, PHP and Rust (Temporal docs).

There are three moving parts. Your application asks the Temporal Service to start a workflow. The service stores the workflow's event history and puts tasks on a task queue. Workers, processes you deploy, poll that queue, run your code and report back what happened.

Workflows Decide, Activities Act

Temporal splits your code in two, and the split is the whole trick.

A workflow is the orchestration: call this, then that, and if this fails, do the other thing. It must be deterministic, because a new worker re-runs it against the recorded history and it has to make the same choices again. The TypeScript SDK runs workflows in a sandbox where Math.random(), Date and setTimeout() are replaced by deterministic versions, and workflow code can't import your activity code, only its types (Temporal docs).

An activity is a normal function that touches the world: an HTTP call, a database write, an email. It can fail, time out and be retried. Activities retry by default; workflows don't (Temporal docs).

Crash It: Event History and Replay

Here is the workflow we ran, the checkout saga built step by step later in this post, next to its real event histories. Pick Worker killed to see the run from the opening.

1export async function checkout(order: Order) {
2  const compensations = [];
3  try {
4    compensations.unshift(() => cancelOrder(order));
5    await createOrder(order);
6    compensations.unshift(() => releaseInventory(order));
7    await reserveInventory(order);
8    compensations.unshift(() => refundPayment(order));
9    await chargePayment(order);
10  } catch (err) {
11    await compensate(compensations); // newest first
12    throw err;
13  }
14  // Past the pivot: retry forward, never undo.
15  const shipmentId = await createShipment(order);
16  await sendConfirmation(order, shipmentId);
17  return shipmentId;
18}

Three things in that history are worth noticing:

1. 01
   A running attempt leaves no trace. Worker 1 started the shipment, then died. The history shows only that the activity was scheduled (event 23), and the server learned about the crash only when the 10-second start-to-close timeout ran out. Temporal's docs recommend always setting that timeout for exactly this reason: the server "doesn't detect failures when a Worker loses communication with the Server or crashes" (Temporal docs).
2. 02
   Retries don't add events. Event 24 is attempt 2, carrying the last failure ("activity StartToClose timeout") as an attribute. In the gateway run, the charge failed twice with a 503 and the history holds one started event: attempt 3.
3. 03
   Worker 2 rebuilt the workflow from the history. It had never seen order-1003. To decide what came after the shipment, it ran the workflow function from the top. createOrder, reserveInventory and chargePayment returned their recorded results, so the card wasn't charged again.

The Temporal Web UI shows the same run. The shipment bar runs from the moment worker 1 started it to the moment worker 2 finished it:

123

1. 1createShipment, marked attempt 2: the first attempt died with worker 1.
2. 2Ten seconds later, after the start-to-close timeout, worker 2 completes it and sends the confirmation.
3. 3COMPLETED, 11.5 seconds end to end.

Retries and Timeouts Are Policy

Every activity call comes with a retry policy. Temporal's defaults are a 1-second initial interval, a backoff coefficient of 2, a maximum interval of 100 times the initial interval, and unlimited attempts, with no error types marked non-retryable (Temporal docs). Each attempt also needs a start-to-close timeout or a schedule-to-close timeout; set neither and the service rejects the activity (Temporal docs).

The question that matters in an outage is how long your work sits idle after the dependency comes back:

With the defaults and a five-minute outage, the activity tries nine times, then waits up to 100 seconds between attempts, so it notices the recovery 27 seconds late. Lower the maximum interval if that matters more than the extra calls.

Two details from our code make retries safe rather than dangerous:

• A business "no" isn't retried. A declined card won't succeed on attempt 50. The activity throws ApplicationFailure.nonRetryable(..., 'CardDeclined'), and the policy also lists CardDeclined in nonRetryableErrorTypes, so the workflow sees the failure at once.
• Every side effect has an idempotency key. Retries are at-least-once. The charge passes the order id as its key, so a retry after a timeout can't charge twice. It's the same idea as idempotency keys in Redis or a payment API.

Where Temporal Fits, and Where It Doesn't

For streams of events, a stream processor is the better tool. See how Apache Flink keeps state and the ad click aggregator design. For agents, durable execution is the same machinery this durable runtime design builds by hand. Temporal names its own users, among them Netflix (150+ engineering teams and 500+ namespaces, by Temporal's account), OpenAI, NVIDIA and Booking.com (Temporal).

The Saga in Temporal

A saga runs a sequence of local steps and undoes the finished ones when a later step fails. In the saga post's terms, a Temporal workflow is an orchestrator, and it's a short one, because the workflow's own state lives in the event history.

Here is the workflow from the runs above, unedited. The whole project, with the activities, a worker, a one-command demo of all three runs and replay tests, is on GitHub: crawlpilot/temporal-checkout-saga.

Each choice in it comes from Temporal's saga guidance or the SDK docs:

• Register before the step. The docs call this "the safer default when Activities have side effects that may occur before failure": a charge can go through and its response still time out. The cost is that every compensation must handle a step that never ran. Ours do: refundPayment refunds only a charge made under the order's key, so it's a no-op when nothing was charged.
• Newest first. unshift keeps the list in reverse order, so the last step done is the first one undone.
• Compensations keep trying. They use a separate retry policy with no attempt limit. If one still fails, the loop logs it and carries on, as the docs advise.
• Cancellation doesn't stop the undo. If someone cancels the workflow mid-charge, the catch block still has to clean up. The TypeScript docs say "cleanup logic must be in a nonCancellable scope" (Temporal docs).
• Rethrow. The workflow fails with the original error, so its status tells the truth.

Pick Card declined to watch the undo list run against the real history:

1export async function checkout(order: Order) {
2  const compensations = [];
3  try {
4    compensations.unshift(() => cancelOrder(order));
5    await createOrder(order);
6    compensations.unshift(() => releaseInventory(order));
7    await reserveInventory(order);
8    compensations.unshift(() => refundPayment(order));
9    await chargePayment(order);
10  } catch (err) {
11    await compensate(compensations); // newest first
12    throw err;
13  }
14  // Past the pivot: retry forward, never undo.
15  const shipmentId = await createShipment(order);
16  await sendConfirmation(order, shipmentId);
17  return shipmentId;
18}

123

1. 1chargePayment fails with CardDeclined, a non-retryable error, so no retry.
2. 2refundPayment runs first: compensations run newest first.
3. 3Then releaseInventory and cancelOrder, and the workflow fails with the original error.

The Parts That Bite in Production

Changing code under running workflows. Replay expects your code to make the same calls in the same order. Add, remove or reorder an activity call and deploy it, and runs that started on the old code fail with a nondeterminism error. In TypeScript you wrap the change in patched('change-id'), which records a marker in the history so old runs take the old branch; later you switch to deprecatePatch() and finally remove both. Worker Versioning is the other route: tag workers by deployment version and pin runs to them (Temporal docs). The example repo's replay test shows both sides: add an activity call before the charge and npm test fails with Temporal's [TMPRL1100] Nondeterminism error; wrap the call in patched() and it passes.

Histories have a ceiling. One run's history is capped at 51,200 events or 50 MB, and you get warnings from 10,240 events or 10 MB (Temporal docs). A workflow that loops forever, such as a subscription that bills monthly, calls continue-as-new: same workflow id, a new run with a fresh history. A long history also costs time on every crash, since a new worker has to fetch and replay all of it.

Compensation isn't rollback. A refund is a new transaction the customer sees, not an erased charge. Steps past the pivot shouldn't be compensated at all. If a shipment really can't be created, that's a decision for a person, which a workflow can wait for as a signal, and the human approval section of the durable runtime post shows how that's raced against a timer.

Idempotency is still your job. Temporal records results exactly once, but runs activities at least once. Every activity that changes something outside needs a key the other side respects.

Check Yourself

Sources

• Temporal: company background: founding, Cadence, named users
• Temporal SDKs
• Retry policies
• Activity timeouts and failure detection
• Workflow basics in the TypeScript SDK
• Cancellation scopes in the TypeScript SDK
• Versioning in the TypeScript SDK
• Workflow execution limits
• Saga pattern
• crawlpilot/temporal-checkout-saga: the runnable code (MIT). npm run demo reproduces the three runs; npm test replays recorded histories.
• Our runs: Temporal TypeScript SDK 1.24.0 against the local dev server from @temporalio/testing, October 25, 2026. Histories: gateway retry, declined card, killed worker (stack traces removed).