When One Zone Goes Dark: Designing for Zone Failure with Static Stability and Cells

At 06:08 UTC on Tuesday 20 October 2026, Google Cloud services in its London region started failing. Google later attributed it to a "power event" in one zone, europe-west2-b, one of the region's three zones. Through the day the Press Association reported problems for customers of Revolut, Lloyds, HSBC and Halifax, for the video platform the courts in England and Wales use for remote hearings, and for Royal Mail, DPD and Rightmove.

This post is not a diagnosis of that incident. As of 25 October, Google has not published its root-cause analysis; it said it would "publish an analysis of this incident once we have completed our internal investigation." What I can do is explain the general mechanics: why losing one zone so often hurts far more than a third of a system, and the designs that keep it to a third.

What Was Reported on 20 October

Here is the incident as far as reputable reports and Google's own status updates go, and no further:

Two details from Google's updates matter for the rest of this post. During recovery it said it was "prioritizing the recovery of existing workloads" and that "customers may not be able to yet create new workloads" (The Register). In other words, the ability to launch new capacity was itself limited while the zone recovered. Reports also differ on exact times: AFP reported Google saying most services were back to normal by 12:10 GMT, while PA reported Google saying at 3pm UK time that most issues were resolved, with a very small number of customers still affected. Which customers lost what, and why, will only be clear when the analysis lands.

Zones Are a Promise About Correlated Failure

A region is a geographic area; a zone is a deployment area inside it. Google's documentation says it "designs zones to minimize the risk of correlated failures caused by physical infrastructure outages" and that "if a zone becomes unavailable, you can transfer traffic to another zone in the same region to keep your services running" (Google Cloud docs). AWS makes the same promise with Availability Zones.

That promise has a precise shape, and it is narrower than people assume. The provider promises that zones are unlikely to fail together. It does not promise that your system survives one failing. That second part is your design: whether the other zones have room for the traffic, whether anything you need lives only in the dark zone, and whether getting back to normal requires an action that can't happen during the incident.

Here is what a three-zone service usually looks like on the whiteboard, and how a zone loss spreads through it.

Read the steps again and notice that only step 2 is the power event. Steps 3 to 6 are design choices, and each is a common route from "one zone" to "the region":

• Capacity that was never there. Step 3 is arithmetic. If the survivors weren't sized for 1.5× load, they fall over.
• Shared dependencies pinned to one zone. A single database primary, a config service, a cache cluster or a NAT gateway in the dark zone takes the healthy zones' requests down with it.
• Recovery that needs the control plane. Launching instances, scaling node pools, changing DNS or pushing config are all control-plane actions, and they are exactly what a mass recovery overloads. Google's own updates said new workloads might not be creatable during recovery.
• Retries and failover storms. Every client retrying at once is a thundering herd. Google's SRE book gives the arithmetic: if three layers each make 4 attempts, "a single user action may create 64 attempts (4^3) on the database" (Site Reliability Engineering, ch. 22). The same chapter describes a cluster failing, its traffic moving to a neighbour at capacity, and the neighbour then failing too.

Static Stability: Pay for the Failure Before It Happens

The AWS Builders' Library article on static stability (Mike Furr, 2019) gives the cleanest framing I know: "In a statically stable design, the overall system keeps working even when a dependency becomes impaired." The service already has what it needs to ride out the failure, so it can "continue operating correctly without having to launch any new EC2 instances." The load-balanced example in the article "requires no action" during an AZ impairment: failing instances drop out of rotation and the rest absorb the traffic.

The same article separates two kinds of machinery. The control plane is "the machinery involved in making changes to a system": launching, scaling, configuring. The data plane does the ongoing work: serving requests. The best practice it gives is to "separate systems along the control and data plane boundary", and make sure the data plane can keep serving with the control plane down.

Here is the same zone loss on a statically stable design:

What changed is mostly money and discipline, not technology. The capacity was bought in advance, the data was replicated before the failure, and nothing on the serving path needed a control-plane call.

The 67% Ceiling, and Why You Should Run Below It

The arithmetic is short. With Z equal zones carrying load L, each zone carries L/Z. Lose one and each survivor carries L/(Z−1). For every survivor to stay at or under 100% at peak, normal utilisation per zone must be at most (Z−1)/Z:

The three-zone row matches the AWS article: "when we use three Availability Zones, we overprovision by 50 percent" (Builders' Library).

The ceiling is the most you can get away with, not a target. Real zones aren't equal, load balancers don't split perfectly, and a failure arrives with a burst of retries on top. In the simulator below, static stability pre-provisions to 60% per zone (illustrative) so the survivors land near 90%, not exactly at the cliff.

Try it. Each tick is one moment of the incident; all numbers are illustrative:

1. 01
   Leave the defaults (70% busy, every defence off) and press Fail zone B. Watch it fail at the front door, then the cross-zone calls fail, then the survivors go past 100% while the autoscaler waits.
2. 02
   Press Reset, turn on static stability only, and fail B again. The capacity is there, yet errors stay high: the retries from the first ticks keep the survivors overloaded.
3. 03
   Add the retry budget. Then add zone-affine routing and see the failure shrink to the dark zone's own requests for a couple of ticks.
4. 04
   Drag utilisation to 65% and 70% with static stability off and watch the "Survives losing one zone?" tile flip.

Step 2 is worth stopping on. Capacity alone didn't save the system, because the overload was being sustained by its own retries. Bronson et al. call this a metastable failure: "a bad state that persists even when the trigger is removed", kept alive by a sustaining effect, and request retries are the first example they give. They also note that "many production systems choose to run in the vulnerable state all the time because it has much higher efficiency than the stable state." Static stability is the decision not to.

Keeping a Failure Inside One Box

Headroom answers "can we absorb the loss?" Isolation answers a different question: how much of the system does one failure touch in the first place? There are three tools, and they work at different scales.

Zone-affine routing. Once a request enters zone A, every downstream call it makes stays in zone A: the API calls the zone-A instance of the pricing service, which reads the zone-A cache and the zone-A replica. A dark zone B then fails only the requests that were already in B, instead of a third of every request's downstream calls everywhere. It also saves cross-zone latency and data-transfer cost on a normal day. The cost is that each zone needs a complete, balanced stack, and writes still need one leader somewhere, so the database is where affinity ends and replication takes over.

Cells. Zones protect you from the provider's failures. Most outages are your own: a bad deploy, a configuration push, one customer's request that crashes the parser. AWS's whitepaper Reducing the Scope of Impact with Cell-Based Architecture defines a cell as one of "multiple isolated instances of a workload": each cell "is independent, does not share state with other cells, and handles a subset of the overall workload requests." A cell router, "the thinnest possible layer", sends each request to its cell by a partition key such as customer ID. The whitepaper's arithmetic: with 10 cells serving 100 requests, a failure in one leaves 90% unaffected. It names exactly the failures zones can't contain: "unsuccessful code deployments" and "poison pill requests".

The router is the one shared component, which is why the whitepaper insists it stays thin: no business logic, a mapping it can serve from memory, and no dependency on the control plane that moves customers between cells. A router that calls a database on every request has rebuilt the single point of failure the cells were meant to remove.

Shuffle sharding sharpens cells when a cell is a set of shared workers rather than a full copy. The Builders' Library article on workload isolation using shuffle sharding works through eight workers. Plain sharding into four pairs means one bad customer takes out a quarter of everyone. Giving each customer a random pair instead yields 28 possible pairs: "the scope of impact due to a problem is just 1/28th. That's 7 times better than regular sharding." Two customers rarely share both workers, so when one customer's traffic kills its pair, almost every other customer still has one healthy worker. The article's Route 53 example assigns each domain four of 2,048 virtual name servers, giving "a staggering 730 billion possible shuffle shards."

Retries Are the Accelerant

A retry is a bet that the failure was transient. During a zone loss it usually isn't, and every client places the same bet at the same moment. Marc Brooker's Builders' Library article on timeouts, retries and backoff gives the worst case: a five-deep call stack retrying three times at each layer increases database load "243x, making it unlikely to ever recover."

The fixes are old and cheap, which makes skipping them hard to excuse:

• Retry at one layer only, usually the one closest to the user, and fail fast everywhere below it.
• A retry budget. Google's SRE book uses a "per-request retry budget of up to three attempts" plus a per-client budget: a request "will only be retried as long as this ratio is below 10%" (SRE book, ch. 21). In the worst case retries alone push volume to "just below 3X"; the per-client budget holds it to about 1.1×. Brooker's article reaches the same place with a local token bucket for retries. An LLM gateway caps its failovers per request for the same reason.
• Exponential backoff with jitter, so the retries that do happen don't arrive as one synchronised wave.
• Load shedding at the front door. When a zone is at capacity, rejecting the excess quickly with a clear error keeps goodput at capacity. Accepting everything and timing out drags goodput toward zero, which is the collapse you saw in the simulator.

The rate limiter post covers the token-bucket mechanics a retry budget uses.

The Defences as One Stack

None of these defences is enough alone; the simulator showed static stability failing without a retry budget. Treat them as a stack in which every layer has to hold. Switch one off to see the failure it was there to stop:

Cells aren't in this stack because they answer a different failure. A zone loss hits every cell's zone-B servers at once; cells earn their cost on bad deploys and poison requests, which hit every zone at once.

Multi-Region Is a Separate Decision

After a regional headline, someone will propose going multi-region. Sometimes that's right. But it is a different project with a different price, and it doesn't replace any of the above. Reports on 20 October quoted a Keele University lecturer saying "true resilience requires designing for regional failures through multi-region deployments" (The Register); that holds for failures that really are regional. A single zone going dark, though, is the failure the zone-level design exists for.

Before going multi-region, check three things:

• What is the failure you are paying for? If the outage that hurt you was one zone, a statically stable multi-zone design fixes it at a fraction of the cost.
• Where does the data live? Stateless services move between regions easily. A database that needs one writer either replicates asynchronously (and loses recent writes in a failover) or synchronously (and pays cross-region latency on every write). That choice is the real design work (the fraud detection design weighs it for a feature store written from two regions).
• Will you actually fail over? A second region that has never taken production traffic is a hope, not a plan. If it isn't exercised regularly, assume it won't work when needed.

For most teams the order is: make one region survive a zone loss with no control-plane action, add cells when your own deploys become the bigger risk, and add a second region only when the business needs to survive the loss of the whole region.

Check Yourself

Keep This

Sources

• The Register, "Google Cloud says a 'power event' caused today's outage", 20 October 2026.
• Press Association via The Irish News, "Google to investigate cloud outage after UK businesses hit by disruption", 20 October 2026.
• AFP via Kuwait Times, "Power outage hits Google Cloud services, disrupting UK banks", 20 October 2026.
• Google Cloud, "Regions and zones".
• Mike Furr, "Static stability using Availability Zones", Amazon Builders' Library.
• AWS, "Reducing the Scope of Impact with Cell-Based Architecture", AWS Well-Architected whitepaper.
• Colm MacCárthaigh, "Workload isolation using shuffle-sharding", Amazon Builders' Library.
• Marc Brooker, "Timeouts, retries, and backoff with jitter", Amazon Builders' Library.
• Google, Site Reliability Engineering, "Handling Overload" and "Addressing Cascading Failures".
• Nathan Bronson, Abutalib Aghayev, Aleksey Charapko and Timothy Zhu, "Metastable Failures in Distributed Systems", HotOS 2021.