Design a Config Rollout System That Can't Take Down Your Whole Fleet
A system design walkthrough for pushing feature flags, WAF rules, routing tables and generated files to every server without letting one bad change go global: validation that treats your own config as input, staged waves with bake time, last-known-good that's actually good, kill switches, and the two clocks that decide every rollout.
Every config rollout is a race between two clocks: how fast a change spreads, and how fast you notice it's bad. If spread wins, one change reaches everything. The whole design is about slowing the first clock where it matters (waves, bake time, gates), speeding up the second (canary metrics, synchronous validation), and making sure that when the race is lost anyway, servers keep their last good config instead of crashing.
Every configuration change you ship starts two clocks. One measures how long it takes the change to reach every server. The other measures how long it takes anyone, human or automated, to notice the change is bad. When the second clock finishes first, you have a minor incident on a few machines. When the first finishes first, you have a global outage.
Read the three biggest infrastructure outages of late 2025 as a race and they look alike. Azure Front Door's bad config hit most edge sites in about four minutes, and its crash appeared about five minutes after the health check (full breakdown). Cloudflare's Bot Management file was pushed network-wide every five minutes by design. AWS's DNS automation applied a stale plan in a region before any check noticed. Control Plane's tracker of 62 major outages since 2020 puts configuration or deployment changes as the cause of 32% of them, the largest single category.
So here's the design for a system whose job is to make that race winnable: a config rollout platform for a large fleet. It covers feature flags, WAF rules, routing tables, rate limits, and the files other systems generate, like ML feature lists or a search suggestion table.
Working requirements, with numbers chosen to be realistic for a large company and labelled as such:
Scale (illustrative): 50,000 servers in 30 regions, 2,000 distinct config keys, a few hundred changes a day. Some files are bytes (a flag), some are megabytes (a routing table or feature file).
Fast when it's safe: an urgent block rule or a kill switch must reach every server in under a minute.
Slow when it's not: a normal change should take long enough to observe at each stage.
Never worse than the last good state: a bad config must not crash a server or make it serve wrong answers.
Every change attributable and reversible: who, what, why, and a one-step undo.
Those last two are the ones people skip. A config system that's merely fast is a very efficient way to distribute mistakes.
Most config systems start like this: a store of key-value pairs, an API to change them, and agents on every server that watch for changes. Facebook's Configerator paper describes the same core loop at its scale, and it's the right core. It's just not enough on its own.
The naive config push
An engineer, or an automated job, writes a new value through the API.
The store records it as a new version. So far, so good: there's history and an author.
Every agent is subscribed, so every agent learns about version 42 within seconds.
Every server reloads. If version 42 is bad, all 50,000 find out together, and the first human hears about it from a pager.
≡
4 steps. Press Play to walk through them, or click any part of the diagram.
An engineer saves a change; it goes into the store; every agent sees it and every server reloads within seconds.
Three things break here, and each maps to a real incident:
01
Nothing checks the value against reality. A generated file that doubles in size is just a new version. (Cloudflare, 18 November.)
02
Nothing waits. The spread clock runs at full speed for every change, urgent or not. (Azure, 29 October.)
03
Nothing defines failure on the server. When a reload fails, the server's behaviour is whatever the parsing code happens to do. (Both of the above.)
The most important line in Cloudflare's November post-mortem is a remediation: treat Cloudflare-generated configuration files with the same validation as user-generated input. Engineers validate what customers send them. They forget that a cron job writing a file at 3 AM is also a stranger.
Validation runs at write time, in the config API, before a version exists. There are four kinds, from cheapest to most expensive:
Check
Catches
Example
Schema
Wrong types, missing fields
timeout_ms is a string
Invariants
Values that are valid but impossible
More than 200 features; a weight of zero on every backend
Diff limits
Changes that are too big to be routine
Feature count changed by more than 20% since the last version; more than 5% of routes removed
Dry run
Values the consumer can't use
Load the file in the real parser, in a sandbox, with the production binary
The diff limit is the cheap check that would have stopped Cloudflare's file. You don't need to know the right number of features. You only need to notice that this file has twice as many as the last one, and make a human confirm it.
The dry run is the one that catches Azure-style bugs, but only if it runs the full processing path, including whatever happens asynchronously. Microsoft's completed repair after its outage was exactly this: fully synchronous processing before production stages. If the consumer processes config in a background thread a few minutes after loading, the dry run has to wait for that thread too.
Insight
Version skew is input too. While your config consumers are mid-upgrade, two binary versions read the same config, and the writers may be two versions as well. Run the dry run against both the current and the next consumer build. Azure's outage came from two valid changes processed by two different control-plane builds.
Now slow the spread clock, but only by as much as each change needs.
Split the fleet into waves of increasing size. The first wave is small and representative: a handful of servers in each kind of environment, carrying real traffic. Each wave has a bake time, and a gate that compares the wave's health metrics with servers still on the old version.
A staged config rollout
1errors, latency, crashes2pass3pass at each wave4fail5Version 42validatedWave 0: canary0.1%, bake 15 minGatecanary vs controlWaves 1–31% → 10% → 50%Wave 4everyoneStop + restore 41automatic
Version 42 has passed validation. It goes to wave 0 only: a small canary set spread across regions and server types.
It bakes. The bake time must be longer than the slowest failure you know about. If a parser can crash five minutes after load, a two-minute bake proves nothing.
The gate compares the canary's error rate, latency and crash count with servers still on version 41. Comparing against a live control group filters out unrelated noise like a traffic spike.
Waves widen, each with its own bake and gate. Later waves can bake for less time because the change has already survived real traffic.
Any failed gate halts the rollout and restores version 41 on every server that got 42. A human is paged after the restore, not before.
≡
5 steps. Press Play to walk through them, or click any part of the diagram.
Version 42 goes to a canary wave, bakes while its metrics are compared with the control group, then widens through larger waves; a failed gate stops it and restores version 41.
Google's SRE workbook calls the comparison against a control group canarying, and it works for config exactly as it does for binaries. The only new idea is that config needs it at all. Facebook's Configerator runs automated canary tests on config changes as a matter of course.
How long should the whole thing take? Here's the race for one bad change under two policies, with illustrative numbers: the spread clock is how much of the fleet has the change, and the detection clock is when an alert or gate fires.
Push everywhere
The whole fleet is broken before any alert fires
Recovery means repairing every server
Spread: 100% of servers in 60 seconds
Detection: errors cross the alert threshold at 3 minutes
Crash surfaces 5 minutes after load
Waves with 15-minute bake
Fifty servers are affected, not fifty thousand
Restore is automatic and scoped to the canary
Spread: 0.1% of servers for the first 15 minutes
Detection: canary gate fails at about 5 to 6 minutes
Crash surfaces 5 minutes after load
vs
Verdict: The bad change is the same. What changes is which clock finishes first.
Two rollout policies for the same bad change (illustrative numbers)
An emergency block rule for an active attack can't sit in a canary for fifteen minutes. So the system has change classes, each with its own policy, decided by the type of key, not by the person in a hurry:
Standard: full waves and bake times. Most changes.
Expedited: shorter bake, same waves, an approver from another team. For time-sensitive fixes.
Emergency: skips waves, but only for keys explicitly marked as safe to fast-path (block rules, kill switches), and only if the key's schema is so narrow that there's little to get wrong. Every use is reviewed afterward.
Cloudflare's 5 December incident is the cautionary tale here: a killswitch, the very thing meant to be the safe fast path, hit a code path that had never been exercised and returned errors for about 25 minutes. Fast paths have to be tested on a schedule, not trusted because they exist.
The first two layers make bad versions rare and limit how far they spread. The third decides what happens when one gets through anyway, and it lives on every server.
The rule is short: a server never replaces a config it validated with one it can't. The agent downloads the new version, validates it locally (schema, invariants, a real parse by the consuming code), and only swaps it in if all of that passes. If it fails, the server keeps the old version, marks itself as "stuck on 41", and reports it.
What a server does with a new version
1passes2fails3after bake: promote 424Download 42checksum + signatureLocal validationparse with real codeAtomic swap41 → 42Keep 41stuck + alertOn-disk LKGsurvives restarts
The agent downloads version 42 and checks its checksum and signature, so a truncated or tampered file never reaches the parser.
It validates locally using the same code that will consume it. If processing is asynchronous, validation waits for the asynchronous part too. Then it swaps atomically: requests see 41 or 42, never half of each.
If validation fails, the server keeps serving with version 41 and reports itself stuck. A fleet where 2% of servers are stuck on the old version is an alert. A fleet where 2% crashed is an outage.
Only after version 42 has baked successfully on this server is it promoted to the on-disk last-known-good, the version used after a restart.
≡
4 steps. Press Play to walk through them, or click any part of the diagram.
Agent downloads version 42, validates it locally, and swaps it in only if validation passes; otherwise it keeps 41 and reports.
That last step is the lesson from Azure. Microsoft's post-incident review says the bad metadata reached most edge sites and also updated the Last Known Good snapshot, so there was no clean version to roll back to. If "last known good" means "last version that loaded", it will faithfully capture any bug that appears after loading. Promote only after bake.
For optional modules, failing static has one more option: run without the module. If a bot-scoring feature file is unusable, the honest fallback is "no bot score, and say so", not "a score of zero for everyone", which is what Cloudflare's older proxy did on 18 November.
Warning
Fail-static has a shelf life. A server stuck on version 41 for a week is running config nobody is looking at, and a server that restarts with no valid local copy has nothing to fall back to. Track how long servers are stuck, page when it's more than a few hours, and ship a baked-in default config with the binary for the cold-start case.
Waves, gates and fail-static handle bad versions. You still need a way to handle a bad feature: something that worked in canary and goes wrong at full scale, or code that's fine until a certain config makes it run.
Undo is a new version equal to an old one, and it goes through the pipeline like any other change, usually as an expedited change. Because it is a known-good value, its gates can be short. It is not a separate code path, which is good: separate code paths are where the bugs are.
Kill switches are a different thing: a small set of booleans, one per risky feature, delivered over the emergency path and read by the feature's code on every request. Flipping one turns the feature off globally without touching its config. Cloudflare's remediation list after November includes more global kill switches for exactly this reason.
A kill switch is only worth having if it works when everything else is broken. That means it must not depend on the system it's protecting: if the dashboard login is behind the feature, you need another way to flip the switch. On 18 November most Cloudflare customers couldn't log in to their dashboard because Turnstile, which guards the login page, was itself down.
The storage underneath is simpler than the rest of the design. Every change is an immutable version:
text
config_version
key "edge/bot-features"
version 42
value_ref s3://config/edge/bot-features/42 (content-addressed)
author "featuregen-job" | "alice@"
reason "weekly model refresh"
class standard | expedited | emergency
validated_by [schema, invariants, diff, dryrun@build-1.84, dryrun@build-1.85]
rollout_state wave_2 (baking, 7 min left) | halted | complete
Two details pay for themselves the first time something goes wrong:
01
Generated config has an author too. "featuregen-job, triggered by a permissions change at 11:05" is the sentence an incident responder needs. Store the inputs a generator read, not only its output.
02
Rollout state is data, not a log line. An on-call engineer should be able to ask "which versions are mid-rollout right now, on which waves?" and get an answer in one query. During Cloudflare's November incident the five-minute flip-flopping looked like an attack for over an hour. A dashboard of config versions shipped over time would have shown that rhythm immediately.
Distribution is the part people expect a config post to be about, and it's the least risky part. A few notes:
Push for small keys, pull for big files. Flags and rules fit in a watch stream (etcd, ZooKeeper or a custom fan-out). Multi-megabyte files go to a content-addressed blob store; the stream carries only the new version's hash. Facebook moved large config distribution to a location-aware distribution system for this reason.
Fan out through a tree. Regional relays serve their region's agents, so 50,000 servers don't hammer one store, and a relay failure only delays one region.
The rollout controller decides who's eligible, not the distribution layer. Agents only fetch versions their wave is cleared for. Keep the "how" (bytes moving) and the "whether" (gates) in different components so a fast distribution layer can't skip a gate.