The Query That Didn't Name Its Database: Cloudflare's 18 November Outage
How a database permissions change doubled a Bot Management feature file, crashed Cloudflare's proxy on every machine that loaded it, and flickered like a DDoS for an hour. Plus the 5 December follow-up, the pattern both share, and what a fifth of the web sitting behind one proxy means for you.
Cloudflare's worst outage since 2019 started with a permissions change that made one SQL query return twice as many rows. The file built from those rows was pushed to every proxy within minutes, broke a hard size limit, and crashed the new proxy outright. Seventeen days later a different change, pushed through the same fast global path, tripped a years-old bug in the old proxy. The shared lesson: configuration your own systems generate is still untrusted input, and the faster you can ship it everywhere, the more it needs gates.
Two numbers explain most of this outage: about 60 and 200. Cloudflare's bot-scoring model used about 60 features, and its newest proxy reserved memory for at most 200. On 18 November 2025 the file listing those features suddenly doubled in size and went past the limit, and the proxy on every machine that loaded it crashed. The SQL that built the file hadn't changed. It filtered column metadata by table name and never said which database, which was harmless until a permissions change at 11:05 UTC let it see a second one.
What follows is a reading of Cloudflare's own post-mortem, written by CEO Matthew Prince the same day, plus its 5 December follow-up. Cloudflare is explicit that the issue was "not caused, directly or indirectly, by a cyber attack or malicious activity of any kind."
Cloudflare's Bot Management gives every request a bot score using a machine-learning model. The model's inputs, called features, are listed in a feature file that is regenerated every five minutes from a ClickHouse query and pushed to every machine in the network, so the model can keep up with new bot behaviour. Fast distribution is the feature, not a bug.
Step through how that pipeline turned a permissions change into a crash.
How one permissions change reached every proxy
Cloudflare changes ClickHouse access controls so users can see metadata for the r0 tables that sit under the distributed tables in the default database. A reasonable, deliberate change.
The feature-generation query reads column names from system.columns filtered by table name only. Now it sees the matching columns in both default and r0, and returns roughly twice the rows.
The duplicated rows double the feature file. Nothing in the pipeline checks that the number of features is plausible.
The file is pushed to every machine on the network within minutes, as designed.
The proxy preallocates memory for up to 200 features, far above the roughly 60 in use. The oversized file exceeds it, the error isn't handled, and the FL2 worker thread panics. Requests that touch Bot Management get 5xx errors.
≡
5 steps. Press Play to walk through them, or click any part of the diagram.
The panic message Cloudflare quoted is the most-shared line of the post-mortem: called Result::unwrap() on an Err value. It's tempting to read it as "Rust's unwrap took down the internet". I think that's the wrong lesson. The limit was deliberate, a preallocation for performance on the hottest path in the company. The loader correctly returned an error when the file broke the limit. What was missing was a decision about what the proxy should do with a bad config file: keep the old one, skip the module, or die. Without that decision written down, unwrap() made it by default.
For the first two hours, Cloudflare's engineers thought they were under attack. The reasons are a lesson in how partial rollouts disguise themselves.
The permissions change was rolling out gradually across the ClickHouse cluster. Every five minutes, the feature file was rebuilt by a query that might land on an updated node (bad file) or one not yet updated (good file). So the network would fail, recover when a good file replaced the bad one, then fail again. Once every node had the change, every file was bad, and the failure became steady.
Two coincidences sealed it. Cloudflare had recently been fending off very large attacks from the Aisuru botnet, and its status page, hosted outside Cloudflare, went down at the same time. The post-mortem says that was a coincidence, but it made the attack theory look confirmed.
Insight
A failure that comes and goes on a fixed beat is a strong hint that something periodic is involved: a cron job, a config refresh, a cache TTL. Attacks rarely keep time. If your alerting can overlay "config versions shipped" on the error graph, the five-minute rhythm is visible in seconds.
Cloudflare was mid-migration from its older proxy, FL, to the Rust rewrite, FL2. They failed differently, and the difference is worth more than the bug.
FL2, the new proxy
Customers see HTTP 5xx
Loud: everyone knows within minutes
Loader rejects the oversized file
Error is unwrapped, so the worker panics
FL, the old proxy
Sites that block low bot scores block real users (false positives)
Quiet: looks like a traffic change, not an outage
No crash
Bot scores are not generated correctly: every request scores zero
vs
Verdict: Neither is safe. The right behaviour for a bad optional module is to keep the last good file and raise an alarm, so traffic flows and scores stay meaningful.
The same bad feature file on Cloudflare's two proxies
FL2 failed closed and took traffic down. FL failed wrong, keeping traffic up while silently giving every request a score that meant "bot", which for customers with bot-blocking rules meant turning away humans. Customers who don't use bot scores weren't affected on FL at all. A quiet wrong answer is often worse than a loud error, because nobody comes to fix it.
The blast spread through dependencies. Turnstile, Cloudflare's CAPTCHA alternative, failed to load. Because Turnstile guards the dashboard login page, most customers couldn't log in to change anything. Workers KV's front end depended on the core proxy and threw 5xx errors. Access, the zero-trust login product, failed authentications until a bypass at 13:05. Even CDN latency rose, because debugging systems burned CPU enriching all the uncaught errors.
On 5 December 2025 at 08:47 UTC it happened again, smaller and faster. While rolling out WAF changes to protect customers from a newly disclosed React vulnerability, Cloudflare used its global killswitch to turn off an internal WAF test ruleset. The killswitch had a well-defined procedure, and it was followed. But Cloudflare had never before applied a killswitch to a rule with an "execute" action. On the older FL1 proxy, the code correctly skipped the rule, then tried to read the result of the rule it had skipped, and Lua threw a nil-value error. Requests returned HTTP 500 for about 25 minutes. Coverage of the post put the affected share at roughly 28% of Cloudflare's HTTP traffic, and only on FL1: FL2's type system ruled out that class of bug.
Put the two side by side and the pattern is hard to miss:
18 November
5 December
Trigger
Database permissions change
Killswitch applied to an internal WAF ruleset
Path to every machine
Feature file pushed every 5 minutes
Global configuration system, seconds
Latent defect
No handling for an oversized file (FL2)
Nil dereference on a never-used code path (FL1)
Duration of main impact
About 3 hours (11:20 to 14:30)
About 25 minutes
Neither trigger was a code deploy. Both were changes to data that the proxy reads at runtime, delivered through a path built to be fast, and both met code that had never seen that shape of input.
Cloudflare's own remediation list from the 18 November post is short and, I think, exactly right:
01
Treat Cloudflare-generated configuration files with the same validation as user-generated input.
02
Add more global kill switches for features.
03
Stop core dumps and error reports from overwhelming system resources.
04
Review failure modes for error conditions across every core proxy module.
Each of those covers a different gap, so it's worth seeing them as a stack. Switch a layer off below to see what gets through without it.
A bad file is stopped or contained. With all four in place, 18 November is a non-event: any one of them would have stopped or contained that file, and each also covers cases the others miss.
Generation-time sanity checkhandled
Without it: A file with twice as many entries as the last one, and more than the proxy can hold, is published as if it were normal. Build: validate generated config like user input; diff against the last version.
Staged rollout with bake timehandled
Without it: The file reaches every machine before anyone sees errors on the first few. Build: canary the file, not just the code.
Keep last known good on load failurehandled
Without it: The loader rejects the file and the worker panics, or serves wrong answers. Build: fail static: keep the previous file and alert.
Module-level kill switchhandled
Without it: Nobody can turn Bot Management off globally while traffic keeps flowing. Build: a tested global kill switch per feature.
Defences for generated config on its way to every proxy. Turn one off to see the gap it leaves.
The third layer is the one I'd build first in my own systems. A proxy that loads config at runtime has to have a written answer to "what if the new file is bad?", and for an optional module that answer should almost always be "keep using the old one and page someone". The config rollout design goes through how to build all four.
Why did a bug in one company's bot-scoring module take down shopping sites, AI chatbots and news sites at once? Because Cloudflare is in front of a startling share of them. W3Techs measured Cloudflare as the reverse proxy for 23.9% of all websites in July 2026, which is 83.9% of every site whose reverse proxy it can identify. In September 2026 its headline was that 25% of all websites now use Cloudflare as a CDN.
Cloudflare's outages are short compared with cloud providers' and very wide. Control Plane's tracker of major incidents since 2020 shows the difference in its medians:
Median duration of major outages per provider, January 2020 to November 2026 (Control Plane tracker)
The tracker's authors warn that counts reflect how well each provider documents incidents and aren't a reliability ranking. But the shape is real: an edge network that pushes config everywhere in seconds can also fix things everywhere in seconds, and it can break everything in seconds. The same tracker found that 55% of the 62 major outages it documented started with a configuration or deployment change, a software bug or automation, not with hardware. Cloudflare's 2026 entries fit: a bug in an Addressing API release that withdrew about 1,100 BYOIP prefixes on 20 February, and a cache change that reordered CNAME records in 1.1.1.1 answers on 8 January.