The Query That Didn't Name Its Database: Cloudflare's 18 November Outage

How a database permissions change doubled a Bot Management feature file, crashed Cloudflare's proxy on every machine that loaded it, and flickered like a DDoS for an hour. Plus the 5 December follow-up, the pattern both share, and what a fifth of the web sitting behind one proxy means for you.

Rahul Bisht

Founder, CrawlPilot

·
Nov 24, 2026
·
Verified Nov 15, 2026
·Engineering·
14 min read
·
The Query That Didn't Name Its Database: Cloudflare's 18 November Outage
Key takeaway
Cloudflare's worst outage since 2019 started with a permissions change that made one SQL query return twice as many rows. The file built from those rows was pushed to every proxy within minutes, broke a hard size limit, and crashed the new proxy outright. Seventeen days later a different change, pushed through the same fast global path, tripped a years-old bug in the old proxy. The shared lesson: configuration your own systems generate is still untrusted input, and the faster you can ship it everywhere, the more it needs gates.

Two numbers explain most of this outage: about 60 and 200. Cloudflare's bot-scoring model used about 60 features, and its newest proxy reserved memory for at most 200. On 18 November 2025 the file listing those features suddenly doubled in size and went past the limit, and the proxy on every machine that loaded it crashed. The SQL that built the file hadn't changed. It filtered column metadata by table name and never said which database, which was harmless until a permissions change at 11:05 UTC let it see a second one.

What follows is a reading of Cloudflare's own post-mortem, written by CEO Matthew Prince the same day, plus its 5 December follow-up. Cloudflare is explicit that the issue was "not caused, directly or indirectly, by a cyber attack or malicious activity of any kind."

Six Hours, Minute by Minute

  1. 11:05
    Permissions change deployed

    Database access controls on the ClickHouse cluster change so users can see metadata for the underlying r0 tables.

    Cloudflare
  2. 11:20
    Core traffic starts failing

    Users see Cloudflare error pages as bad feature files reach the network.

    Cloudflare
  3. 11:32 – 13:05
    Chasing Workers KV

    The investigation focuses on degraded Workers KV and suspects an attack.

    Cloudflare
  4. 13:05
    Bypasses reduce impact

    Workers KV and Access are routed around the failing proxy path.

    Cloudflare
  5. 14:24
    Bad file generation stopped

    Creation and propagation of new Bot Management files is halted; a known-good file is tested.

    Cloudflare
  6. 14:30
    Main impact resolved

    A correct file is deployed globally and core traffic largely recovers.

    Cloudflare
  7. 17:06
    All services restored

    Remaining services that entered bad states are restarted individually.

    Cloudflare
18 November 2025, from Cloudflare's post-mortem (all times UTC)

From a Permissions Change to a Panic

Cloudflare's Bot Management gives every request a bot score using a machine-learning model. The model's inputs, called features, are listed in a feature file that is regenerated every five minutes from a ClickHouse query and pushed to every machine in the network, so the model can keep up with new bot behaviour. Fast distribution is the feature, not a bug.

Step through how that pipeline turned a permissions change into a crash.

How one permissions change reached every proxy
wider visibility1same columns, twice2≈2× features3loads new file4panic → 5xx5Permission change11:05, can now see r0ClickHouse queryfilters by table nameFeature filerebuilt every 5 minGlobal pushevery machineFL2 proxylimit 200 featuresVisitorssee 5xx pages
  1. Cloudflare changes ClickHouse access controls so users can see metadata for the r0 tables that sit under the distributed tables in the default database. A reasonable, deliberate change.
  2. The feature-generation query reads column names from system.columns filtered by table name only. Now it sees the matching columns in both default and r0, and returns roughly twice the rows.
  3. The duplicated rows double the feature file. Nothing in the pipeline checks that the number of features is plausible.
  4. The file is pushed to every machine on the network within minutes, as designed.
  5. The proxy preallocates memory for up to 200 features, far above the roughly 60 in use. The oversized file exceeds it, the error isn't handled, and the FL2 worker thread panics. Requests that touch Bot Management get 5xx errors.

5 steps. Press Play to walk through them, or click any part of the diagram.

Permissions change → metadata query returns duplicate rows → feature file doubles → pushed network-wide → proxy hits its 200-feature limit and panics.

The panic message Cloudflare quoted is the most-shared line of the post-mortem: called Result::unwrap() on an Err value. It's tempting to read it as "Rust's unwrap took down the internet". I think that's the wrong lesson. The limit was deliberate, a preallocation for performance on the hottest path in the company. The loader correctly returned an error when the file broke the limit. What was missing was a decision about what the proxy should do with a bad config file: keep the old one, skip the module, or die. Without that decision written down, unwrap() made it by default.

Why It Flickered Like an Attack

For the first two hours, Cloudflare's engineers thought they were under attack. The reasons are a lesson in how partial rollouts disguise themselves.

The permissions change was rolling out gradually across the ClickHouse cluster. Every five minutes, the feature file was rebuilt by a query that might land on an updated node (bad file) or one not yet updated (good file). So the network would fail, recover when a good file replaced the bad one, then fail again. Once every node had the change, every file was bad, and the failure became steady.

Two coincidences sealed it. Cloudflare had recently been fending off very large attacks from the Aisuru botnet, and its status page, hosted outside Cloudflare, went down at the same time. The post-mortem says that was a coincidence, but it made the attack theory look confirmed.

Insight
A failure that comes and goes on a fixed beat is a strong hint that something periodic is involved: a cron job, a config refresh, a cache TTL. Attacks rarely keep time. If your alerting can overlay "config versions shipped" on the error graph, the five-minute rhythm is visible in seconds.

Two Proxies, Two Different Failures

Cloudflare was mid-migration from its older proxy, FL, to the Rust rewrite, FL2. They failed differently, and the difference is worth more than the bug.

FL2, the new proxy
  • Customers see HTTP 5xx
  • Loud: everyone knows within minutes
  • Loader rejects the oversized file
  • Error is unwrapped, so the worker panics
FL, the old proxy
  • Sites that block low bot scores block real users (false positives)
  • Quiet: looks like a traffic change, not an outage
  • No crash
  • Bot scores are not generated correctly: every request scores zero

Verdict: Neither is safe. The right behaviour for a bad optional module is to keep the last good file and raise an alarm, so traffic flows and scores stay meaningful.

The same bad feature file on Cloudflare's two proxies

FL2 failed closed and took traffic down. FL failed wrong, keeping traffic up while silently giving every request a score that meant "bot", which for customers with bot-blocking rules meant turning away humans. Customers who don't use bot scores weren't affected on FL at all. A quiet wrong answer is often worse than a loud error, because nobody comes to fix it.

The blast spread through dependencies. Turnstile, Cloudflare's CAPTCHA alternative, failed to load. Because Turnstile guards the dashboard login page, most customers couldn't log in to change anything. Workers KV's front end depended on the core proxy and threw 5xx errors. Access, the zero-trust login product, failed authentications until a bypass at 13:05. Even CDN latency rose, because debugging systems burned CPU enriching all the uncaught errors.

Seventeen Days Later, a Different Change Down the Same Path

On 5 December 2025 at 08:47 UTC it happened again, smaller and faster. While rolling out WAF changes to protect customers from a newly disclosed React vulnerability, Cloudflare used its global killswitch to turn off an internal WAF test ruleset. The killswitch had a well-defined procedure, and it was followed. But Cloudflare had never before applied a killswitch to a rule with an "execute" action. On the older FL1 proxy, the code correctly skipped the rule, then tried to read the result of the rule it had skipped, and Lua threw a nil-value error. Requests returned HTTP 500 for about 25 minutes. Coverage of the post put the affected share at roughly 28% of Cloudflare's HTTP traffic, and only on FL1: FL2's type system ruled out that class of bug.

Put the two side by side and the pattern is hard to miss:

18 November5 December
TriggerDatabase permissions changeKillswitch applied to an internal WAF ruleset
Path to every machineFeature file pushed every 5 minutesGlobal configuration system, seconds
Latent defectNo handling for an oversized file (FL2)Nil dereference on a never-used code path (FL1)
Duration of main impactAbout 3 hours (11:20 to 14:30)About 25 minutes

Neither trigger was a code deploy. Both were changes to data that the proxy reads at runtime, delivered through a path built to be fast, and both met code that had never seen that shape of input.

The Fix Isn't "Be More Careful"

Cloudflare's own remediation list from the 18 November post is short and, I think, exactly right:

  1. 01
    Treat Cloudflare-generated configuration files with the same validation as user-generated input.
  2. 02
    Add more global kill switches for features.
  3. 03
    Stop core dumps and error reports from overwhelming system resources.
  4. 04
    Review failure modes for error conditions across every core proxy module.

Each of those covers a different gap, so it's worth seeing them as a stack. Switch a layer off below to see what gets through without it.

A bad file is stopped or contained. With all four in place, 18 November is a non-event: any one of them would have stopped or contained that file, and each also covers cases the others miss.

  1. Generation-time sanity checkhandled

    Without it: A file with twice as many entries as the last one, and more than the proxy can hold, is published as if it were normal. Build: validate generated config like user input; diff against the last version.

  2. Staged rollout with bake timehandled

    Without it: The file reaches every machine before anyone sees errors on the first few. Build: canary the file, not just the code.

  3. Keep last known good on load failurehandled

    Without it: The loader rejects the file and the worker panics, or serves wrong answers. Build: fail static: keep the previous file and alert.

  4. Module-level kill switchhandled

    Without it: Nobody can turn Bot Management off globally while traffic keeps flowing. Build: a tested global kill switch per feature.

Defences for generated config on its way to every proxy. Turn one off to see the gap it leaves.

The third layer is the one I'd build first in my own systems. A proxy that loads config at runtime has to have a written answer to "what if the new file is bad?", and for an optional module that answer should almost always be "keep using the old one and page someone". The config rollout design goes through how to build all four.

A Fifth of the Web Behind One Proxy

Why did a bug in one company's bot-scoring module take down shopping sites, AI chatbots and news sites at once? Because Cloudflare is in front of a startling share of them. W3Techs measured Cloudflare as the reverse proxy for 23.9% of all websites in July 2026, which is 83.9% of every site whose reverse proxy it can identify. In September 2026 its headline was that 25% of all websites now use Cloudflare as a CDN.

Cloudflare's outages are short compared with cloud providers' and very wide. Control Plane's tracker of major incidents since 2020 shows the difference in its medians:

Median duration of major outages per provider, January 2020 to November 2026 (Control Plane tracker)

The tracker's authors warn that counts reflect how well each provider documents incidents and aren't a reliability ranking. But the shape is real: an edge network that pushes config everywhere in seconds can also fix things everywhere in seconds, and it can break everything in seconds. The same tracker found that 55% of the 62 major outages it documented started with a configuration or deployment change, a software bug or automation, not with hardware. Cloudflare's 2026 entries fit: a bug in an Addressing API release that withdrew about 1,100 BYOIP prefixes on 20 February, and a cache change that reordered CNAME records in 1.1.1.1 answers on 8 January.

If Your Site Sits Behind Cloudflare

Cheat sheet
Cloudflare's November and December outages

What happened

  • 18 Nov: a permissions change doubled a generated feature file; FL2 panicked on its 200-feature limit.
  • 5 Dec: a killswitch on an 'execute' rule hit a years-old nil bug in FL1 for about 25 minutes.
  • Both were runtime config, pushed globally through a fast path.

For your own config pipelines

  • Validate generated config like user input: size, counts, diff against the last version.
  • Roll config out in stages with bake time, exactly like code.
  • On a bad file, keep the last known good one and alert; don't crash and don't serve wrong answers.

If you depend on Cloudflare

  • Host your status page and incident comms off the provider you're reporting on.
  • Know which of your own logins sit behind Turnstile or Access.
  • Decide in advance whether bot-score rules should fail open or closed.

Reading an incident live

  • A failure that comes and goes on a fixed beat points at something periodic, not an attacker.
  • Overlay config versions on error graphs.

Sources