The Safety Check Passed, Then the Edge Crashed: Azure Front Door, 29 October

Microsoft's post-incident review of the 29 October 2025 Azure Front Door outage, read step by step: two valid configuration changes across two build versions, a crash that arrived five minutes after the health check, a poisoned last-known-good snapshot, and why recovery took eight hours. Plus the 2026 pattern and what it means for anything you deploy globally.

Rahul Bisht

Founder, CrawlPilot

·
Dec 2, 2026
·
Verified Nov 15, 2026
·Engineering·
15 min read
·
The Safety Check Passed, Then the Edge Crashed: Azure Front Door, 29 October
Key takeaway
Azure Front Door's deployment safety did what it was built to do: watch each stage and stop if health drops. The crash came five minutes after the check, by which time the bad config was on most of the edge and had been saved as the rollback point. A health gate that looks too soon is not a gate. Neither is a rollback snapshot that copies whatever is currently running.

Microsoft's protection system for Azure Front Door (AFD) watches each stage of a configuration rollout and halts it if the edge gets sick. On 29 October 2025 it worked as designed. It checked, saw healthy servers, and let the change through. About five minutes later the servers started crashing. By then the configuration had reached most of the world's edge sites and had been recorded as the Last Known Good version, the one you'd roll back to.

This post reads Microsoft's final post-incident review (PIR), tracking ID YKYN-BWZ, step by step. Microsoft publishes PIRs to affected customers and on its status history; I'm linking a copy that a customer status page reposted because it's publicly readable without a login. Where I add interpretation, I say so.

Eight Hours, Twenty-Four Minutes

The PIR puts customer impact between 15:41 UTC on 29 October and 00:05 UTC on 30 October 2025. The first thing to notice in the timeline is how little time separated "change introduced" from "everywhere".

  1. 15:35
    Incompatible metadata introduced

    A sequence of customer config changes across two control-plane builds produces metadata the data plane can't handle.

    Microsoft PIR
  2. 15:39
    Most edge sites reached

    Propagation reaches a majority of edge sites; the Last Known Good snapshot is updated with the bad metadata.

    Microsoft PIR
  3. 15:41
    Customer impact begins

    Edge processes start crashing; timeouts and DNS resolution failures follow.

    Microsoft PIR
  4. 15:43
    Protection system blocks changes

    The configuration protection system activates and blocks new and in-flight changes.

    Microsoft PIR
  5. 15:45
    Global

    Impact has replicated across all edge sites globally.

    Microsoft PIR
  6. 17:10
    Manual surgery on the snapshot

    Engineers begin editing the Last Known Good snapshot by hand to remove the problem configuration.

    Microsoft PIR
  7. 17:50
    Clean snapshot at every edge

    The updated snapshot is available to all edge sites.

    Microsoft PIR
  8. 18:30 – 20:20
    Traffic rebalanced

    AFD DNS recovers; traffic is rebalanced by hand, then by automatic traffic management from 20:20.

    Microsoft PIR
  9. 00:05 (Oct 30)
    Mitigated

    Microsoft confirms impact mitigated.

    Microsoft PIR
Azure Front Door, 29–30 October 2025, from Microsoft's PIR (all times UTC)

Four minutes from introduction to a majority of the edge, ten to global. Two hours and fifteen minutes from first alert to a clean config at every site. Then more than six hours to bring traffic back without knocking the edge over again. Fast propagation, slow recovery, and the asymmetry between them is the real subject of this post.

What Sits Behind Azure Front Door

AFD is Microsoft's global Layer 7 edge: a CDN, TLS terminator, web application firewall and global load balancer in one. Customers put their own sites behind it, and so does Microsoft. That's why a failure in AFD looked, from outside, like "Azure is down", "Microsoft 365 is down" and "the Azure Portal is down" all at once.

What AFD sits in front of
AFD EDGE SITEresolve namenearest siteUsersanywhere in the worldAFD DNSruns on edge sitesAFD edgeTLS, WAF, routingMicrosoft servicesPortal, M365, EntraCustomer appsyour origins
Users reach Microsoft's first-party services and thousands of customer apps through the same AFD edge, which also runs the DNS that resolves them.

Note the zone. The PIR says the crash also hit AFD's internal DNS service, which runs on the same edge sites. When the edge process fell over, the thing that tells clients which edge to use fell over with it. Users didn't just get errors; they got intermittent DNS failures, which every client handles differently and many cache.

This also wasn't AFD's first bad week. ThousandEyes notes that three weeks earlier, on 9 October, AFD had a different outage attributed to erroneous metadata propagating through the system.

Two Valid Changes, Two Builds, One Incompatible Result

Here is the part of the PIR that I think every platform team should read twice. Nothing invalid was submitted.

How valid changes produced a crash
change 11change 2122loads fine, then crashes ~5 min later3Customer changesvalid, non-maliciousControl planebuild Aold versionControl planebuild Bnew versionConfig metadataincompatible combinationEdge data planelatent bug
  1. A customer makes a sequence of ordinary configuration changes. During a control-plane rollout, some are handled by the old build and some by the new one.
  2. Each build writes metadata that is correct by its own rules. Together they form a combination no test had produced, because pre-production didn't test features across different control-plane builds.
  3. The edge loads the metadata without complaint. A latent bug crashes the data plane later, during asynchronous processing. By then the health check for that stage has already passed.

3 steps. Press Play to walk through them, or click any part of the diagram.

Customer changes processed by two control-plane builds → incompatible metadata → passes per-stage health checks → crashes the data plane asynchronously minutes later.

Two lessons hide in this picture. First, version skew is a configuration input. During any rolling upgrade of a control plane, two versions are writing to the same store, and the data plane has to handle the union of what both can produce. Microsoft's repair list includes expanding backward-compatibility testing across control-plane builds, which is the direct fix.

Second, load-time validation isn't the same as runtime safety. The edge accepted the metadata. The crash lived in a code path that ran later, on a timer or a queue. If your validation step is "parse it and see if the process stays up", you've only checked the parser.

The Five-Minute Gap in the Safety Net

This is the heart of the incident. AFD's safety system checked health at each stage of the rollout. Each check happened before the crash, because the crash was asynchronous and arrived about five minutes after the config landed.

The gate as it ran
  • Misses failures that surface minutes later
  • Lets each stage advance in about a minute
  • Promotes the config to Last Known Good on the way
  • Push to a stage
  • Check health right away
  • Healthy, so continue
The gate it needed
  • Catches asynchronous crashes before the next stage
  • Keeps the rollback point clean
  • Push to a stage
  • Process the config fully and synchronously first
  • Bake long enough for delayed failures
  • Only then promote

Verdict: Microsoft's completed repairs do both: fully synchronous processing before production stages, and more rollout stages with longer bake time.

What the stage gate checked, and what it needed to check

And then the detail that turned a bad rollout into a long outage: the Last Known Good snapshot was updated as the config propagated. By 15:39, before customer impact even began, the rollback target already contained the bad metadata. "Roll back to LKG" would have rolled forward into the crash.

Warning
A rollback point is only useful if it's older than the problem. If "last known good" means "last version that loaded", it will capture any bug that shows up after loading. Promote a version to LKG only after it has baked in production for longer than your slowest known failure takes to appear.

Why "Just Roll Back" Wasn't Available

With the latest LKG poisoned, Microsoft had two options: roll back to an older snapshot, or fix the current one. The PIR says it chose not to roll back to earlier LKG versions for stability reasons. My reading, not Microsoft's words: an older snapshot would have undone every legitimate customer change made since, at a time when nobody could say which of those changes were safe, and swapping a large config on every edge site at once is itself a risky operation.

So the recovery went like this:

  1. 01
    17:10 – engineers started editing the latest LKG by hand to remove the problem configuration.
  2. 02
    17:30 – all customer configuration changes were blocked from reaching the data plane, so nothing new could interfere. (Microsoft kept AFD config changes blocked at the ARM level until 5 November, a week after.)
  3. 03
    17:40 to 17:50 – the cleaned snapshot was deployed to every edge site.
  4. 04
    18:30 onward – every edge site reloaded its configuration, and traffic was rebalanced gradually, first by hand and then, from 20:20, automatically.

That last step is why the tail lasted until after midnight. An edge site that has just come back has cold caches and empty connection pools. Send it a full share of global traffic at once and it falls over again. AWS learned the same thing the week before: recovery is a load event, and it has to be metered.

Who Failed Over and Who Didn't

The PIR has one sentence that every architect should keep: Microsoft Entra, Intune and Azure AD B2C failed over successfully, while some Azure Portal components without fallback strategies stayed degraded. The Portal itself was moved away from AFD at 17:26 UTC.

Same company, same edge, same outage. The difference was whether a team had built and tested a path that didn't go through AFD. Among Microsoft's completed repairs: migrating critical first-party infrastructure to an active-active setup with fail-away. In other words, making the Entra outcome the default for everything else too.

What Microsoft Committed To

The repair items in the PIR, as published, with Microsoft's own completion estimates. I haven't verified which of the estimated items have since shipped.

AreaRepairStatus in the PIR
Deployment safetyFix the control-plane incompatibility and the data-plane bugCompleted
Deployment safetyFully synchronous processing before production stagesCompleted
Deployment safetyMore rollout stages, longer bake timeCompleted
Deployment safetyDecouple config processing from traffic-serving instancesEstimated January 2026
Deployment safetyCut propagation time from 45 to about 15 minutesEstimated January 2026
Deployment safetyBackward-compatibility testing across control-plane buildsEstimated February 2026
Blast radiusActive-active with fail-away for critical first-party servicesCompleted
Blast radius"Micro cell" isolation of customer configuration and trafficEstimated June 2026
RecoveryRestore configurations within one hour using local cachingCompleted
RecoveryBring recovery down to about 10 minutesEstimated March 2026

Two items stand out. Decoupling config processing from the traffic-serving process means a crash while parsing config no longer takes the proxy down with it, the same separation Cloudflare's feature-file outage argues for. And micro cells are the cell-based architecture from the zone-failure post applied to config: one customer's bad combination should only be able to break that customer's cell.

The Microsoft Layer Under Everyone Else

Microsoft held 19% of worldwide cloud infrastructure revenue in Q3 2026 according to Synergy Research Group, second only to Amazon. Add Microsoft 365, Entra sign-in and Xbox, and an edge failure at Microsoft reaches a wider slice of daily life than its cloud share alone suggests.

The 2026 record suggests AFD's lesson generalises across Azure. Control Plane's outage tracker lists Azure's 2026 incidents with causes paraphrased from Microsoft's own reports, and the change-driven ones rhyme with 29 October:

  1. Feb 2–3, 2026
    Policy remediation breaks storage access

    A policy remediation wrongly disabled anonymous storage access across multiple regions.

    Control Plane
  2. Apr 24, 2026
    Control-plane lock contention

    Lock contention in a networking control-plane partition in East US.

    Control Plane
  3. Jul 23, 2026
    Repair tooling defect

    A defect in automated repair tooling in West US.

    Control Plane
  4. Sep 30 – Oct 1, 2026
    Updates overload the control plane

    Resumed gateway OS updates plus a recent change overloaded the networking control plane across regions.

    Control Plane
Change-driven Azure incidents in 2026, as summarised by Control Plane's tracker

Across all four providers it tracks, Control Plane found 55% of major outages since 2020 began with a configuration or deployment change, a software bug or automation. The edge rarely dies on its own. Someone, or something, changes it.

A Checklist for Anything You Deploy Globally

Check yourself

Q1Why couldn't Microsoft simply roll back to the Last Known Good configuration?

Q2Why did the per-stage health checks pass?

Cheat sheet
Azure Front Door, 29 October 2025

What happened

  • Valid customer changes across two control-plane builds made incompatible metadata.
  • The edge crashed about 5 minutes after loading it, after per-stage health checks had passed.
  • The Last Known Good snapshot already held the bad metadata, so there was no clean rollback.

Gates that actually gate

  • Process config fully and synchronously before calling a stage healthy.
  • Bake each stage longer than your slowest known failure takes to show.
  • Promote to last-known-good only after bake, never on load.

Version skew

  • During a control-plane rollout, the data plane must handle output from both versions.
  • Test feature combinations across adjacent builds, not just the newest.

Recovery and blast radius

  • Keep config parsing out of the traffic-serving process.
  • Rebalance traffic onto recovering sites gradually.
  • Give critical services a tested path that doesn't go through the edge (Entra did; parts of the Portal didn't).

If you run anything that pushes configuration to many machines (feature flags, WAF rules, routing tables, ML feature files), the config rollout design builds these gates into one system.

Sources