One Empty DNS Record: Inside the AWS us-east-1 Outage of October 2025

A step-by-step reading of AWS's own post-event summary for the 19–20 October 2025 us-east-1 outage: the DynamoDB DNS race condition, why EC2 stayed broken after DNS came back, how health checks removed healthy capacity, what AWS changed, and why one region can stall a quarter of the cloud.

Rahul Bisht

Founder, CrawlPilot

·
Nov 20, 2026
·
Verified Nov 15, 2026
·Engineering·
18 min read
·
One Empty DNS Record: Inside the AWS us-east-1 Outage of October 2025
Key takeaway
For about three hours, DynamoDB's own name in us-east-1 resolved to nothing, because two copies of an automation raced and one deleted the DNS plan the other had just applied. Fixing DNS took under three hours. The other twelve went to systems recovering from DynamoDB being gone: EC2's host leases, a network-state backlog, and load-balancer health checks that removed healthy capacity. The lesson isn't "DNS is fragile". It's that recovery is its own load event.

At 11:48 PM Pacific on Sunday 19 October 2025, asking DNS for dynamodb.us-east-1.amazonaws.com started returning an answer with no addresses in it. No error, no timeout, no stale IP: an empty record. Every client that needed a new connection to DynamoDB in AWS's busiest region, including a long list of AWS's own services, now had nowhere to send it.

AWS published a detailed post-event summary a few days later. It's one of the clearest incident write-ups a hyperscaler has released, and it rewards a slow read, because the interesting part isn't the bug. It's what happened in the twelve hours after the bug was fixed. Everything below comes from that summary unless another link says otherwise; where I add interpretation, I say so.

Three Outages Wearing One Name

AWS describes the event as running from 11:48 PM PDT on 19 October to 2:20 PM PDT on 20 October, with three distinct periods of impact. Read as one 15-hour outage, it looks like AWS took 15 hours to fix DNS. Read as three overlapping outages, each caused by the recovery of the one before it, it makes sense.

  1. Oct 19, 11:48 PM
    DynamoDB endpoint resolves to nothing

    The regional endpoint's DNS record is left empty. New connections to DynamoDB in us-east-1 fail.

    AWS summary
  2. Oct 20, 12:38 AM
    Cause located

    Engineers identify DynamoDB's DNS state as the source of the outage.

    AWS summary
  3. Oct 20, 2:25 AM
    DNS restored

    All DNS information is restored; cached empty answers expire by 2:40 AM.

    AWS summary
  4. Oct 20, 2:25 AM
    EC2 host leases stampede

    DropletWorkflow Manager tries to re-establish every lapsed host lease at once and falls into congestive collapse.

    AWS summary
  5. Oct 20, 5:30 AM
    Load balancer errors begin

    Network Load Balancer connection errors start as new instances launch without network state.

    AWS summary
  6. Oct 20, 9:36 AM
    Automatic failover switched off

    AWS disables NLB's automatic health-check failover, which restores healthy capacity.

    AWS summary
  7. Oct 20, 1:50 PM
    EC2 back to normal

    All EC2 APIs and instance launches operate normally.

    AWS summary
  8. Oct 20, 2:20 PM
    Event ends

    AWS marks the event over; some services still drain backlogs for hours.

    AWS summary
The us-east-1 event as AWS's post-event summary times it (all times PDT)

How DynamoDB Keeps Its Own Phone Book

DynamoDB in one region sits behind a large fleet of load balancers, and that fleet changes all the time as capacity is added, removed or fails. So nobody edits those records by hand. AWS's summary describes two automated components:

  • The DNS Planner watches load-balancer health and capacity and produces versioned plans: which load balancers each endpoint should point at, and with what weights.
  • Three DNS Enactors, one per Availability Zone, take the latest plan and apply it to Route 53 as a transaction. Three copies exist on purpose: if one zone's Enactor is down, the others keep DNS current.
DynamoDB's DNS automation, as designed
health signals1plan N, N+1, …2apply + clean up3resolve dynamodb.us-east-14Load balancershealth + capacityDNS Plannerwrites plan N3 Enactorsone per zoneRoute 53regional endpointAny clientincl. AWS services
  1. The Planner watches the health and capacity of DynamoDB's load balancers.
  2. It writes a new numbered plan whenever the right set of load balancers changes, which is often.
  3. Each Enactor independently picks up the newest plan, applies it to Route 53, and later cleans up plans that are far older than the one it applied.
  4. Clients resolve the endpoint name and get the IPs from whatever plan is live.

4 steps. Press Play to walk through them, or click any part of the diagram.

Planner writes numbered plans; three independent Enactors apply the newest plan to Route 53; clients resolve the endpoint and connect.

Every piece of this is reasonable. Redundant appliers, versioned plans, garbage collection of old versions: you'd draw the same diagram in an interview. The bug lives in the gaps between the steps.

The Race, Step by Step

Each Enactor checks that its plan is newer than the one already applied, but it does that check once, before it starts writing. AWS's summary says one Enactor was unusually slow that night, retrying updates on several endpoints. Step through what happened next.

How two correct Enactors produced an empty record
checked: 100 is newer ✓ (then stalls)1applies plan 1402writes plan 100 over 1403cleanup: delete plans ≪ 1404resolve → empty5Enactor Aslow, holding plan 100Enactor Bfast, holding plan 140Route 53 recorddynamodb.us-east-1Plan storeversions 100 … 140Clientsnew connections
  1. Enactor A picks up an older plan (illustrative number 100), confirms it is newer than what's live, then gets delayed retrying writes on other endpoints.
  2. Meanwhile the Planner keeps producing plans. Enactor B picks up a much newer one and applies it quickly. The record is correct.
  3. Enactor A finally writes its old plan over B's newer one. Its freshness check passed minutes ago, so nothing stops it.
  4. Enactor B runs cleanup and deletes plans many generations older than the one it applied, including the old plan Enactor A just made live.
  5. Deleting the live plan removes every IP from the regional endpoint. Every new connection to DynamoDB in us-east-1 fails to resolve.

5 steps. Press Play to walk through them, or click any part of the diagram.

Enactor A is delayed with an old plan; Enactor B applies a newer plan and cleans up old ones; A's stale write lands, then B's cleanup deletes the plan that is now live.

The plan numbers above are illustrative; AWS didn't publish them. The shape is from the summary: a delayed Enactor applied a much older plan, and the other Enactor's cleanup then deleted that plan because it was many generations behind the one it had just applied.

This is a textbook time-of-check to time-of-use race, and it has a textbook fix: make the freshness check part of the write. A conditional write ("apply plan 100 only if the live version is still below 100") would have refused Enactor A's stale write. So would a fencing token, which the job scheduler design uses for exactly the same reason. The cleanup job had the mirror-image flaw: "delete old plans" is safe only if it also asks "is this plan live right now?"

Insight
Redundancy turned a slow worker into a dangerous one. With a single Enactor, a delay would have meant stale DNS. With three acting independently and a cleanup that assumed it knew which plan was live, a delay produced a state none of them was designed to handle.

Why the Automation Couldn't Put It Back

The obvious question is why the system didn't just apply the next plan a minute later. AWS's answer: deleting the active plan left the system in an inconsistent state that blocked any Enactor from applying subsequent updates. The automation's own invariants were broken, so it stopped, and humans had to repair the record. Engineers identified DynamoDB's DNS state as the source at 12:38 AM, restored all DNS information by 2:25 AM, and the last cached empty answers expired by 2:40 AM.

That last detail matters. Even after the record was correct, clients that had cached the empty answer kept failing until the TTL ran out. AWS's early updates told customers to flush DNS caches if they still couldn't resolve the endpoint. Negative answers get cached too.

DNS Came Back at 2:25. EC2 Didn't.

Here's where most summaries of this outage stop, and where the useful part starts.

EC2 manages its physical servers (AWS calls them droplets) through a component called the DropletWorkflow Manager (DWFM). DWFM holds a lease on every host and renews it with periodic state checks. Those checks depend on DynamoDB. From 11:48 PM, every check failed, and leases quietly expired across the region. Running instances kept running. But a host without a lease can't be used for a new launch.

When DynamoDB came back at 2:25 AM, DWFM started re-establishing leases, for the whole region at once.

Congestive collapse in EC2's host manager
state checks work again1renew all leases2timeouts → requeue34:14 AM: throttle45:28 AM: all leases back5DynamoDBback at 2:25 AMDWFMhost lease managerLease workevery host at oncePhysical hostsno lease, no launchesEngineersthrottle + restart
  1. DynamoDB resolves again. DWFM's state checks can succeed, so it starts renewing leases.
  2. Every lease in the region lapsed in the same three hours, so every host needs renewing now. The work arrives as one wave.
  3. Renewals take long enough to time out. Timed-out work is requeued, so DWFM spends its capacity on attempts that never finish. This is congestive collapse: more load, less useful throughput.
  4. At 4:14 AM engineers throttle incoming work and selectively restart DWFM hosts, which clears the queues.
  5. By 5:28 AM every droplet lease in the region is re-established. Launches can finally find hosts.

5 steps. Press Play to walk through them, or click any part of the diagram.

DynamoDB returns; DWFM tries to renew every lapsed lease at once; work times out and is requeued, so the queue grows; throttling plus restarts clear it.

Congestive collapse is the failure mode where a system under load gets less done than it would at lower load, because it's spending its time on work that times out and gets retried. Retries are an accelerant, which is the point of retry budgets. The mitigations here were the classic ones: throttle admission so each piece of work can finish, and restart to drop the stuck queue.

Then a second backlog surfaced. EC2's Network Manager, which pushes network configuration to new instances, had its own queue of delayed state. From 6:21 AM new instances could launch but lacked connectivity until their network state arrived, and propagation times didn't return to normal until 10:36 AM.

Health Checks That Removed Healthy Capacity

The third outage is my favourite, because it was caused by a safety feature.

Network Load Balancer health-checks its targets and, when a zone looks unhealthy, can fail DNS away from that zone. During the backlog, new EC2 instances were added to NLB's health-check system before their network state had propagated. Their checks failed. The nodes were healthy; they just weren't reachable yet. Nodes flapped between healthy and unhealthy, were removed from DNS and returned to it, and the volume of flapping overloaded the health-check subsystem itself, which delayed checks further and triggered automatic zone failover. Failover removed capacity from multi-zone load balancers.

AWS spotted this at 6:52 AM and at 9:36 AM disabled automatic health-check failover for NLB, which brought the healthy capacity back. It re-enabled failover at 2:09 PM, when the event was nearly over.

When failover helps
  • Moves traffic away in seconds, no human needed
  • One zone is genuinely broken
  • Health checks measure the target, not the checker
  • Other zones have spare capacity
When failover hurts
  • Automation shrinks the fleet during the incident
  • Healthy nodes fail checks for an unrelated reason (missing network state)
  • The health-check system itself is overloaded
  • Removing capacity raises load on what's left

Verdict: Give fail-away automation a speed limit. AWS's fix is a velocity control on how much capacity one NLB can remove.

What automatic failover assumes, and what it met on 20 October

The general rule I take from this: any automation that removes capacity needs a ceiling on how much it may remove, and that ceiling should hold even when every signal says "remove more". A health check reports what the checker can see. When the checker is the thing that's sick, the report is noise.

What AWS Changed

The remediation list in the post-event summary maps neatly onto the three outages:

  1. 01
    DynamoDB: the DNS Planner and Enactor automation was disabled worldwide. AWS says it will fix the race condition and add protections against applying incorrect plans before turning it back on.
  2. 02
    NLB: a velocity control limits how much capacity a single load balancer can remove when health-check failures trigger zone failover.
  3. 03
    EC2: a new test suite exercises DWFM's recovery workflow, and data-propagation systems will rate-limit incoming work based on queue size.

Notice what's missing: nothing here is about DNS as a technology. Every fix is about automation that acts on stale state, recovery that arrives all at once, or protection that removes too much.

One Region, a Quarter of the Cloud

So far this is a story about one region. The reason it made front pages is concentration.

Share of worldwide cloud infrastructure services revenue, Q3 2026 (Synergy Research Group)

AWS still holds the largest share of a market that Synergy puts at $168 billion in Q3 2026 alone. And us-east-1 isn't just one region among many. AWS's own summary shows global features leaning on it: IAM console sign-in failed for IAM and Identity Center users in us-east-1, and root and federated users saw errors signing in from other regions. DynamoDB global table replicas elsewhere kept serving but lagged on replication to and from us-east-1. Then there are the thousands of SaaS products whose control planes happen to live in Virginia because it was AWS's first region and is still its default.

Estimates of the cost vary by an order of magnitude, which tells you how hard this is to measure. Cyber-risk modeller CyberCube put insured losses at $38 million to $581 million; Parametrix estimated losses to US companies at $500 million to $650 million. Insurance Business notes that cyber business-interruption waiting periods of typically eight to 12 hours kept insured claims down, so most of the economic loss stayed with the businesses that went dark.

And it wasn't the last us-east-1 incident. Control Plane's outage tracker lists a May 7–8, 2026 thermal event in one us-east-1 zone that took 21.5 hours to reach majority recovery. Different cause, same address.

Note
"Control the world" overstates it, but not by much. When a quarter of rented compute sits with one company and that company's oldest region holds pieces of global sign-in, a race condition in a DNS updater becomes a problem for banks, games, airlines and doorbells at the same time.

What to Change in Your Own Stack

You can't fix AWS's Enactors. You can stop building the same traps.

Check yourself

Q1Why didn't DynamoDB's automation repair the empty record by applying the next plan?

Q2What actually removed healthy capacity from Network Load Balancers?

Cheat sheet
Lessons from the us-east-1 outage

Writes on shared state

  • Make the freshness check part of the write: conditional writes or fencing tokens, not check-then-act.
  • Cleanup jobs must ask whether a version is live before deleting it.
  • Have a path to repair state when the automation's own invariants break.

Recovery is a load event

  • Expect every lapsed lease, cache and session to come back at the same moment.
  • Throttle admission by queue size so work can finish instead of timing out.
  • Test the recovery workflow, not only the steady state (AWS added a DWFM test suite).

Automation that removes capacity

  • Cap how much capacity failover can remove, however loud the signals.
  • Treat an overloaded health checker as missing data, not as bad nodes.

Your dependency on one region

  • Prefer regional endpoints (such as regional STS) over global ones.
  • Pre-provision capacity elsewhere so you don't need launches during an incident.
  • Know which SaaS vendors you rely on run in us-east-1.

The zone-failure post covers static stability, the idea that you shouldn't need to launch anything to survive a failure. That was the difference this time between customers who rode it out and customers who waited for EC2 launches until the afternoon.

Sources