Your browser cannot save progress right now. You can still investigate; keep this tab open or enable local storage to save.
This case has been updated. Compatible choices are kept; revisit the evidence before drawing your conclusion.
01 · The briefing
Begin with the question.
Amazon DynamoDB is a managed database used by many AWS services and customers. On the night of 19 October 2025 in Pacific time, the public regional endpoint of DynamoDB in the US East (N. Virginia) region, us-east-1, began failing DNS resolution, and clients could not connect from 11:48 PM PDT. The disruption did not stop there. Amazon EC2 could not complete many instance launches, Network Load Balancers saw connection errors, and other services that depend on DynamoDB, including Lambda, AWS STS and console sign-in for IAM users, were affected. Amazon's post-event summary says the event ended at 2:20 PM PDT on 20 October and describes three distinct periods of impact; the last, failed EC2 launches, ran from 2:25 AM to 10:36 AM, with connectivity problems on new instances resolved by 1:50 PM. The first task for responders was to find the source, and Amazon says its engineers had identified DynamoDB's DNS state by 12:38 AM. This investigation separates the DNS mechanism that emptied the record from the knock-on effects that followed. The record used here is Amazon's own post-event summary, with one news report for the public timeline. Keep the question concrete: which sequence of updates left the endpoint with no addresses, what the system's checks failed to prevent, and why recovery of dependent services took longer than the DNS fix.
The timeline
At 11:48 PM PDT, DynamoDB's public endpoint in us-east-1 begins failing DNS resolution, and clients cannot establish new connections.
By 12:38 AM PDT, engineers identify DynamoDB's DNS state as the source of the outage.
At 2:25 AM PDT, DNS information is restored, and cached records expire by 2:40 AM.
EC2 lease recovery stalls. Engineers restart DWFM hosts at 4:14 AM, and leases with all droplets in the region are re-established by 5:28 AM.
From 5:30 AM to 2:09 PM PDT, some Network Load Balancer customers see connection errors caused by health check failures. Automatic failover is disabled at 9:36 AM.
EC2 launches fully recover at 1:50 PM PDT, and the event ends at 2:20 PM PDT.
A researched reconstruction for learning. These reasoning steps simplify a complex record; no score can capture the full history.
02 · The evidence
What does the record show?
Read each exhibit before settling on an explanation. The source links below every exhibit lead to the record behind it.
Exhibits reviewed 0 of 5
Timing and reach of the DynamoDB failure
Amazon's post-event summary says that between 11:48 PM PDT on 19 October and 2:40 AM PDT on 20 October, customers saw increased DynamoDB API error rates in us-east-1, because clients could not establish new connections. Customers using global tables could still reach replicas in other regions but experienced replication lag. The summary names a latent race condition in the DynamoDB DNS management system as the root cause. It establishes the period, the symptom and the cause as Amazon states them. It is the operator's own account and gives no independent measurements of error rates.
Source record s1aws.amazon.com
How the DNS system is split
The summary describes two independent components. The DNS Planner monitors the health and capacity of load balancers and produces a new plan of load balancers and weights for each endpoint. The DNS Enactor applies a plan to Amazon Route 53 with a transaction, and three Enactors run redundantly in three Availability Zones. Before applying a plan, an Enactor makes a one-time check that its plan is newer than the one already applied. It establishes the design, including the redundancy and the freshness check. It does not include code, so the timing logic is described only at this level.
Source record s1aws.amazon.com
The race that left the record empty
Amazon says one Enactor was unusually delayed while retrying endpoint updates. Meanwhile the Planner produced newer plans, and a second Enactor applied one of them quickly and then began clean-up, which deletes plans much older than the one just applied. The delayed Enactor then wrote its much older plan, overwriting the newer one, because its freshness check had gone stale. The clean-up then deleted the older plan, which was now the active one, and all IP addresses for the regional endpoint were removed. The summary establishes this sequence. It is reconstructed from a written summary, not from logs.
Source record s1aws.amazon.com
EC2 leases and the backlog after recovery
Amazon's summary says that DWFM hosts keep leases with droplets, the physical servers that host EC2 instances, and that the leases began to time out during the DynamoDB event because the state checks depended on DynamoDB. After DynamoDB recovered at 2:25 AM, re-establishing leases across the fleet overwhelmed DWFM until engineers throttled work and restarted hosts at 4:14 AM. Network Manager then worked through a backlog of network state propagation until 10:36 AM. It establishes the dependency chain Amazon describes. It does not show how many launches failed in total.
Source record s1aws.amazon.com
A news report on the DNS attribution
The Verge, updated on 20 October 2025, reports that Amazon attributed the outage to resolution issues in its domain name system, and that the AWS status dashboard first reported issues in US-EAST-1 at 3:11 AM ET on Monday. The article also reports that some platforms were back online about four hours after Amazon's first report. It establishes that Amazon publicly named DNS as the cause, and it gives a public timestamp. It does not describe the Enactor race or the DNS record, so it confirms the category of cause, not the mechanism.
Source record s2theverge.com
03 · Your reasoning
Follow the strongest explanation.
Choose an interpretation, then compare it with the evidence. You can revise your choice. The goal is understanding, not a perfect score.
04 · The findings
The chain behind the outcome.
Your notebook
Case report
Next in engineering & technology: A computer that kept restarting
The empty record was a timing failure inside an automated system rather than a single bad change. DynamoDB's DNS management split planning from enacting, and three redundant Enactors applied plans to Route 53 behind a freshness check that ran once. One Enactor fell far behind while the Planner kept producing newer plans, and a second Enactor applied one and then cleaned up. When the delayed Enactor wrote its older plan, the check it had passed was no longer true, so the older plan overwrote the newer one. The clean-up then deleted that older plan, which was now the active one, and every address for the regional endpoint was removed. Recovery needed operators to intervene. DNS information was restored at 2:25 AM Pacific time, and cached records expired by 2:40 AM. The knock-on effects lasted longer. EC2 leases timed out while DynamoDB was down, and re-establishing them overwhelmed DWFM until hosts were restarted. NLB health checks removed capacity until automatic failover was disabled. Amazon says it disabled the DNS Planner and Enactor automation worldwide and will fix the race before turning it back on. The lesson is that a check made long before a write does not guarantee the state at the moment of writing.
Contributing causes and conditions
- DynamoDB's DNS automation was split into a Planner that produces versioned plans and Enactors that apply them to Route 53.
- One Enactor was delayed while retrying endpoint updates, and in the meantime newer plans were produced.
- A freshness check ran once before application, so a long-delayed Enactor could write an old plan after it had become stale.
- Clean-up deleted the older plan that had become active, which removed every IP address from the regional endpoint and required manual repair.
- EC2 lease checks depended on DynamoDB, so EC2 launch recovery lagged until the lease backlog was cleared.
- Health-check failover in NLB removed capacity during recovery, causing connection errors until automatic failover was disabled at 9:36 AM.
OutcomeAWS us-east-1 outage (2025)
Take the method with you
- A safety check that runs once, early, can become stale if the work it guards is slow, so checks should be made at the point of the write.
- Automation that deletes old state needs its own guard, because clean-up that trusts the active state can remove the only working version.
- When a foundational dependency fails, its recovery can depend on the services that broke, so recovery paths should not rely on the failed component.
You can read findings at any time. Exploring every reasoning step marks this case as complete in your notebook.
The open record
Sources & further reading
Read the original record and its scope before relying on a conclusion. Source references support the exhibit summaries; Faultory’s reasoning prompts are editorial reconstructions.
- s1
- s2
Written and curated for Faultory by Yunus Emre Vurgun, with AI-assisted research and drafting. Sources and interpretations can need correction. Read our editorial standards or report an issue.
From the journal
Field notes that use this case
Reading a post-incident review and what it leaves outA company's own post-incident review can be accurate and still incomplete. Learn to check who wrote it, when, in which time zone, and what it cannot show.
Trust in the update pipeline: why one file reaches everyoneSoftware reaches users through tests, staging and review. Four cases show what those steps assume, and why one bad update can spread at once.
Keep investigating
