us-east-1 goes down and takes a large chunk of the internet with it

AWS's us-east-1 region suffered a multi-hour outage yesterday that affected a very large number of services and, through them, a very large fraction of consumer internet applications.

AWS's public summary attributes the trigger to a latent race condition in the automation that manages DynamoDB's DNS records, which resulted in an empty record set for a regional endpoint and no automatic recovery path.

the cascade#

The failure sequence is the interesting part.

DynamoDB's endpoint became unresolvable. That alone would be bad. What made it a regional event is that an enormous number of AWS's own services use DynamoDB internally. The EC2 instance launch path, IAM's control plane, Lambda's invocation machinery, and dozens of others depend on it.

So the failure propagated: DynamoDB down means new EC2 instances cannot launch, which means autoscaling cannot replace failing capacity, which means load shedding, which means more failures. Network Load Balancer health checks destabilized. The recovery itself was slowed by the backlog of queued work that had accumulated.

This is textbook metastable failure: a system that is stable under normal load and stable under no load, but which, once pushed past a threshold, sustains its own failure through retry amplification and queue buildup even after the original trigger is fixed.

why us-east-1#

It is the oldest region, the largest, and the default in approximately every tutorial ever written. Several global AWS control planes are homed there — IAM, CloudFront configuration, Route 53's control plane, and others. That means a us-east-1 event has global blast radius even for customers with no resources in the region.

That architecture is a historical artifact. It is also extremely hard to change now, which is a lesson about early decisions in systems that grow.

the honest customer takeaway#

The reflexive response is "multi-region." Before you spend a year on that, do the arithmetic.

Multi-region active-active is genuinely hard. Data consistency across regions, failover testing that actually works, doubled infrastructure cost, and a substantially more complex system that fails in new ways. Many organizations that attempt it end up with a system that is less reliable overall because the complexity introduces more failure modes than the regional risk it removes.

The dependency you cannot escape. If your multi-region architecture depends on a global control plane that lives in us-east-1, you did not achieve independence. Check this specifically. A lot of people discovered it yesterday.

What is actually worth doing, in order:

1. Know your dependencies. Most teams cannot enumerate what their service requires to start. Write it down. The exercise is revealing.
2. Static stability. Design so existing capacity keeps serving when the control plane is unavailable. If your service needs to call an API to keep running, it will stop when that API stops. Cache aggressively, fail open where safe, and do not require a control plane call on the request path.
3. Graceful degradation. Decide in advance which features can be turned off. A checkout that works without recommendations is much better than a site that is down.
4. Exponential backoff with jitter, and circuit breakers. Retry storms are what turns an incident into an outage. This is the single highest-leverage code change available.
5. Multi-region for the tier that genuinely warrants it. Which is usually not everything.

the industry-level observation#

A meaningful fraction of the world's software depends on a small number of regions operated by a small number of companies. That concentration produces excellent reliability most of the time and correlated failure occasionally.

There is no individual fix. Every company independently choosing the most reliable provider produces exactly this concentration. It is a collective action problem, and the only actors who can address it are regulators thinking about systemic risk, who are — belatedly — starting to.