Four conventions fix most of it, and none of them need a tool.

1. one channel, created at declaration#

Not the team channel. Not a thread in the team channel. A dedicated channel per incident, named predictably: inc-2026-08-29-checkout-errors.

Why it matters: the incident becomes searchable as a unit, the timeline is the channel, and people who join late can read from the top instead of asking what happened. It also means the incident does not drown the normal channel, and the normal channel does not drown the incident.

Archive it afterwards rather than deleting it. It is the primary source for the postmortem.

2. named roles, stated out loud#

Three, at minimum, posted in the channel as the first message:

IC: @priya       — decides, delegates, does not debug
Comms: @marcus   — status page, stakeholders, customer support
Ops: @sam        — hands on keyboard

The incident commander not debugging is the rule people resist and the one that matters most. The moment the IC opens a terminal, nobody is tracking the whole picture, and the incident gets longer.

For a small incident one person can hold two roles. They should still say which ones, because the alternative is everyone assuming someone else is doing comms.

3. a pinned status message, edited in place#

One message, pinned, rewritten as things change:

STATUS 14:22 — Checkout failing for ~12% of users since 13:58.
Cause: unknown. Suspect the payments deploy at 13:55.
Now: rolling back that deploy (@sam), ETA 5 min.
Impact: card payments only, wallet payments unaffected.
Next update: 14:35

Five lines: what is broken, since when, what we think, what we are doing, when the next update comes.

This single convention removes most of the noise, because it answers the question that generates the noise — "what's the current state?" — without anyone having to ask. Everyone joining reads the pin instead of scrolling.

Always include the next-update time. It is what stops people asking for updates.

4. mark the speculation#

The most common way incidents go wrong is that a guess gets repeated until it becomes the working theory, and then twenty minutes go into the wrong system.

A one-word convention fixes it:

FACT: error rate went from 0.1% to 12% at 13:58:20
FACT: the payments deploy completed at 13:55:41
GUESS: the deploy caused it
ACTION: rolling back to confirm

Facts have evidence attached. Guesses are labelled as guesses. Actions say who is doing them.

It looks pedantic for about four minutes and then it saves the incident, because somebody reading the channel can tell the difference between what is known and what somebody said.

what to keep out#

Speculation from people who are not investigating. Well-meant and it fills the channel that responders are trying to read. If you are not on a role, watch.

"Is it fixed yet?" The pinned status has the next update time.

Root-cause analysis during the incident. Restore first. The good question at 14:22 is "what makes this stop", not "why did this happen". Why is a question for Thursday.

Blame, in any form, including jokes. It is in a permanent record that a person will read afterwards.

the handoff#

Long incidents cross shift boundaries and the handoff is where context dies. It needs to be explicit and in the channel:

HANDOFF 22:00 — @priya → @dan (IC)
Ruled out: deploy (rolled back, no change), CDN (unaffected regions also failing)
Current theory: connection pool exhaustion on the orders DB
In flight: @sam is capturing pg_stat_activity every 30s → thread above
Not yet tried: failover to replica
Customers: status page updated 21:40, support has the template

Ruled out, current theory, in flight, not tried, customer state. Five lines, and the incoming IC starts with the accumulated knowledge instead of rediscovering it.

why bother#

The channel is not a side effect of the incident. During the incident it is the coordination mechanism, and afterwards it is the only complete record of what was known and when.

A channel that follows these four conventions produces a postmortem that half writes itself, and — more importantly — an incident that ends sooner, because the people fixing it spent their attention on the system rather than on the chat.

the failure pattern#

The typical postmortem:

1. Timeline of what happened. Accurate, detailed, useful.
2. Root cause. Usually a single technical fact.
3. Action items. Five to twelve of them.
4. Filed. Two action items get done. The rest age out.

Six months later, a similar incident, with a similar document.

Three things are wrong here.

problem one: "root cause" is singular#

Complex systems do not fail because of one thing. They fail because several conditions aligned, each of which was individually survivable.

The Cloudflare outage in November: a permissions change, a query that returned duplicates, a fixed buffer size, an error path that panicked instead of degrading, and a config pipeline that propagated globally in minutes. Remove any one and it does not happen, or it is much smaller.

Picking one and calling it "the root cause" means you fix one and leave four.

Better framing: contributing factors, plural, each with its own assessment of whether it is worth addressing. Some will not be — that is a legitimate decision if it is made explicitly.

problem two: action items without owners and dates are wishes#

An action item that says "improve monitoring for the config pipeline" with no owner, no date, and no definition of done is a sentence, not a plan.

The fix is unglamorous:

• Every action item has one named person. Not a team. A person.
• Every action item has a date. If nobody will commit to a date, it is not going to happen and you should delete it and say so.
• Every action item has a definition of done that someone else could verify.
• They go in the same backlog as feature work, prioritized against it. An action item in a separate "incident follow-up" list that is never sprint planned is a list of things that will not be done.

And the one that actually forces it: review the open action items at the start of the next postmortem. Nothing motivates completion like a room full of people looking at your undone item from last quarter's incident while discussing this quarter's similar one.

problem three: nobody asks about the near misses#

The incidents you review are the ones that broke through. For every one, there were several that did not — a bad deploy caught by a canary, a config error someone noticed in review, a query that would have taken down the database if it had run on Monday instead of Sunday.

Those contain the same information at a fraction of the cost, and almost nobody collects them.

Add a lightweight channel for it. "I nearly broke prod today, here is how." No document, no meeting, no blame. Just a note. The pattern that emerges over a quarter is more valuable than any individual postmortem.

the questions that produce useful findings#

Replace "what was the root cause" with:

• "What made this hard to detect?" Detection time is usually the largest component of impact and is the most improvable.
• "What made this hard to diagnose?" Usually missing observability. This produces the highest-value action items.
• "What made recovery slow?" Often a missing runbook, a missing kill switch, or a rollback that was not actually tested.
• "Who knew something that would have helped, and why did that not reach the responders?" This is an organizational question and it is frequently the real finding.
• "What did we do that helped?" Genuinely important. Practices that worked should be named so they get kept.

the blameless part, done correctly#

Blameless does not mean nobody made a mistake. It means the analysis focuses on why the mistake was possible and easy, rather than on the person.

"Alice deployed without running the tests" is blame and it is also useless. "The deploy path does not require tests to pass, and the shortcut that skips them is the fastest way to deploy" is the same fact stated in a way you can act on.

If the answer to "why did they do that" is "because the system made it easy and the correct path was hard," you have found something.

If the answer is genuinely "they were careless," you still fix the system, because the next person will also be careless eventually. Humans are the constant; the system is the variable.

The chain of events is a small masterpiece of the genre.

what happened#

A database permissions change caused a query that generates a Bot Management feature configuration file to return duplicate rows. The query had been returning one row per feature; after the permissions change it returned rows from multiple underlying schemas.

The generated file therefore roughly doubled in size.

The proxy that consumes this file preallocates a fixed-size buffer sized against a limit of 200 features — comfortably above the ~60 actually in use. The doubled file exceeded that limit.

The Rust code handling this hit an unrecoverable error path and the proxy panicked rather than degrading. Because the configuration file propagates network-wide every few minutes, the failure propagated network-wide within minutes.

Recovery was complicated by the fact that the bad file kept regenerating and redeploying.

the lessons, which are old#

Configuration is code and needs the same rigor. This was not a code deploy. It was a data change that propagated to production automatically with no staging, no canary, and no validation. Config deployment pipelines are consistently held to a lower standard than code deployment pipelines, and config causes a large share of major outages.

Fixed-size limits need to fail soft. The 200-feature limit was reasonable. Panicking when exceeded was not. The correct behavior for a proxy encountering an oversized config is to log loudly, alert, and continue with the previous known-good version.

Validate generated artifacts before propagation. A size check, a schema check, a sanity check on row count against the previous version — any of these would have caught it. Generated files should be validated as rigorously as user input, because the generator can be wrong.

Blast radius follows deployment speed. Config that propagates globally in minutes is a feature until it propagates a bad config globally in minutes. Staged rollout applies to configuration too.

Have a kill switch for automated pipelines. Much of the recovery time went to stopping the thing that kept redeploying the bad file. Every automated deployment path needs a way to stop it that does not require fixing the underlying problem first.

the Rust note#

This will be used as an argument about Rust, and it should not be.

The panic was a deliberate choice at that call site — the code used a construct that terminates on error rather than propagating it. That is a design decision about error handling, available in any language. In C the equivalent code would have written past the buffer, which is worse.

The actual lesson is about where you choose to make errors fatal. In a proxy handling live traffic, almost nothing should be fatal. Degrade, alert, continue. "Fail fast" is good advice for a batch job and bad advice for a load balancer.

the credit due#

Cloudflare published a detailed technical postmortem within a day, named the specific code path, and did not hide behind "an issue with a third-party provider."

That is how it should be done and it is rarer than it should be. A company that publishes real postmortems earns more trust than one that never has visible incidents, because the second one is not telling you about them.

AWS's public summary attributes the trigger to a latent race condition in the automation that manages DynamoDB's DNS records, which resulted in an empty record set for a regional endpoint and no automatic recovery path.

the cascade#

The failure sequence is the interesting part.

DynamoDB's endpoint became unresolvable. That alone would be bad. What made it a regional event is that an enormous number of AWS's own services use DynamoDB internally. The EC2 instance launch path, IAM's control plane, Lambda's invocation machinery, and dozens of others depend on it.

So the failure propagated: DynamoDB down means new EC2 instances cannot launch, which means autoscaling cannot replace failing capacity, which means load shedding, which means more failures. Network Load Balancer health checks destabilized. The recovery itself was slowed by the backlog of queued work that had accumulated.

This is textbook metastable failure: a system that is stable under normal load and stable under no load, but which, once pushed past a threshold, sustains its own failure through retry amplification and queue buildup even after the original trigger is fixed.

why us-east-1#

It is the oldest region, the largest, and the default in approximately every tutorial ever written. Several global AWS control planes are homed there — IAM, CloudFront configuration, Route 53's control plane, and others. That means a us-east-1 event has global blast radius even for customers with no resources in the region.

That architecture is a historical artifact. It is also extremely hard to change now, which is a lesson about early decisions in systems that grow.

the honest customer takeaway#

The reflexive response is "multi-region." Before you spend a year on that, do the arithmetic.

Multi-region active-active is genuinely hard. Data consistency across regions, failover testing that actually works, doubled infrastructure cost, and a substantially more complex system that fails in new ways. Many organizations that attempt it end up with a system that is less reliable overall because the complexity introduces more failure modes than the regional risk it removes.

The dependency you cannot escape. If your multi-region architecture depends on a global control plane that lives in us-east-1, you did not achieve independence. Check this specifically. A lot of people discovered it yesterday.

What is actually worth doing, in order:

1. Know your dependencies. Most teams cannot enumerate what their service requires to start. Write it down. The exercise is revealing.
2. Static stability. Design so existing capacity keeps serving when the control plane is unavailable. If your service needs to call an API to keep running, it will stop when that API stops. Cache aggressively, fail open where safe, and do not require a control plane call on the request path.
3. Graceful degradation. Decide in advance which features can be turned off. A checkout that works without recommendations is much better than a site that is down.
4. Exponential backoff with jitter, and circuit breakers. Retry storms are what turns an incident into an outage. This is the single highest-leverage code change available.
5. Multi-region for the tier that genuinely warrants it. Which is usually not everything.

the industry-level observation#

A meaningful fraction of the world's software depends on a small number of regions operated by a small number of companies. That concentration produces excellent reliability most of the time and correlated failure occasionally.

There is no individual fix. Every company independently choosing the most reliable provider produces exactly this concentration. It is a collective action problem, and the only actors who can address it are regulators thinking about systemic risk, who are — belatedly — starting to.

how it works#

The loop:

1. A malicious package version is installed. Its postinstall runs.
2. The payload harvests credentials from the machine — npm tokens, GitHub tokens, cloud provider credentials — using TruffleHog-style secret scanning.
3. It exfiltrates them to a public repository created in the victim's GitHub account.
4. It uses the stolen npm token to publish trojanized versions of every package that token can publish to.
5. Those packages get installed. Go to 1.

Step four is the difference between an incident and an outbreak. Previous npm compromises required an attacker to manually obtain credentials for each package. This one propagates on its own, and its rate of spread is proportional to how many packages the compromised maintainers control.

Additional behavior observed: creating public forks of private repositories in victims' organizations, and installing a GitHub Actions workflow for persistence.

why this was inevitable#

Every precondition has been in place for years:

• Packages execute arbitrary code on install, by default.
• Publishing credentials are commonly stored on developer machines in plaintext.
• One credential frequently controls many packages.
• Nothing in the publishing flow requires human presence.

Given those four facts, a worm is not a clever attack. It is the obvious one. Security researchers have described this exact scenario in talks for the better part of a decade.

the immediate response#

npm has accelerated changes it had already announced: shorter token lifetimes, mandatory two-factor for high-impact publishing, restrictions on classic tokens, and a strong push toward trusted publishing.

Those are the right changes. They are also the changes that were "coming" for years and are now arriving under emergency conditions, which is how infrastructure security usually improves.

what to do right now#

Rotate every npm token you have, especially any on a developer machine or in a CI secret store. Assume anything that was on a machine that ran an install during the window is exposed.

Enable trusted publishing on every package you maintain. This removes the long-lived token entirely — publishing is authorized by an OIDC assertion from a specific workflow in a specific repository.

Audit your GitHub account for repositories and workflows you did not create. The persistence mechanism was a workflow file; it survives credential rotation.

Turn off install scripts everywhere you can. This remains the single highest value control and remains widely unused.

Adopt a version cooldown. Every one of these incidents has a window between publication and detection measured in hours. A 72-hour delay before adopting new versions costs you nothing and avoids nearly all of them.

the structural conclusion#

The npm ecosystem's install-time code execution is a design decision from 2010 that made sense when the registry had a few thousand packages maintained by people who mostly knew each other.

It does not make sense now, and every incident makes the argument more forcefully. Other ecosystems handle this differently — Go has no install-time execution at all, and it turns out you can build a package ecosystem without it.

The migration cost for npm is enormous and the cost of not migrating is this, repeatedly, with escalating sophistication. At some point the arithmetic flips. It may have just flipped.

Combined weekly download counts for the affected packages are on the order of two billion.

the phish#

An email from npmjs.help — a lookalike domain — claiming the account required two-factor re-verification, with a link to a credential harvesting page that also captured the TOTP code.

The maintainer has written publicly about it. The email was well-constructed, it arrived at a plausible time, and the domain was close enough to pass a quick glance.

This is worth saying clearly: the target was a competent, security-aware, long-time open source maintainer. Phishing that captures a TOTP in real time defeats the second factor. The control that actually stops this is a hardware security key or passkey, where the authentication is bound to the origin and a lookalike domain simply cannot complete it.

If you publish packages, use a passkey or hardware key. Today. TOTP is not sufficient against this attack and has not been for years.

the payload#

Browser-targeted, not server-targeted, which is unusual and clever.

The injected code hooked window.ethereum and intercepted fetch and XMLHttpRequest, watching for cryptocurrency transactions and swapping destination addresses for attacker-controlled ones — selecting a visually similar address to survive a casual glance at the confirmation dialog.

Because these packages are ubiquitous transitive dependencies of frontend build tooling, the payload had a plausible path into a very large number of shipped bundles.

Actual losses appear to have been small. Detection was fast — within a couple of hours — and most builds during the window did not pull the affected versions.

why the damage was limited#

Three things, in order:

1. Lockfiles. Most production builds resolve to pinned versions. A new malicious release does not enter an existing lockfile without someone running an update.
2. Speed of detection. The community noticed within hours. Someone diffed the published tarball against the repository and the mismatch was obvious.
3. Payload specificity. Targeting crypto wallets is narrow. A payload targeting build-time credential theft would have done far more damage.

That third point should not be comforting. The same access with a better payload would have been much worse.

the controls, again#

The same short list as every one of these:

• Phishing-resistant auth on publishing accounts. Non-negotiable.
• Trusted publishing via OIDC rather than long-lived tokens.
• Lockfiles, committed, with exact versions in production.
• --ignore-scripts in CI.
• A cooldown before adopting new versions.

None of these are new. All of them are still not universally adopted, which is the actual story.

the structural observation#

chalk adds color to terminal output. It is about a thousand lines. It has two billion weekly downloads because it is a transitive dependency of nearly everything in the JavaScript ecosystem.

A thousand-line utility maintained by a volunteer sits in the trusted computing base of a substantial fraction of the world's software, and the ecosystem's security posture depends on that person's email hygiene.

That is not a criticism of the maintainer, who did nothing unreasonable. It is a description of a system that has grown a dependency structure nobody designed and nobody can now change.

The fix is not "fewer dependencies." The fix is publishing infrastructure where a single compromised human credential cannot ship code to two billion machines. The registry has to make that impossible, because asking every maintainer to be unphishable has failed for a decade.

the payload#

The script searched for:

• npm and GitHub tokens
• SSH private keys
• Cloud provider credentials in the usual locations
• Environment variables matching credential-shaped patterns
• Cryptocurrency wallet files

It then created a public repository in the victim's GitHub account named with a recognizable prefix and pushed the harvested data to it.

Some variants additionally invoked locally-installed AI coding CLI tools with a prompt asking them to enumerate sensitive files — using the developer's own agent as a discovery mechanism. That detail is new and it is going to be studied.

the entry point#

Compromised publishing credentials. The specifics of how they were obtained matter less than the pattern: a maintainer's token, or a CI workflow with publishing rights, was reachable by an attacker.

This is the dominant supply chain attack shape. Not typosquatting, not dependency confusion, not a malicious contribution. Take over the account of a legitimate maintainer of a package people already depend on.

the controls that would have stopped it#

In order of effectiveness:

1. --ignore-scripts. The payload was in postinstall. An install that does not run lifecycle scripts does not run the payload.

npm ci --ignore-scripts

Set it in .npmrc for your CI and see what breaks. For most projects the answer is: nothing, or one package that needs a native build, which you allowlist.

2. Trusted publishing. OIDC-based publishing from a verified CI workflow rather than a long-lived token. A token that does not exist cannot be stolen. npm supports this now and adoption is the bottleneck.

3. Install cooldown. The malicious versions were live for hours before removal. A policy of not adopting a version until it is 24 to 72 hours old would have avoided this entirely, and avoids most incidents of this shape, because the window between publish and detection is short.

{ "minimumReleaseAge": 4320 }

4. Short-lived credentials. The payload harvested long-lived tokens. If your CI uses OIDC federation to get a fifteen-minute cloud credential, there is nothing durable to steal.

if you were affected#

The order matters:

1. Rotate everything the machine had access to. npm tokens, GitHub PATs, SSH keys, cloud credentials. Assume everything readable was read.
2. Check for the public repository in your GitHub account and delete it — but capture its contents first for your incident record.
3. Audit for persistence. Check ~/.bashrc, ~/.zshrc, ~/.profile, shell history, cron, launch agents, and git hooks in your repositories.
4. Review recent activity on every account whose credentials were on that machine.

Rotation before cleanup. Cleanup on a compromised machine while live credentials are still valid is how you spend an afternoon on remediation and the attacker spends it on access.

the trend#

This is one of several npm incidents this year and they are getting more sophisticated: better targeting, cleaner payloads, and now the use of local AI tooling as an attack primitive.

The ecosystem's structural weakness has not changed. A single maintainer's credentials protect a package that millions of machines install and execute automatically. Every control above mitigates; none of them fix that.

The fix requires the registry to make unattended publishing hard by default, and that is a change to how a lot of people work. It is going to happen anyway, because the alternative is this every few weeks.