Cloudflare falls over because of a config file

Cloudflare had a significant outage on Tuesday, returning 5xx errors across a large portion of its network for several hours. Their published postmortem is detailed and worth reading in full.

The chain of events is a small masterpiece of the genre.

what happened#

A database permissions change caused a query that generates a Bot Management feature configuration file to return duplicate rows. The query had been returning one row per feature; after the permissions change it returned rows from multiple underlying schemas.

The generated file therefore roughly doubled in size.

The proxy that consumes this file preallocates a fixed-size buffer sized against a limit of 200 features — comfortably above the ~60 actually in use. The doubled file exceeded that limit.

The Rust code handling this hit an unrecoverable error path and the proxy panicked rather than degrading. Because the configuration file propagates network-wide every few minutes, the failure propagated network-wide within minutes.

Recovery was complicated by the fact that the bad file kept regenerating and redeploying.

the lessons, which are old#

Configuration is code and needs the same rigor. This was not a code deploy. It was a data change that propagated to production automatically with no staging, no canary, and no validation. Config deployment pipelines are consistently held to a lower standard than code deployment pipelines, and config causes a large share of major outages.

Fixed-size limits need to fail soft. The 200-feature limit was reasonable. Panicking when exceeded was not. The correct behavior for a proxy encountering an oversized config is to log loudly, alert, and continue with the previous known-good version.

Validate generated artifacts before propagation. A size check, a schema check, a sanity check on row count against the previous version — any of these would have caught it. Generated files should be validated as rigorously as user input, because the generator can be wrong.

Blast radius follows deployment speed. Config that propagates globally in minutes is a feature until it propagates a bad config globally in minutes. Staged rollout applies to configuration too.

Have a kill switch for automated pipelines. Much of the recovery time went to stopping the thing that kept redeploying the bad file. Every automated deployment path needs a way to stop it that does not require fixing the underlying problem first.

the Rust note#

This will be used as an argument about Rust, and it should not be.

The panic was a deliberate choice at that call site — the code used a construct that terminates on error rather than propagating it. That is a design decision about error handling, available in any language. In C the equivalent code would have written past the buffer, which is worse.

The actual lesson is about where you choose to make errors fatal. In a proxy handling live traffic, almost nothing should be fatal. Degrade, alert, continue. "Fail fast" is good advice for a batch job and bad advice for a load balancer.

the credit due#

Cloudflare published a detailed technical postmortem within a day, named the specific code path, and did not hide behind "an issue with a third-party provider."

That is how it should be done and it is rarer than it should be. A company that publishes real postmortems earns more trust than one that never has visible incidents, because the second one is not telling you about them.