Every year or two a major provider has a very public bad day, and the internet briefly remembers that single points of failure include famous ones. The lessons repeat with remarkable consistency.
Lesson one: configuration is the threat
The headline incidents of recent years were overwhelmingly self-inflicted, bad config pushes and software defects propagating globally in seconds. Scale does not protect against this; scale is what propagates it. Provider diversity is the only structural defense, because two networks rarely ship the same mistake simultaneously. The post-incident reports are public and remarkably candid, which makes them the cheapest education in the industry for anyone willing to read primary sources.
Lesson two: failover must pre-exist
Teams that fared well during incidents had steering already live and rehearsed. Teams that had a second CDN but no automated routing spent the outage doing DNS surgery by hand, which is multi-CDN theater rather than multi-CDN. The rehearsal gap is the difference between owning infrastructure and owning a diagram of it, and incidents are rude about revealing which one you have.
One more pattern from the incident record: the failovers that worked were boring by design, static thresholds, simple rules, pre-authorized decisions, while the clever ones hesitated. Sophisticated steering logic that weighed many signals sometimes weighed them for minutes while users stared at errors, because nobody had pre-answered the question of how bad is bad enough. The lesson is not against sophistication but for pre-commitment: whatever the logic, the thresholds that trigger it must be decided in calm, written down and rehearsed, because mid-incident is the most expensive possible venue for a philosophy discussion.
Lesson three: the blast radius includes your vendors
Your status page, your auth provider and your monitoring may share the failing CDN. Resilience audits should trace the whole dependency chain, not just your own delivery. Dependency tracing has a concrete method: list every third-party domain on your critical path, resolve where each is served from, and count the single points of failure you inherited without deciding to.
In practice
Steal the discipline without waiting for an incident: schedule a quarterly failover drill, timed, with the on-call team that would actually run it. Publish the minutes-to-clean-failover number internally. Nothing else in this article improves that number; only rehearsal does, and the teams that fared well in every public incident were, without exception, the rehearsed ones.
We rehearse failover as part of multi-CDN engagements, because an untested failover is a hypothesis.
