Real outages. Collect the lesson.

Real outages from public postmortems: what broke, why, and what the team changed. No blame. Could've been any of us.

Get the next card by email Get the next card

Website

Free. Unsubscribe in one click.

Read the latest story →

1. #2 · Google CloudDown 4 h 11 min A router upgrade planned for several days, done in 13 minutes WTF: The network was built so that one router could fail and nobody would notice. But the work order had every router on it, and nothing in the process said to wait between them. So a plan spread over several days became 13 minutes of unplugging.
2. #3 · SlackDown 6 h 20 min An app posted billions of messages, and Slack's free workspaces went quiet WTF: The message limit is there to protect the free plan. On this day it worked exactly as built, and it stopped the teams themselves, not the app. For about six hours they couldn't talk in their own workspace. Preliminary
3. #4 · RailwayDown 5 min For five minutes, every domain on Railway said it didn't exist WTF: For five minutes, Railway's proxies told the whole internet that its domains didn't exist. The routing service had only said "I can't answer right now". And the parallel pipelines came from a change made for resilience: routing moved out of the monolith so it keeps working during a major cloud outage, and the monolith's single pipeline was what kept the schema first.
4. #5
   next week

See the whole set →