Real outages. Collect the lesson.
Real outages from public postmortems: what broke, why, and what the team changed. No blame. Could've been any of us.
Free. Unsubscribe in one click.
Read the latest story →- #7 · BinanceDown ~2 days No key was stolen. $6.3 billion still left. WTF: Everyone guards the private keys. Here, by every account so far, the keys stayed safe. The weak point was the system allowed to use them. If a system can sign, then whoever can talk to that system can sign too. And the outage came after the theft. Turning withdrawals back on safely took longer than the theft itself: about 20 minutes to take the money, about two days to let people move theirs again. Preliminary
- #8 · GitHubDown 19 min The pause was on. The automation kept going. WTF: The team did the right thing after the first outage. They found the bad machine and pressed pause on the automation. The automation had stopped listening to pause after a recent update, so it brought the same machine back, and the same outage happened 96 minutes later.
- #9 · RailwayDown ~2 days The attacker called endpoints nobody had used in years WTF: Railway first learned the attacker had its source code from the endpoints they probed: ones that had not been used for weeks, months or even years, with no live references left in the system. Nobody calls those. Unless they are reading your code.
Every card ends with a hug
The teams in these reports had a very bad day and wrote it up so the rest of us could learn. The last line of every card thanks them. #HugOps
Bonus cards: free tools
All tools →
Bonus · Translator
What the status page means
“We are investigating reports of elevated error rates.” → Something is broken. We don't know what yet.
Translate an update → Bonus · BingoYour team's outage bingo
Pick 9 squares, stamp them when they happen, and share the card with your team.
Make your team's card → Bonus · TemplateA kind postmortem template
10 parts, with hints and an example, ready for Docs or Notion.
Get the template →