The whole set
9 cards so far. Nine sleeves a page, newest first, a tab for each cause.
-
#9 · RailwayDown ~2 days
The attacker called endpoints nobody had used in years
WTF: Railway first learned the attacker had its source code from the endpoints they probed: ones that had not been used for weeks, months or even years, with no live references left in the system. Nobody calls those. Unless they are reading your code. -
#8 · GitHubDown 19 min
The pause was on. The automation kept going.
WTF: The team did the right thing after the first outage. They found the bad machine and pressed pause on the automation. The automation had stopped listening to pause after a recent update, so it brought the same machine back, and the same outage happened 96 minutes later. -
#7 · BinanceDown ~2 days
No key was stolen. $6.3 billion still left.
WTF: Everyone guards the private keys. Here, by every account so far, the keys stayed safe. The weak point was the system allowed to use them. If a system can sign, then whoever can talk to that system can sign too. And the outage came after the theft. Turning withdrawals back on safely took longer than the theft itself: about 20 minutes to take the money, about two days to let people move theirs again. Preliminary -
#6 · Google CloudDown 6 h 10 min
One zone lost power, and the whole region felt it
WTF: The facility had batteries to cover the gap and generators to take over. Even the controller that starts the generators had a backup. Both controllers failed, the batteries ran out after about seven minutes, and the generators stayed quiet. Then losing one zone, the exact failure regions are built to survive, made the whole region slow. Preliminary -
#5 · GitHubDown 8 h 25 min
The notifications that took down what they notify about
WTF: Notifications exist to tell us when something happens on an issue or a pull request. Here the notifications pipeline used up the database capacity that issues and pull requests needed, so they stopped working. Then the notifications were the last thing to recover: the backlog took almost five more hours to send. -
#4 · RailwayDown 5 min
For five minutes, every domain on Railway said it didn't exist
WTF: For five minutes, Railway's proxies told the whole internet that its domains didn't exist. The routing service had only said "I can't answer right now". And the parallel pipelines came from a change made for resilience: routing moved out of the monolith so it keeps working during a major cloud outage, and the monolith's single pipeline was what kept the schema first. -
#3 · SlackDown 6 h 20 min
An app posted billions of messages, and Slack's free workspaces went quiet
WTF: The message limit is there to protect the free plan. On this day it worked exactly as built, and it stopped the teams themselves, not the app. For about six hours they couldn't talk in their own workspace. Preliminary -
#2 · Google CloudDown 4 h 11 min
A router upgrade planned for several days, done in 13 minutes
WTF: The network was built so that one router could fail and nobody would notice. But the work order had every router on it, and nothing in the process said to wait between them. So a plan spread over several days became 13 minutes of unplugging. -
#1 · Google CloudDown 14 h 55 min
A 3-millisecond power blip, and a data hall at 44°C
WTF: Backup power, a second feed and spare cooling were all in the design. In the end, a hand switching the pumps from "auto" to "hand" brought the water back.
Seen a great postmortem? Send it to us and it may become a card.