Stories about Rare WTF
3 cards so far. Nine sleeves a page, newest first, a tab for each cause.
-
#7 · BinanceDown ~2 days
No key was stolen. $6.3 billion still left.
WTF: Everyone guards the private keys. Here, by every account so far, the keys stayed safe. The weak point was the system allowed to use them. If a system can sign, then whoever can talk to that system can sign too. And the outage came after the theft. Turning withdrawals back on safely took longer than the theft itself: about 20 minutes to take the money, about two days to let people move theirs again. Preliminary -
#6 · Google CloudDown 6 h 10 min
One zone lost power, and the whole region felt it
WTF: The facility had batteries to cover the gap and generators to take over. Even the controller that starts the generators had a backup. Both controllers failed, the batteries ran out after about seven minutes, and the generators stayed quiet. Then losing one zone, the exact failure regions are built to survive, made the whole region slow. Preliminary -
#1 · Google CloudDown 14 h 55 min
A 3-millisecond power blip, and a data hall at 44°C
WTF: Backup power, a second feed and spare cooling were all in the design. In the end, a hand switching the pumps from "auto" to "hand" brought the water back.
Seen a great postmortem? Send it to us and it may become a card.