Stories about Rare WTF

3 cards so far. Nine sleeves a page, newest first, a tab for each cause.

  1. #7 · BinanceDown ~2 days

    No key was stolen. $6.3 billion still left.

    WTF: Everyone guards the private keys. Here, by every account so far, the keys stayed safe. The weak point was the system allowed to use them. If a system can sign, then whoever can talk to that system can sign too. And the outage came after the theft. Turning withdrawals back on safely took longer than the theft itself: about 20 minutes to take the money, about two days to let people move theirs again.
    Preliminary

    Preliminary · Outage on 29 Oct 2026

  2. #6 · Google CloudDown 6 h 10 min

    One zone lost power, and the whole region felt it

    WTF: The facility had batteries to cover the gap and generators to take over. Even the controller that starts the generators had a backup. Both controllers failed, the batteries ran out after about seven minutes, and the generators stayed quiet. Then losing one zone, the exact failure regions are built to survive, made the whole region slow.
    Preliminary

    Preliminary · Outage on 20 Oct 2026

  3. #1 · Google CloudDown 14 h 55 min

    A 3-millisecond power blip, and a data hall at 44°C

    WTF: Backup power, a second feed and spare cooling were all in the design. In the end, a hand switching the pumps from "auto" to "hand" brought the water back.

    Full postmortem · Outage on 15 Jul 2026

Seen a great postmortem? Send it to us and it may become a card.