- Who
- Google Cloud, europe-west4-a
- Outage on
- 15 Jul 2026
- Down for
- 14 h 55 min
What happened
The power grid outside a data center dropped its voltage for 3 milliseconds. It hit both power feeds. Side B switched to backup cleanly. Side A's backup failed because of faulty electrical parts.
At the same moment, the chiller controller went offline. The chillers had power, but nobody told the water pumps to start again. The spare cooling was out for construction work. Two hours later the hall reached 44°C, and machines were shut down to protect them.
The WTF
Backup power, a second feed and spare cooling were all in the design. In the end, a hand switching the pumps from "auto" to "hand" brought the water back.
The lesson
Our backups have their own small dependencies, and we rarely test those. A spare that is "out for construction" is not a spare. We've all had a redundant system that was quietly down when we needed it.
List what your backup needs to work, not only the backup itself.
#HugOps
To the Google Cloud team: thank you for a full report, down to the minute. Writing "manually switched from auto to hand mode" in public is honest, and it helps all of us.