A 3-millisecond power blip, and a data hall at 44°C

The grid dipped for less than a blink. The backup failed on one side, a chiller controller went dark, and in the end a hand on a switch brought the water back.

Read the whole story ↓
Who
Google Cloud, europe-west4-a
Outage on
15 Jul 2026
Down for
14 h 55 min
Source
Google Cloud incident report

What happened

The power grid outside a data center dropped its voltage for 3 milliseconds. It hit both power feeds. Side B switched to backup cleanly. Side A's backup failed because of faulty electrical parts.

At the same moment, the chiller controller went offline. The chillers had power, but nobody told the water pumps to start again. The spare cooling was out for construction work. Two hours later the hall reached 44°C, and machines were shut down to protect them.

The WTF

Backup power, a second feed and spare cooling were all in the design. In the end, a hand switching the pumps from "auto" to "hand" brought the water back.

Bingo 0 of 9. A rare one. Nobody saw this coming. Make your team's card →

The lesson

Our backups have their own small dependencies, and we rarely test those. A spare that is "out for construction" is not a spare. We've all had a redundant system that was quietly down when we needed it.

List what your backup needs to work, not only the backup itself.

#HugOps

To the Google Cloud team: thank you for a full report, down to the minute. Writing "manually switched from auto to hand mode" in public is honest, and it helps all of us.

Pass the card on

Download the card