- Who
- Google Cloud, europe-west2
- Outage on
- 20 Oct 2026
- Down for
- 6 h 10 min
What happened
05:53 UTC06:53 BST
A grid disruption hits the data center behind europe-west2-b. UPS batteries take the load; the generators don't start.
06:00 UTC07:00 BST
The batteries run out. Servers, network and storage in zone b go offline.
06:08 UTC07:08 BST
Monitoring and probers alert Google's engineers. Workloads fail over to the other zones, which run out of room; the region sees 500 and 503 errors.
12:10 UTC13:10 BST
Most workloads are working again.
14:03 UTC15:03 BST
A small tail of workloads recovers. Google closes the incident.
The WTF
The facility had batteries to cover the gap and generators to take over. Even the controller that starts the generators had a backup. Both controllers failed, the batteries ran out after about seven minutes, and the generators stayed quiet. Then losing one zone, the exact failure regions are built to survive, made the whole region slow.
The lesson
Most of us don't run generators. But many of us run in three zones and expect the other two to take the load if one goes away. That works only if they have room for it while everyone else is failing over too, and if calls to the lost zone fail fast instead of hanging.
In the UK, banks including Lloyds, HSBC and Revolut reported problems for their customers that morning, according to AP. Recovery at the provider and recovery for its customers didn't happen at the same moment. Google closed its incident at 14:03 UTC. Revolut said its services were working again later, in the evening. Our own "recovered" can come hours after our provider's.
Google calls this a preliminary report; a final one with preventive actions will follow. So far, the facility team has reset and tested the generator controllers.
Plan your failover for the minute when everyone fails over at once.
Check this in your system
#HugOps
To the Google Cloud team: thank you for a preliminary report within two days that names the exact part that failed, both generator controllers. And thank you for explaining how one zone's outage spread to the rest of the region. That part helps everyone who designs for zone failure.