One zone lost power, and the whole region felt it

On 20 October, a grid disruption hit the data center behind zone europe-west2-b. Both controllers that start the generators failed, the batteries ran out, and the failover to the other zones overloaded the whole region.

Based on Google's preliminary incident report. We'll update it when the final report is out.

Read the whole story ↓
Who
Google Cloud, europe-west2
Outage on
20 Oct 2026
Down for
6 h 10 min
Source
Google Cloud: Multiple services were experiencing issues in the europe-west2 region

What happened

  1. 05:53 UTC06:53 BST

    A grid disruption hits the data center behind europe-west2-b. UPS batteries take the load; the generators don't start.

  2. 06:00 UTC07:00 BST

    The batteries run out. Servers, network and storage in zone b go offline.

  3. 06:08 UTC07:08 BST

    Monitoring and probers alert Google's engineers. Workloads fail over to the other zones, which run out of room; the region sees 500 and 503 errors.

  4. 12:10 UTC13:10 BST

    Most workloads are working again.

  5. 14:03 UTC15:03 BST

    A small tail of workloads recovers. Google closes the incident.

The WTF

The facility had batteries to cover the gap and generators to take over. Even the controller that starts the generators had a backup. Both controllers failed, the batteries ran out after about seven minutes, and the generators stayed quiet. Then losing one zone, the exact failure regions are built to survive, made the whole region slow.

Bingo 0 of 9. A rare one. Nobody saw this coming. Make your team's card →

The lesson

Most of us don't run generators. But many of us run in three zones and expect the other two to take the load if one goes away. That works only if they have room for it while everyone else is failing over too, and if calls to the lost zone fail fast instead of hanging.

In the UK, banks including Lloyds, HSBC and Revolut reported problems for their customers that morning, according to AP. Recovery at the provider and recovery for its customers didn't happen at the same moment. Google closed its incident at 14:03 UTC. Revolut said its services were working again later, in the evening. Our own "recovered" can come hours after our provider's.

Google calls this a preliminary report; a final one with preventive actions will follow. So far, the facility team has reset and tested the generator controllers.

Plan your failover for the minute when everyone fails over at once.

Check this in your system

#HugOps

To the Google Cloud team: thank you for a preliminary report within two days that names the exact part that failed, both generator controllers. And thank you for explaining how one zone's outage spread to the rest of the region. That part helps everyone who designs for zone failure.

Pass the card on

Download the card