The pause was on. The automation kept going.

On 7 October, a new machine in the service that tells GitHub where to send writes started with old records and routed writes to retired database servers. The team paused automated maintenance to keep it out. An automation that didn't honor the pause brought it back.

Read the whole story ↓
Who
GitHub, write operations across the site
Outage on
7 Oct 2026
Down for
19 min
Source
GitHub: Incident with Git Operations, Pull Requests and Actions

What happened

  1. 15:06 UTC08:06 PDT

    A new machine with old records rewrites write routing. Writes fail across GitHub; reads mostly work.

  2. 15:16 UTC08:16 PDT

    A machine with the correct configuration becomes leader again. Writes recover. The team finds the new machine and pauses automated maintenance.

  3. 16:52 UTC09:52 PDT

    A maintenance task that doesn't honor the pause puts the same machine back in charge. Writes fail again.

  4. 17:01 UTC10:01 PDT

    Writes recover.

  5. 17:02 UTC10:02 PDT

    Engineers shut the machine down.

The WTF

The team did the right thing after the first outage. They found the bad machine and pressed pause on the automation. The automation had stopped listening to pause after a recent update, so it brought the same machine back, and the same outage happened 96 minutes later.

Bingo 3 of 9:
  • Config change
  • Bad deploy
  • Database
Make your team's card →

The lesson

Most of us have a "stop" switch somewhere: a pause flag, a maintenance mode, a feature toggle. We usually find out whether it works on the day we need it. GitHub's automation now honors the pause again. A test that presses our own pause switch from time to time would tell us before an incident does.

The first outage started with old data on a new machine. GitHub is now clearing retired servers from the records before any new machine is added, and keeping new machines from becoming leader until their records are checked. Systems will also refuse a configuration that points to servers that don't exist and keep the last good one instead.

Reads were fine the whole time. Only writes failed.

Check this in your system

#HugOps

To the GitHub team: thank you for a summary that covers both outages together and says plainly that the second one happened because a pause wasn't honored. That detail is the useful part for everyone else, and it isn't an easy one to write.

Pass the card on

Download the card