- Who
- GitHub, write operations across the site
- Outage on
- 7 Oct 2026
- Down for
- 19 min
What happened
15:06 UTC08:06 PDT
A new machine with old records rewrites write routing. Writes fail across GitHub; reads mostly work.
15:16 UTC08:16 PDT
A machine with the correct configuration becomes leader again. Writes recover. The team finds the new machine and pauses automated maintenance.
16:52 UTC09:52 PDT
A maintenance task that doesn't honor the pause puts the same machine back in charge. Writes fail again.
17:01 UTC10:01 PDT
Writes recover.
17:02 UTC10:02 PDT
Engineers shut the machine down.
The WTF
The team did the right thing after the first outage. They found the bad machine and pressed pause on the automation. The automation had stopped listening to pause after a recent update, so it brought the same machine back, and the same outage happened 96 minutes later.
- Config change
- Bad deploy
- Database
The lesson
Most of us have a "stop" switch somewhere: a pause flag, a maintenance mode, a feature toggle. We usually find out whether it works on the day we need it. GitHub's automation now honors the pause again. A test that presses our own pause switch from time to time would tell us before an incident does.
The first outage started with old data on a new machine. GitHub is now clearing retired servers from the records before any new machine is added, and keeping new machines from becoming leader until their records are checked. Systems will also refuse a configuration that points to servers that don't exist and keep the last good one instead.
Reads were fine the whole time. Only writes failed.
Check this in your system
#HugOps
To the GitHub team: thank you for a summary that covers both outages together and says plainly that the second one happened because a pause wasn't honored. That detail is the useful part for everyone else, and it isn't an easy one to write.