- Who
- Issues, Pull Requests and Actions
- Outage on
- 12 Oct 2026
- Down for
- 8 h 25 min
What happened
11:51 UTC04:51 PDT
github.com starts returning HTTP 500 errors. Issues, Pull Requests, Projects and Actions fail; notifications are delayed.
12:21 UTC05:21 PDT
GitHub opens an incident for degraded Actions, Issues and Pull Requests.
12:33 UTC05:33 PDT
The status page lists what fails: the Actions tab, the Checks API, queueing runs, creating issues, requesting reviews, also through the APIs.
13:02 UTC06:02 PDT
Pull Requests work normally again.
13:38 UTC06:38 PDT
The first mitigation didn't fully work. Actions, Issues and Projects still see delays and errors.
14:36 UTC07:36 PDT
Issues and Actions are mitigated.
15:19 UTC08:19 PDT
A fix for notifications is in. They resume, behind a backlog.
20:16 UTC13:16 PDT
The notifications backlog is cleared. Resolved.
The WTF
Notifications exist to tell us when something happens on an issue or a pull request. Here the notifications pipeline used up the database capacity that issues and pull requests needed, so they stopped working. Then the notifications were the last thing to recover: the backlog took almost five more hours to send.
- Database
- Limit hit
The lesson
Work that runs in the background often reads from the same database as the pages people are waiting on. Most days that's fine. When one background query gets expensive, or runs far more often than it should, it takes capacity from everyone.
GitHub fixed the code path that sent the expensive query, and says it is making the database layer more resilient. For our own systems, the question is simple: if a background job starts reading too much, does it slow down only itself, or does it take the user-facing pages with it?
Give background jobs a database budget of their own, so a bad query slows the job and not the product.
Check this in your system
#HugOps
To the GitHub team: thank you for naming the exact path, a notifications query on shared replicas, and for posting updates all through a long day. The backlog part is the bit nobody likes to write about.