The notifications that took down what they notify about

On 12 October, GitHub's notifications pipeline started sending a high volume of an expensive query to the shared database replicas. Issues, Pull Requests, Projects and Actions lost the capacity they needed and returned errors.

Read the whole story ↓
Who
Issues, Pull Requests and Actions
Outage on
12 Oct 2026
Down for
8 h 25 min
Source
GitHub status: Incident with Actions, Pull Requests and Issues

What happened

  1. 11:51 UTC04:51 PDT

    github.com starts returning HTTP 500 errors. Issues, Pull Requests, Projects and Actions fail; notifications are delayed.

  2. 12:21 UTC05:21 PDT

    GitHub opens an incident for degraded Actions, Issues and Pull Requests.

  3. 12:33 UTC05:33 PDT

    The status page lists what fails: the Actions tab, the Checks API, queueing runs, creating issues, requesting reviews, also through the APIs.

  4. 13:02 UTC06:02 PDT

    Pull Requests work normally again.

  5. 13:38 UTC06:38 PDT

    The first mitigation didn't fully work. Actions, Issues and Projects still see delays and errors.

  6. 14:36 UTC07:36 PDT

    Issues and Actions are mitigated.

  7. 15:19 UTC08:19 PDT

    A fix for notifications is in. They resume, behind a backlog.

  8. 20:16 UTC13:16 PDT

    The notifications backlog is cleared. Resolved.

The WTF

Notifications exist to tell us when something happens on an issue or a pull request. Here the notifications pipeline used up the database capacity that issues and pull requests needed, so they stopped working. Then the notifications were the last thing to recover: the backlog took almost five more hours to send.

Bingo 2 of 9:
  • Database
  • Limit hit
Make your team's card →

The lesson

Work that runs in the background often reads from the same database as the pages people are waiting on. Most days that's fine. When one background query gets expensive, or runs far more often than it should, it takes capacity from everyone.

GitHub fixed the code path that sent the expensive query, and says it is making the database layer more resilient. For our own systems, the question is simple: if a background job starts reading too much, does it slow down only itself, or does it take the user-facing pages with it?

Give background jobs a database budget of their own, so a bad query slows the job and not the product.

Check this in your system

#HugOps

To the GitHub team: thank you for naming the exact path, a notifications query on shared replicas, and for posting updates all through a long day. The backlog part is the bit nobody likes to write about.

Pass the card on

Download the card