Story #4 · Railway · 3 min read

For five minutes, every domain on Railway said it didn't exist

On 30 September, a new routing service went live before the database change it needed. Route lookups failed, and Railway's proxies answered every new request with 404 Not Found.

Published 12 Oct 2026 · by the healthcheck.wtf team

Read the whole story ↓
Who
Every Railway-hosted domain, all regions
Outage on
30 Sep 2026
Down for
5 min
Source
Railway incident report

What happened

  1. 07:29 UTC00:29 Pacific

    A change to the routing control plane is merged after staging tests: a schema update plus a new routing service.

  2. 07:35 UTC00:35 Pacific

    About now, the new routing service goes live in every region before the schema update finishes. Every route lookup fails, and the proxies answer 404. Alerts page the on-call team.

  3. 07:40 UTC00:40 Pacific

    About now, the schema update finishes. The next retry works and domains route normally again.

  4. The question nobody answered: Routing came back by itself when the schema landed. What would it have taken to stop it if the migration had been slow?

  5. 07:44 UTC00:44 Pacific

    Engineers name the cause: the routing service came online before its schema.

  6. 07:47 UTC00:47 Pacific

    Full recovery confirmed. No rollback needed.

The WTF

For five minutes, Railway's proxies told the whole internet that its domains didn't exist. The routing service had only said "I can't answer right now". And the parallel pipelines came from a change made for resilience: routing moved out of the monolith so it keeps working during a major cloud outage, and the monolith's single pipeline was what kept the schema first.

Bingo 2 of 9:
  • Bad deploy
  • Database
Make your team's card →

The lesson

Two things are worth checking in our own systems.

First, does our proxy or client treat "the lookup failed" the same as "nothing found"? Those two need different answers. Railway's proxies would normally have fallen back to the last route they knew, and that fallback wasn't there on the day.

Second, what makes sure a migration runs before the code that needs it? If the answer is "the pipeline happens to run in that order", a refactor can quietly remove it. Railway is making the routing service refuse to go live until the matching schema is in place, forcing one order for migration and rollout, and adding region-by-region rollouts.

Make your new code check for its schema before it takes traffic.

Check this in your system

#HugOps

To the Railway team: thank you for a report that shows the timeline to the minute, explains why a lookup error became a 404, and says what you're changing.

Pass the card on

Download the card