- Who
- Railway, builds, deploys and stored environment variables
- Outage on
- 20 Oct 2026
- Down for
- ~2 days
What happened
07:28 UTC00:28 PDT
20 Oct. The attacker partly escapes build isolation and takes a build-server credential.
13:58 UTC06:58 PDT
An alert on unusual registry calls pages the on-call engineer.
17:28 UTC10:28 PDT
The build credential is replaced. Builds come back after the fix.
03:26 UTC20:26 PDT
21 Oct. Using the copied code, the attacker gets cloud credentials through a misconfigured OIDC binding.
04:07 UTC21:07 PDT
They start reading encrypted customer variables, AI provider keys first, and decrypting some of them.
07:14 UTC00:14 PDT
Railway disables every cloud credential. Builds stop again at 07:22.
09:24 UTC02:24 PDT
The attacker is fully out of Railway's systems.
22:49 UTC15:49 PDT
22 Oct. Builds are back for every plan, on new isolated build machines.
The WTF
Railway first learned the attacker had its source code from the endpoints they probed: ones that had not been used for weeks, months or even years, with no live references left in the system. Nobody calls those. Unless they are reading your code.
- Config change
The lesson
A credential is only as narrow as the systems that accept it. The build credential was meant to push images, but a second system trusted anything that held it. Railway now gives each build its own short-lived virtual machine and a credential that can push one image and nothing else.
Old things are still things. An endpoint nobody calls, an OIDC binding nobody remembers and a deprecated service all count as a way in for someone who reads the code. Railway has audited its infrastructure to make sure deprecated systems are fully shut down.
Railway told the AI providers before most customers, so stolen keys could be revoked while it was still securing its systems. It also says broad notice brought 10 to 100 times more traffic and slowed recovery. Its runbook now sets a fixed time to measure impact, after which it notifies everyone anyway.
Check this in your system
#HugOps
To the Railway team: thank you for a long, detailed report on a hard week, including the parts that don't look good, like the OIDC binding and why customers heard later than the providers. Security write-ups this open are rare, and the rest of us learn from every one.