Stories about Config change
4 cards so far. Nine sleeves a page, newest first, a tab for each cause.
-
#9 · RailwayDown ~2 days
The attacker called endpoints nobody had used in years
WTF: Railway first learned the attacker had its source code from the endpoints they probed: ones that had not been used for weeks, months or even years, with no live references left in the system. Nobody calls those. Unless they are reading your code. -
#8 · GitHubDown 19 min
The pause was on. The automation kept going.
WTF: The team did the right thing after the first outage. They found the bad machine and pressed pause on the automation. The automation had stopped listening to pause after a recent update, so it brought the same machine back, and the same outage happened 96 minutes later. -
#3 · SlackDown 6 h 20 min
An app posted billions of messages, and Slack's free workspaces went quiet
WTF: The message limit is there to protect the free plan. On this day it worked exactly as built, and it stopped the teams themselves, not the app. For about six hours they couldn't talk in their own workspace. Preliminary -
#2 · Google CloudDown 4 h 11 min
A router upgrade planned for several days, done in 13 minutes
WTF: The network was built so that one router could fail and nobody would notice. But the work order had every router on it, and nothing in the process said to wait between them. So a plan spread over several days became 13 minutes of unplugging.
Seen a great postmortem? Send it to us and it may become a card.