My 3AM pipeline run broke prod and my first instinct made it worse

I rolled out a new SAST rule to our CI pipeline at 3AM last Tuesday, thought I tested it locally, but the regex was greedy and flagged every hardcoded string as a secret. It started failing builds for like 40 repos before I noticed the alert noise. My first instinct was to push a quick revert, but that triggered a fresh scan on the main branch and took down our staging API for 20 minutes. Now I keep a rollback playbook printed on my desk, has anyone else burned themselves trying to fix a bad rule before reading the logs first?

1 comments

1 Comment

finleyl551mo ago

Wait, did you check if the rollback playbook actually covers the blast radius or just the fix itself? Most people write those for the technical fix, but the real damage here was the second scan triggering on main. A good rule of thumb I picked up is to disable the webhook or gate on the repo side first, not push a code change. That stops any new scans from starting while you sort out the regex. Then you can fix the rule, test it on a fork, and only then re-enable things. Your playbook should have a freeze step at the top, before any revert or rollback, because that's the part that saves you from doubling the outage.