Found out that a single bad line of code caused a 2 hour outage at my company

Was looking through our git log last week after a big system crash and found a comment from a dev that said "oops, forgot to close this loop." Dug into it and realized one missing bracket in a JavaScript file brought down our whole checkout page for 120 minutes. I had no idea such a tiny thing could break an entire site. Has anyone else seen something this small cause a huge mess?

2 comments

2 Comments

lane.angela22d ago

Honestly I used to be one of those people who rolled their eyes at code reviews. Thought they were a waste of time and that I knew what I was doing. Then last year I had a similar thing happen at my old job where a missing semicolon in a PHP file killed our payment gateway for an hour and a half. It was the dumbest thing and I felt so stupid after. Now I'm the person who double checks every single bracket and semicolon before I push anything. It's crazy how one tiny mistake can just domino into a total disaster like that.

rosep8722d ago

There's a great blog post going around about how the Mars Climate Orbiter crashed because of a unit conversion error. Makes you think twice about the little things, right?