HBO Max caused a bit of a stir among its subscribers last week when they accidentally sent out an email titled “Integration Test Email #1” to some large number of their subscribers, which spawned a lot of clever commentary and snarky remarks on Twitter. A few days later HBO Max sent out an apology blaming an unnamed intern for the boo-boo, which spawned a lot of “Dear Intern” tweets with many commiserating with the unnamed intern by sharing their own stories of mishaps from their younger days.
There was also a thread of other developers and their stories of how they broke production too. One of my favorite stories has to be from Patrick McKenzie about his worst production bug while he was the one-man engineering team for his startup.
There’s a bit of an adage among programmers on the internet: There are two kinds of developers. Those who have broken production and those who are about to.
In my long career, I’ve either been lucky enough to have never caused any big production issues (either that or I’ve blocked out their memory somehow.) Part of it was probably because when I was a junior developer they almost never gave me access to production (like a sensible company!) Not to mention there’s always been several layers of reviews, testing and QA between me and production. And by the time I was senior enough to be handling production myself, I knew enough to be super careful with everything. Not that I didn’t work with production problems of course, I was just lucky enough to never be the direct cause of them myself. I think. But if the adage above is in any way true, should I be worried that eventually the law of averages will catch up to me, and maybe I should quite while I’m ahead?
See Also
Likes