Never Release Software Before a Leave

This is one of the first rule I learned and of course i learned it the hard way: Never ever release any piece of code into production before going to bed, vacations, leaving your computer or even before Christmas.

Even without taking Murphy's Law into consideration, it is only best practice to remain available immediately after any change to your IT production environment. And of course you only deploy changes after having carefully developing them in a dedicated environment and testing them in yet another environment (classic 3-tier environment).

This rule, we - at SW Combine - learned it the hard way several times, releasing some new game features right before going to bed only to find a crashed server only a few hours later when waking up. naturally we decided not to let that happen anymore and to never ever release anything before leaving.

However we had underestimated the dark side of temptation. Despite all our good resolutions and even also some pretty decent precautions, we released a very important game upgrade immediately before going to bed and while having the main developer also leaving for Japan for two weeks only hours later. And guess when it all happened ? Around midnight on the 23 of December 2008 when everyone was more thinking to partying, eating, and drinking than coding and when availability was a scarce resource.

So what was the result ? Well we got a crashed server. We got instabilities on our MySQL database leading to further random crashes and we got over 270 reported bugs during the following two weeks.

And do we regret it ?
Not a single minute. The release of this important patch has federated the team into a joined effort like we too rarely had before. All available developers put their hands into the mud (1). During the last fifteen days, we fixed over 170 of these bugs and released a patch every 2 or 3 days (5 patches in total) to solve the immediate problems. Aside of the new features fixes, a lot of minor improvements were released by the rest of the team contributing to the global effort and improvement of the game. Ultimately the entire team feels stronger and more motivated after these two weeks. We did the opposite of what every software development manual would say and yet we have no regret. This is probably one of those exceptions which confirms the rule.

I would not want to end this post without thanking those who worked hard and long on this upgrade: kudos and thanks go to Sin, Khan, BK for the main effort but also Mach, Mikel, Aurius, Git, Jeb, Syn, Jen and those I forgot. You helped, it counted, we progressed.

(1) I know some who will surely try to kill me for implicating that their code could be compared to mud. ;)