Evolution of Star Wars Combine Development Process

That's it! We just migrated the Star Wars Combine online game to our new servers. We perform such an upgrade of hardware every 3 years roughly to cope with growth of both users and resources. We also use these upgrades as an opportunity to improve our development processes. Or maybe it is the opposite: we decide to update our development processes and we upgrade our hardware at the same time.

To make it short when we we started the Combine back in 1998, all we had as environment was a few text flat files as database, some HTML files for the website and a downloadable client developed by a single person who uploaded the compiled software on the server so that our players could download it. Back then the server was in fact just a small account on a free hosting services whose name I have long forgotten. Do not forget that we are just a bunch of volunteers working on this project during their free time and for free (or least for the sake of code).

Over time, we switched from a client/server model to a pure web-based model. Our architecture became clearer in our mind and since 2006 we were using 2 servers: one as production environment and one as development environment. Since a couple of years, we even decided to release our new features and bugfixes as weekly patches, allowing us to fix most problems on the development environment before going live.

This was already an improvement but we still had several issues regarding code quality due to our developers working in parallel and most of them having very different programming levels and background.

Now, since our recent Development Meeting of last February in Berlin, a new process has been introduced along with new tools. First we have created a virtual machine our server so that developers can work locally on their repository. That VM includes the database and the web server. When they perform some changes, all they need is to save their file and refresh the concerned VM webpage. Once they are happy with their changes, they can commit them to the development server.

On the development server, we have testers performing functional tests and providing input to the developers. On top of the functional testing, we have some unit tests that ensure that new developments will not break existing code (and features). Then, once all our tests are passed, we can proceed with our weekly release. Database schema changes are pushed and the production code is updated.

Even if this new process require more efforts from the developers such as writing unit tests and spending more time in the testing phase, the middle-term and long-term benefits of this new process will be invaluable. We are creating today a way to decrease tomorrow's bugs.

At the same time, we introduced a powerful database dedicated server to cope of the increasing database demands generated by our natural growth and increasing game complexity. Our frond-end web server can now only take care of generating webpages and will so open us the gate for future load-balancing.

Exporting Files from SharePoint Content Database

Murphy's Law can apply anywhere anytime and my recent re-installation of SharePoint is certainly no exception to the rules.

Soon after I finished recreating our SharePoint intranet, I stared sharing it with my immediate colleagues. While I was not really expecting some "Oh!" and "Aaah!" cheerful comments, I was still hoping for some kind of "Cool, it's back, thanks for the tool.".

But no, even that was expecting too much. The very first comment I received was an assassin comment like: "And I suppose you managed to retrieve all files stored in the previous SharePoint database ?". That comment triggered a machine-gun like stare from me to my colleague. No, I did not manage so far to retrieve these files and yes I had tried already. SQL Management Studio did not really help me by the way.

I had been cornered and really had to find a solution this time. So I returned to the web, googling frantically, trying various unsuccessful solutions until finally my patience and stubbornness got rewarded. I found found a great article from Keith Richie describing how to export the site content from a SharePoint Database for recovery purpose.

His article include a small source code and instruction to compile and run the program. It worked almost like a charm from the first run. Yet, as some of the recovered files were quite large, I had to boost the buffer size (as I worked locally this was not a problem) and even tweak the SQL query to work on a per directory basis to limit the execution run time. Happily this time SQL Server Management Studio was of great help to test and fine-tune the queries. Within 30 minutes from the moment I read Keith's post I had recovered all files I needed.

Today all files are imported in the new SharePoint and I preciously kept the program source and instruction ... just in case one day I need to recover files from another SharePoint database.

Never Release Software Before a Leave

This is one of the first rule I learned and of course i learned it the hard way: Never ever release any piece of code into production before going to bed, vacations, leaving your computer or even before Christmas.

Even without taking Murphy's Law into consideration, it is only best practice to remain available immediately after any change to your IT production environment. And of course you only deploy changes after having carefully developing them in a dedicated environment and testing them in yet another environment (classic 3-tier environment).

This rule, we - at SW Combine - learned it the hard way several times, releasing some new game features right before going to bed only to find a crashed server only a few hours later when waking up. naturally we decided not to let that happen anymore and to never ever release anything before leaving.

However we had underestimated the dark side of temptation. Despite all our good resolutions and even also some pretty decent precautions, we released a very important game upgrade immediately before going to bed and while having the main developer also leaving for Japan for two weeks only hours later. And guess when it all happened ? Around midnight on the 23 of December 2008 when everyone was more thinking to partying, eating, and drinking than coding and when availability was a scarce resource.

So what was the result ? Well we got a crashed server. We got instabilities on our MySQL database leading to further random crashes and we got over 270 reported bugs during the following two weeks.

And do we regret it ?
Not a single minute. The release of this important patch has federated the team into a joined effort like we too rarely had before. All available developers put their hands into the mud (1). During the last fifteen days, we fixed over 170 of these bugs and released a patch every 2 or 3 days (5 patches in total) to solve the immediate problems. Aside of the new features fixes, a lot of minor improvements were released by the rest of the team contributing to the global effort and improvement of the game. Ultimately the entire team feels stronger and more motivated after these two weeks. We did the opposite of what every software development manual would say and yet we have no regret. This is probably one of those exceptions which confirms the rule.

I would not want to end this post without thanking those who worked hard and long on this upgrade: kudos and thanks go to Sin, Khan, BK for the main effort but also Mach, Mikel, Aurius, Git, Jeb, Syn, Jen and those I forgot. You helped, it counted, we progressed.

(1) I know some who will surely try to kill me for implicating that their code could be compared to mud. ;)

Between Theory & Practice

Today I found back an old article I had written for a few friends of mine a few years ago while working as R&D software programmer and wanted to share the story again as it remains pretty much true and applies to other fields of IT as well.. Do not worry; you do not need to have a master in advanced telecom technologies to understand what is written below.

Back then my job consisted to design and implement a communication bus (SMBus); in other words think telephone line as it is easier to understand. This digital bus was composed of 2 wires: the clock wire and the data wire. The following devices were connected on the bus: a PC, a laptop battery (yes, you can talk to a battery, the very same battery you have inside your laptop and believe me it will answer you), and a power supply control card with a microcontroller (MCU) and a small memory chip on it. All these 4 devices had their own bus address (think phone number), and they were able to call each other in order to send and receive data (like talking on the phone or sending a fax). Last detail, all these components were grounded (linked to the ground) for electrical security reason. The theoretical schematic of the bus looked like this.

Didn't it look so nice on paper ?

In order to check the data transmissions and receptions (ie, monitor the communication), I had also connected an oscilloscope on the bus.

Now, the reality was slightly different and the result I obtained looked rather like this:

Tell me you got scared by reality.

But don't trust Murphy's Law on this one as it actually worked. This being said, I am confident you will feel reassured by knowing that similar busses are present inside your computers, your car, the trains or the planes you take, and so on.