Evolution of Star Wars Combine Development Process

That's it! We just migrated the Star Wars Combine online game to our new servers. We perform such an upgrade of hardware every 3 years roughly to cope with growth of both users and resources. We also use these upgrades as an opportunity to improve our development processes. Or maybe it is the opposite: we decide to update our development processes and we upgrade our hardware at the same time.

To make it short when we we started the Combine back in 1998, all we had as environment was a few text flat files as database, some HTML files for the website and a downloadable client developed by a single person who uploaded the compiled software on the server so that our players could download it. Back then the server was in fact just a small account on a free hosting services whose name I have long forgotten. Do not forget that we are just a bunch of volunteers working on this project during their free time and for free (or least for the sake of code).

Over time, we switched from a client/server model to a pure web-based model. Our architecture became clearer in our mind and since 2006 we were using 2 servers: one as production environment and one as development environment. Since a couple of years, we even decided to release our new features and bugfixes as weekly patches, allowing us to fix most problems on the development environment before going live.

This was already an improvement but we still had several issues regarding code quality due to our developers working in parallel and most of them having very different programming levels and background.

Now, since our recent Development Meeting of last February in Berlin, a new process has been introduced along with new tools. First we have created a virtual machine our server so that developers can work locally on their repository. That VM includes the database and the web server. When they perform some changes, all they need is to save their file and refresh the concerned VM webpage. Once they are happy with their changes, they can commit them to the development server.

On the development server, we have testers performing functional tests and providing input to the developers. On top of the functional testing, we have some unit tests that ensure that new developments will not break existing code (and features). Then, once all our tests are passed, we can proceed with our weekly release. Database schema changes are pushed and the production code is updated.

Even if this new process require more efforts from the developers such as writing unit tests and spending more time in the testing phase, the middle-term and long-term benefits of this new process will be invaluable. We are creating today a way to decrease tomorrow's bugs.

At the same time, we introduced a powerful database dedicated server to cope of the increasing database demands generated by our natural growth and increasing game complexity. Our frond-end web server can now only take care of generating webpages and will so open us the gate for future load-balancing.

Kongregate: Casual Meta-Game ?

Since a few weeks now, I play some casual games and other flash games on Kongregate. Kongregate is a portal promoting new games and helping developers to develop better games by getting more feedback and to earn more money because it remains the main reason for work.

By playing Kongregate's games, it is possible to earn points, badges, and level up. Earned badges and total scores appear on the players' profiles; often motivating them to play earn more badges by playing more games.

This game-around-the-games mechanism - or Meta-game - is quite interesting as players are motivated to play more to games they like; until they earn all badges for example. It also pushes players to discover and play new games.

This mechanism is also enforced thanks to the existence of Kongai, an online battle-oriented card game. Kongai relies on similar (but less complex) concepts than older classical card games like Magic The Gathering or Decipher' Star Wars Customizable Cards Game. Cards can be of course purchased but also and preferably earned through playing any game hosted on Kongregate. This is called challenges). More cards can be earned by playing Kongai itself. As for any collectible cards game, these come with several editions and you can view them through an online card album.


This concept of Meta-games or games around games is quite thrilling. Once you start caring for your score, the race is up and you will quickly try to reach just another level. As Kongregate proposes thousands of games, you will always find some that you like or care enough to earn one more badge. The online card game attached around is also a good idea even if in this particular case the Kongai game itself could probably be a bit more sophisticated.

Now I wonder if a meta-game concept could be used with massively multiplayers online role-playing games (MMORPG) ?

Never Release Software Before a Leave

This is one of the first rule I learned and of course i learned it the hard way: Never ever release any piece of code into production before going to bed, vacations, leaving your computer or even before Christmas.

Even without taking Murphy's Law into consideration, it is only best practice to remain available immediately after any change to your IT production environment. And of course you only deploy changes after having carefully developing them in a dedicated environment and testing them in yet another environment (classic 3-tier environment).

This rule, we - at SW Combine - learned it the hard way several times, releasing some new game features right before going to bed only to find a crashed server only a few hours later when waking up. naturally we decided not to let that happen anymore and to never ever release anything before leaving.

However we had underestimated the dark side of temptation. Despite all our good resolutions and even also some pretty decent precautions, we released a very important game upgrade immediately before going to bed and while having the main developer also leaving for Japan for two weeks only hours later. And guess when it all happened ? Around midnight on the 23 of December 2008 when everyone was more thinking to partying, eating, and drinking than coding and when availability was a scarce resource.

So what was the result ? Well we got a crashed server. We got instabilities on our MySQL database leading to further random crashes and we got over 270 reported bugs during the following two weeks.

And do we regret it ?
Not a single minute. The release of this important patch has federated the team into a joined effort like we too rarely had before. All available developers put their hands into the mud (1). During the last fifteen days, we fixed over 170 of these bugs and released a patch every 2 or 3 days (5 patches in total) to solve the immediate problems. Aside of the new features fixes, a lot of minor improvements were released by the rest of the team contributing to the global effort and improvement of the game. Ultimately the entire team feels stronger and more motivated after these two weeks. We did the opposite of what every software development manual would say and yet we have no regret. This is probably one of those exceptions which confirms the rule.

I would not want to end this post without thanking those who worked hard and long on this upgrade: kudos and thanks go to Sin, Khan, BK for the main effort but also Mach, Mikel, Aurius, Git, Jeb, Syn, Jen and those I forgot. You helped, it counted, we progressed.

(1) I know some who will surely try to kill me for implicating that their code could be compared to mud. ;)

Color Bulles Solitaire - Another Cool Casual Game

Color Bulles Solitaire is another casual games that I particularly enjoy. It is simple, challenging, yet fun and not totally repetitive. Its main disadvantage comes from the difficulty at some upper levels. Indeed, starting with level 7, it becomes extremely difficult to win and most of your success (or failure) will depend solely on chance and on the order in which you receives the balls.

I'm also not too happy with the scoring system. Why having to deal with a score when the real purpose is to pass through levels ?

WoW Connect - Good Casual Game

I consider a good casual game any game where I can play for only a few minutes without any time constraint and that I enjoy playing again a day or two after; and this for a few weeks.

WoW Connect (based on a World of Warcraft theme) is a typical casual game I really enjoy. Besides, it is good for your memory and observation mind.

Overloading Second Life

Today, I learned something funny from Kristian Köhntopp, a Principal consultant from MySQL Berlin. Second Life, the famous online virtual world, uses a partitioning system based on the geographical location of its universe. In short, this means that each server supporting Second Life only supports a small area of the universe. If more than 200 players gathers in the same room or building or island, then you have a serious chance to see it crash.

Why would it be this funny ?
Simply because I wrote my Master thesis on Networked Virtual Environments back in 2001-2002 back when I was still studying at the Free University of Brussels. For it, I describe a model of partitioning which was also based on the geographical locations and I even wrote a small prototype with two servers using my own online game the Star Wars Combine.

We live in such a small world ...