Evolution of Star Wars Combine Development Process

That's it! We just migrated the Star Wars Combine online game to our new servers. We perform such an upgrade of hardware every 3 years roughly to cope with growth of both users and resources. We also use these upgrades as an opportunity to improve our development processes. Or maybe it is the opposite: we decide to update our development processes and we upgrade our hardware at the same time.

To make it short when we we started the Combine back in 1998, all we had as environment was a few text flat files as database, some HTML files for the website and a downloadable client developed by a single person who uploaded the compiled software on the server so that our players could download it. Back then the server was in fact just a small account on a free hosting services whose name I have long forgotten. Do not forget that we are just a bunch of volunteers working on this project during their free time and for free (or least for the sake of code).

Over time, we switched from a client/server model to a pure web-based model. Our architecture became clearer in our mind and since 2006 we were using 2 servers: one as production environment and one as development environment. Since a couple of years, we even decided to release our new features and bugfixes as weekly patches, allowing us to fix most problems on the development environment before going live.

This was already an improvement but we still had several issues regarding code quality due to our developers working in parallel and most of them having very different programming levels and background.

Now, since our recent Development Meeting of last February in Berlin, a new process has been introduced along with new tools. First we have created a virtual machine our server so that developers can work locally on their repository. That VM includes the database and the web server. When they perform some changes, all they need is to save their file and refresh the concerned VM webpage. Once they are happy with their changes, they can commit them to the development server.

On the development server, we have testers performing functional tests and providing input to the developers. On top of the functional testing, we have some unit tests that ensure that new developments will not break existing code (and features). Then, once all our tests are passed, we can proceed with our weekly release. Database schema changes are pushed and the production code is updated.

Even if this new process require more efforts from the developers such as writing unit tests and spending more time in the testing phase, the middle-term and long-term benefits of this new process will be invaluable. We are creating today a way to decrease tomorrow's bugs.

At the same time, we introduced a powerful database dedicated server to cope of the increasing database demands generated by our natural growth and increasing game complexity. Our frond-end web server can now only take care of generating webpages and will so open us the gate for future load-balancing.

Never Release Software Before a Leave

This is one of the first rule I learned and of course i learned it the hard way: Never ever release any piece of code into production before going to bed, vacations, leaving your computer or even before Christmas.

Even without taking Murphy's Law into consideration, it is only best practice to remain available immediately after any change to your IT production environment. And of course you only deploy changes after having carefully developing them in a dedicated environment and testing them in yet another environment (classic 3-tier environment).

This rule, we - at SW Combine - learned it the hard way several times, releasing some new game features right before going to bed only to find a crashed server only a few hours later when waking up. naturally we decided not to let that happen anymore and to never ever release anything before leaving.

However we had underestimated the dark side of temptation. Despite all our good resolutions and even also some pretty decent precautions, we released a very important game upgrade immediately before going to bed and while having the main developer also leaving for Japan for two weeks only hours later. And guess when it all happened ? Around midnight on the 23 of December 2008 when everyone was more thinking to partying, eating, and drinking than coding and when availability was a scarce resource.

So what was the result ? Well we got a crashed server. We got instabilities on our MySQL database leading to further random crashes and we got over 270 reported bugs during the following two weeks.

And do we regret it ?
Not a single minute. The release of this important patch has federated the team into a joined effort like we too rarely had before. All available developers put their hands into the mud (1). During the last fifteen days, we fixed over 170 of these bugs and released a patch every 2 or 3 days (5 patches in total) to solve the immediate problems. Aside of the new features fixes, a lot of minor improvements were released by the rest of the team contributing to the global effort and improvement of the game. Ultimately the entire team feels stronger and more motivated after these two weeks. We did the opposite of what every software development manual would say and yet we have no regret. This is probably one of those exceptions which confirms the rule.

I would not want to end this post without thanking those who worked hard and long on this upgrade: kudos and thanks go to Sin, Khan, BK for the main effort but also Mach, Mikel, Aurius, Git, Jeb, Syn, Jen and those I forgot. You helped, it counted, we progressed.

(1) I know some who will surely try to kill me for implicating that their code could be compared to mud. ;)

Evaluate Your WebSite

Would you like to know how much your website is worth ?

It is possible. Simply go to WebsiteOutlook and try with yours.

You can find below "my" own website value (Star Wars Combine at www.swcombine.com) but I can already tell you that it is out of the question to sell it for such a small price, especially when considering I live in Europe and the US dollar to Euro exchange rate is ridiculously low. Our 10 years of collective investments and efforts are definitively worth more.

My site is worth $33601.9.
How much is yours worth?

And I'm not even talking about the unveiled potential.

Ten years of Online Gaming Presence

Historically, it was on the 3rd of December 1998 that Frederic "Fizzban" Waymann, Mario "PtJedi" Dominguez and myself (known online as Veynom) decided to start an online game on the ashes of a defunct one called Star Wars Simulation. Unlike any game editor, while we had a few hundreds players around us eager to play, we had absolutely no game to offer. We could only propose hope, dreams, and a perpetual answer that would quickly became famous: Soon™. Yet, on that day, we founded the Star Wars Combine.

Ten years after, hundreds of people have contributed to the growing success of the Star Wars Combine. Our membership has always grown (even if sometimes slowly) despite ups and downs in the game development. showing that one of our biggest successes is doubtlessly the community.

Now, like once every ten years, I will rather tell a small story instead of boring the world to death with a “we’re the best” type of speech. So here comes one of my Uncle Vey ’ stories which I will name "The Lost Days".

A long time ago in a not so far far away galaxy …
errr no. Wrong story.

Today is the 3rd of December. Today is the official tenth anniversary of the Star Wars Combine. Yet today is Year 10 Day 3 by standard Combine Galactic Time (CGT) - an in-game time reference. Where does the difference come from ?

Is the memory of that old Sim Master acting crazy and forcing him to believe the Combine was started on December 3 while it was in fact on December the first ?
Did some well-known thief manage to steal two days from the Combine ?
Did an Imperial historian make a mistake while re-copying the official Combine chronicles ?
Is this … a feature ?

Absolutely not. This is no feature, this is a bug !
This is simply a bug due to how CGT code converting calendar dates to our own system which does not stake leap years into account. As consequences, we lose a day every four years and one of those years we will need to fix these days. Enjoy,

Overloading Second Life

Today, I learned something funny from Kristian Köhntopp, a Principal consultant from MySQL Berlin. Second Life, the famous online virtual world, uses a partitioning system based on the geographical location of its universe. In short, this means that each server supporting Second Life only supports a small area of the universe. If more than 200 players gathers in the same room or building or island, then you have a serious chance to see it crash.

Why would it be this funny ?
Simply because I wrote my Master thesis on Networked Virtual Environments back in 2001-2002 back when I was still studying at the Free University of Brussels. For it, I describe a model of partitioning which was also based on the geographical locations and I even wrote a small prototype with two servers using my own online game the Star Wars Combine.

We live in such a small world ...