Last week we released Hunter Score 2.0. From a backend perspective, this was the biggest change we’ve had in quite a while, and it touched quite a few systems. For those that don’t know the details from a game design perspective, please refer to Alena’s post.

In this blog post, I will discuss what has been done from a technical perspective.

Preamble: theHunter’s server architecture pre-Hunter Score 2.0
Before I can explain how the new architecture looks, I would like to go over how the old (and current, mostly, see further) architecture works and is set up. We’ll do this back to front, meaning we’ll discuss the layer the furthest away from the player first.

– _Storage_: The SQL database: We use google Cloud SQL to store all the data related to the game, nothing special here, Google takes care of our database, it’s mostly stable.

– _Caching_: Google App Engine’s managed Memcache, with 10GB of dedicated memory.

– _Servers_: Google App Engine (PHP and Python): This is where it gets more challenging. We use Google App Engine to serve the website, as well as the API for the game. The website is a mix of PHP (inherited from the olden times before Expansive Worlds and Avalanche Studios owned theHunter) and html/css/javascript. The API servers meanwhile are written in Python 2.7, both the PHP and Python code are running on App Engine, and connect to the Cloud SQL database and memcache.

– _Cloudflare_: Cloudflare provides an additional layer of caching, it will cache static files such as images, html, css and javascript files, so our web server does not get swamped.

– _Website/Launcher_: The html/css/js served by the web server will be rendered by either your browser, or the game launcher. It is a single page app built using pretty old versions of marionette/backbone frameworks(the ones released back when the website was built).

Challenges
There are several pieces in this stack that are posing challenges for us in the dev team, the main one being the current server/web stack.

– These old versions of marionette and backbone are incredibly verbose and troublesome to work with, making any web changes slow and hard to test.

– The App Engine servers have no unit tests nor automated integration tests, this makes them prone to breakage, especially with the weakly typed languages that are being used.

– The servers have no sane error checking, making it impossible to tell the difference between a user sending an incorrect request and genuine server errors, this makes logs dramatically less useful than they should be.

– Database consistency was not considered during its development, meaning that errors can sometimes cause data corruption, which burdens both the players and our support team.

– Much of our static game data (such as which achievements are available to players, which animals spawn where, etc.) are stored in JSON files. While this is a practical format for developers, these files are most often changed by the game design team, this leads to occasional breakage whenever a single comma or quotation mark is forgotten.

What we’re moving towards
With the release of HS 2.0, we made the first steps towards improving on all this to move towards a more stable server architecture. The idea is that we will minimize the amount of changes made to the Python/PHP servers and instead start building on a new server stack, written in Go and hosted on Google GKE (Google Container Engine: A Managed Kubernetes cluster).

How does this improve on our problems, and why aren’t we just fixing the current servers instead? Well, I’m glad you asked, there are several reasons:

1) The main and most important reason is that, to fix the current servers, we’d need to rewrite them anyway, a lot of individual pieces are poorly designed (from a technical POV) and exhibit unexpected and undocumented behaviour in many cases. Writing new pieces and making sure they are well documented and don’t exhibit such undocumented behaviour solves this issue.

2) Weak typing and lack of unit tests: Python and PHP are both weakly typed languages, combined with the lack of unit tests, this makes any large refactoring or change extremely prone to breakage and passes the whole burden of ensuring correctness for any change onto the QA team, this is not sustainable. Rewriting piece by piece from scratch and making sure we have good unit tests coverage and use a strongly typed language for the new pieces solves this issue.

3) This allows us to move static game data to an administration panel, much easier to use by the game design team.

Upgrade approach
We are not going to be rewriting the whole server stack in one go, this would take months if not years, and would completely stop development of new features.

Our path forward consists in rewriting piece by piece as we make new features, the first step in this was made with the release of HS 2.0, we’ve moved the achievement system from JSON files to our administration panel, we’ve made an administration panel for ranks, and we’ve written an entirely new server for all of it that is now hosted on kubernetes. The new achievement and ranking system does API calls to our new servers written in Go, which are hosted in a kubernetes cluster on Google Cloud.

Compared the the Python servers, we’ve already noticed a huge increase in stability and performance compared to the old, the logging is also much better and they have full unit test coverage.

Hope this was of some interest to you. If you have any questions, please do not hesitate to ask! I am looking forward to working towards a more stable and faster theHunter experience.

Discussion