GTA: Online?

After Matt’s recent opinion piece about the entitled attitude of some gamers, I decided to dig a little deeper into the GTA Online debacle… The following paragraph is an actual comment taken from the Rockstar Support Forum; this is actually one of the more eloquently put examples of the Rockstar failed side of the argument:

“We paid our hard earned money to play this & basically only got half of what we were supposed to. We have every right to be pissed, epscially [SIC] when we see our friends online playing & sending us invites to a game we can’t even play. They had months of time with all the preorders put in, plus an additional 2 weeks to get it straightened out. They knew in advance how many people to expect to be online. In my view, there’s no excuse good enough for Rockstar not having this sht [SIC] sorted out & ready to go. Maybe next time they should beta test it before launch of the game as a safety precaution.”

Having worked for multinational corporations and knowing a bit about the red tape and rigmorale that can surround large infrastructure investments, I decided to contact an expert in the field and get their independent opinion on a few elements of this debate.

Chris Gill is a Senior Software Engineer who works for a UK based web developer, which runs their own server farm; he is not connected to Rockstar or GTA Online, but he has a far greater understanding of the intricacies of these things than I do… We asked his technical opinon on whether Rockstar should have been able to ensure all users could access GTA Online, having had two weeks to ‘get it straightened out’ and would a beta test have helped avoid this?

“Rockstar Games will operate their own server network, basically the GTA ‘cloud’. If that doesn’t have enough capacity they will either need to upgrade the hardware (more memory/CPU etc.) or more likely just buy more servers to increase it. This hardware will need to be ordered, delivered, plugged in, installed and networked. The additional servers may need additional rack space (the cabinets they keep them in), which again will need to be bought in and set-up. Plus there may be extra requirements for power and/or cooling (more servers means more heat, means more air conditioning required to keep it all cool). Certainly the latter of these would not necessarily be quick and easy to install in any data centre.

It’s also not simply a case of plugging a new server into a rack and off you go. There are all the software considerations of how does that new server extend the network, what roles does it play, what data can it access (shared in another part of the network usually), what other servers can it talk to, how has it affected the overall security surface of the network and most importantly how do you spread the game engine across it in such a way as to increase performance without introducing bugs? Once you have nailed all that you must test the whole network topology again. All of this stuff takes time to set-up, even if you have millions of dollars to throw at it.

Network bandwidth will also be a consideration. The problem with a single release date is that all users want to log-on at exactly the same time (as opposed to a phased release, such as geographically by continent for example) which means all the bandwidth you can muster might still not be enough for a single data centre. World of Warcraft had lots of problems every time a new expansion was released, as people queued up to gain access to the new world/realm. Often people had to wait hours to log-on to their server, as seems to be happening with GTA V.

Geographic diversity is obviously an option here. Have servers in Europe serve customer in Europe, the same in North America etc. This is all fine and sensible but it also increases the complexity of communication and synchronisation of these distinct server setups. You also have to add in the communication between these networks which takes bandwidth and processing (although usually this is on dedicated lines, away from the public ones). Additionally, any rollout of updates to game code is now over multiple networks.

Available bandwidth will also be limited by your overall carrier network and surrounding infrastructure. Simply buying more bandwidth is not always easy. If you need 350 Gb burst capacity to handle all your concurrent logins and the cables in the road only allow for 50 Gb you are limited until your providers can physically lay new cables.

Sales figures (and previous game popularity) should help you get most of this right though, roughly speaking. If you know you’ve sold 15 m copies on pre-order, you get your servers in order quick smart. However there are other things at play here that probably are the major cause of the recent outages.

More often than not the big group of servers in the GTA ‘cloud’ will all have distinct roles, or handle distinct areas of the world (both the in-game world and our own planet). It only takes one of these roles to get too busy (an authentication server, a map server, or a server to deliver the user data for example) and you get a bottleneck in the system that affects everything, despite load on other areas being light. This kind of bottleneck cannot always be anticipated prior to ‘real’ people hitting the network en-mass, despite all the stress tests, bots, scripts and beta testing that would be undertaken.

It’s also true that scripts and even beta testers can’t really simulate millions of ‘real’ people hitting the site with their own emotions and frustrations. If they can’t load the game straight away they might logout and login again or restart the game, which in turn could simply compound the effects they were already seeing by adding further load to the servers.”

Industry analysts were putting the number of pre-orders between 2.5 and 7 million across both consoles in Sept and suggested that GTA V would make $1 billion in the first month… it did that within the first three days (~17 million copies sold in 3 three days instead of the anticpated 30 days based on my back of the envelope calculations); is it so surprising that Rockstar weren’t able to please everyone on the day of the GTA online launch, with the huge demand and the reality of what’s involved in upgrading a server farm?

The truth is we’ll probably never be able to truly answer that question; we don’t have an exact figure for pre-orders that Rockstar were working with and we don’t know their internal sales projections and their anticipation of how many people who bought the game would attempt to access the online portion upon launch. The important thing now is how quickly the service stabilises and the ongoing support from Rockstar. This evening a title update has been released for PS3 to hopefully resolve ‘commonly reported issues’ and the Xbox 360 update is coming shortly.

Let us know your thoughts in the comments, on Facebook or on Twitter @CodecMoments.