DepriveClub

Evolution Studios’ DriveClub has stalled upon launch and the PS+ version has been indefinitely delayed after it saw “a lot of activity and new social behaviours right now” that ended up “pushing the servers to their absolute limits.”

Post by Driveclub.

Flashback to just over a year ago when GTA: Online was launched and we discussed the strained servers with industry expert Chris Gill:

“Rockstar Games will operate their own server network, basically the GTA ‘cloud’. If that doesn’t have enough capacity they will either need to upgrade the hardware (more memory/CPU etc.) or more likely just buy more servers to increase it. This hardware will need to be ordered, delivered, plugged in, installed and networked. The additional servers may need additional rack space (the cabinets they keep them in), which again will need to be bought in and set-up. Plus there may be extra requirements for power and/or cooling (more servers means more heat, means more air conditioning required to keep it all cool). Certainly the latter of these would not necessarily be quick and easy to install in any data centre.

It’s also not simply a case of plugging a new server into a rack and off you go. There are all the software considerations of how does that new server extend the network, what roles does it play, what data can it access (shared in another part of the network usually), what other servers can it talk to, how has it affected the overall security surface of the network and most importantly how do you spread the game engine across it in such a way as to increase performance without introducing bugs? Once you have nailed all that you must test the whole network topology again. All of this stuff takes time to set-up, even if you have millions of dollars to throw at it.

Network bandwidth will also be a consideration. The problem with a single release date is that all users want to log-on at exactly the same time (as opposed to a phased release, such as geographically by continent for example) which means all the bandwidth you can muster might still not be enough for a single data centre. World of Warcraft had lots of problems every time a new expansion was released, as people queued up to gain access to the new world/realm. Often people had to wait hours to log-on to their server, as seems to be happening with GTA V.

Geographic diversity is obviously an option here. Have servers in Europe serve customers in Europe, the same in North America etc. This is all fine and sensible but it also increases the complexity of communication and synchronisation of these distinct server setups. You also have to add in the communication between these networks which takes bandwidth and processing (although usually this is on dedicated lines, away from the public ones). Additionally, any rollout of updates to game code is now over multiple networks.

Available bandwidth will also be limited by your overall carrier network and surrounding infrastructure. Simply buying more bandwidth is not always easy. If you need 350 Gb burst capacity to handle all your concurrent logins and the cables in the road only allow for 50 Gb you are limited until your providers can physically lay new cables.

Sales figures (and previous game popularity) should help you get most of this right though, roughly speaking. If you know you’ve sold 15 m copies on pre-order, you get your servers in order quick smart. However there are other things at play here that probably are the major cause of the recent outages.

More often than not the big group of servers in the GTA ‘cloud’ will all have distinct roles, or handle distinct areas of the world (both the in-game world and our own planet). It only takes one of these roles to get too busy (an authentication server, a map server, or a server to deliver the user data for example) and you get a bottleneck in the system that affects everything, despite load on other areas being light. This kind of bottleneck cannot always be anticipated prior to ‘real’ people hitting the network en-mass, despite all the stress tests, bots, scripts and beta testing that would be undertaken.

It’s also true that scripts and even beta testers can’t really simulate millions of ‘real’ people hitting the site with their own emotions and frustrations. If they can’t load the game straight away they might logout and login again or restart the game, which in turn could simply compound the effects they were already seeing by adding further load to the servers.”

Of course R* sold some 15 million copies of GTA 5 in the first week of sales, so one can cut them a little slack when it comes to server load, even if they did deliberately delay the online launch to (unsuccessfully) mitigate some of these issues; DriveClub has only sold 375,000 copies in week one. Given that relatively low number vs. the 3.5 million PS+ subscribers, it really makes you wonder how the infrastructure was ever going to be able to support the game.

You can’t help but feel that Evolution Studios are a little naive to be surprised that they are “facing new challenges which we haven’t encountered as a development team before”. That’s not something we’d expect to hear from the team that delivered us 3 MotorStorm games that had a very decent online offering. Is this going to become a trend that we see more often with enhanced social aspects of games and integrated multiplayer? Or is this the first real mis-step we’ve seen from a first party Sony studio in a long while? Discuss…