14 Nov 2011

The Supermassive Beta

How many beta testers does it take to change a lightbulb?

The rather surprising answer is 89.

  • Fifty to just carry on as if light doesn’t matter and continue killing swamp rats or whatever.
  • Ten to raise a ticket complaining about the complete lack of lightbulb operation and DEMANDING that it be rectified THIS INSTANT.
  • Two to proclaim that lightbulbs themselves are overrated, and that when they started playing MMOs all they had was candles. And they jolly well liked it.
  • Five to argue that lightbulbs could actually do with a buff
  • Six to counter-argue that it’s actually daylight that needs to be buffed, and that lightbulbs themselves are slightly overpowered.
  • One to troll the entire discussion by stating that “lightblbs r for lozers lol”
  • Fifteen to give up the mission in frustration, going on a textual tirade about the pointlessness of lightbulbs and asking who designs a stupid quest around lightbulbs anyway.

And one level designer who sighs inwardly as he moves the light switch to a more prominent location.

There’s been a few eyebrow-raising incidents lately in beta-circles. Firstly there was the announcement that Blizzard were going to give everyone signing up for the Annual Pass entry into the Mists of Pandaria beta. I’m not sure how many people that’s likely to be, but I reckon it’s easily in the tens of thousands and possibly in the hundreds of thousands.

Not to be outdone, there was also the recent announcement from BioWare that everyone who’d previously signed up for the Star Wars: The Old Republic Beta would get a chance to play at an upcoming weekend test. This invite list is likely to be in the order of millions of players.

There’s a question that immediately crops up.When do you reach critical mass – the point when there are so many testers raising tickets that your QA department implodes under the crushing weight of so much data, creating a testing black hole that threatens to engulf other nearby departments?

There’s also the comparison of testing approaches. The long test where the gamer can relax and bathe in the hot, fresh waters of unseen MMO content, delicately selecting the finest quests from the fruitbowl of offerings direct from the designers. Or the stress test, where you have a horde of rampaging gamers banging on your server doors like an invasion of Justin Bieber fans armed with pneumatic drills and JCBs.

I’m reminded that these games have grown to be truly massive these days, and as a result need massive amounts of testing before they’re released on a massively eager gaming population. It’s a far cry from when getting a beta invite was like receiving the One Ring via email. You’d spend all night playing “my precious beta” before shuffling into work the following morning looking grey and mumbling incoherent babblings about shooting dragons with railguns.

The interesting bit is that testing has moved on – bug reports are a thing of the past and data mining is in full swing. BioWare has been compiling heat maps based on player data – everything from where characters die to what mobs get avoided the most. Tickets and forum posts might tell you what a player thinks, but data collection will tell you what they do. And the more players you get in, the more accurate your heat map becomes.

It’s clear that beta testing is changing to include a great deal more science and statistics. Whether we’ll continue to be wanted for our thoughts and opinions is a little more unclear. Is there a future where beta tests are measured and analysed purely by the lines of generated datapoints?

Keep testing. For Science.

Like this? Try these other related posts:

Tags: , ,

2 Responses to The Supermassive Beta

  1. Effraeti says:

    The Datamining bit is a good insight. Perhaps Blizzard expects to get more useful information themselves that way.

    On a lighter note, I signed up for the Annual Pass. Is it bad that I only did so because of the mount? I really have no interest in Diablo 3 (sacrilege, I know) and I am not planning to participate in the MoP Beta. I saw too many people who were already burnt out on the dungeons by the time the expansion released. I think I had far more fun waiting until release, and taking a few days off from work and seeing everything for the first time. :)

    Nothing against all the wonderful people who Beta test. The expansion would not have been what it was without them, I am sure.

    ~ Effy
    Effraeti recently posted..A Community of SupportMy Profile

    • Gazimoff says:

      There’s nothing wrong with signing up to the annual pass for that reason. If you’re likely to be playing Warcraft for the whole year it makes perfect sense.

      Last year I was very lucky to get into the beta through some kind people at Curse. As a result I churned out half a dozen videos, talking about some of the new content and Mage class mechanics.

      I’m with you though – burning yourself out on content during beta is a silly thing to do. Much better to test well, provide buckets of feedback and be safe in the knowledge you’re doing a good job. The thanks from players to beta testers when the game goes live in a polished state is a great reward :)