"We just finished level three and need to tighten up the graphics a little bit," says one of them. He turns to his friend, smiling like he just won the lottery. "Hey, I can't believe we got jobs doing this." "I know," the other guy says. "And my mom said I would never get anywhere with these games."
For a very long time, that's how people have imagined the life of a video game tester, not as 9-to-5 job but as the fantasy of teenagers everywhere. Who wouldn't want to sit on a comfy couch and play games all day, taking the occasional break to tighten up the graphics on level three?
Reality is a little different. Video game quality assurance (QA), as testing is called, is often perceived as "playing games for a living" but might better be described as breaking them. It's a low-paying, occasionally rewarding, often frustrating job that has both more and less to do with the quality of today's games than you might expect.
A professional QA tester doesn't just sit by the television, crack a Mountain Dew, and saunter through level 5 of the latest shooter; he or she spends 14 straight hours running into different walls to see if they're all solid. Proper video-game testing is more akin to abstract puzzle-solving than it is to getting a top score in Donkey Kong, despite what you may have seen in college commercials like Westwood's. "It takes a very specific attitude and outlook to really be good in the QA world," a veteran game tester told me. "It goes beyond a passion for video games, and definitely beyond the notion that you get to play video games for a living."
Game testers are, by nature, underappreciated; they are only noticed when something goes wrong. QA veterans say the job is stressful, tedious, and often seen as a doorway to other parts of game development rather than a viable career path. Often, testers work on temporary contracts or for outsourcing companies that don't let them communicate directly with a game's developers. And when a game is particularly buggy or broken---as many recent releases have been---it's customary for observers to blame QA. They are the safeguards, after all---the final wall between a programmer's mistakes and a customer's money. It's in the name: Quality Assurance. They're supposed to assure quality.
But when a big game ships broken, is QA really to blame? How could testers possibly not find some of the bugs that show up in the games we play? Why do so many servers break all the time? Just what do QA people do all day, anyway?
Over the past few months I've had extensive conversations with several dozen current and former QA testers---many of whom spoke anonymously in order to protect their careers---in an attempt to explore the world of video game testing and try to explain what it's really like to play games for a living. Some said they hated working in QA; others said they couldn't imagine doing anything else. Almost all agreed on one thing: not a lot of people understand how QA actually works.
For as long as there have been video games, there have been bugs. Some are relatively harmless and have even attained mythological status, like Pokémon's enigmatic Missingno. Others are seared into video game history, like Minus World, an unbeatable Super Mario Bros.level that you can only access by glitching into a wall. Many bugs have been discovered, abused, and enjoyed by the assiduous speedrunning community---how else would you beat Ocarina of Timein 17 minutes?
Those are the friendly bugs, though. Most video game glitches are irritating at best and game-breaking at worst, which is why every game goes through quality assurance, an extensive testing process implemented to ensure that everything's working properly. The term "QA" draws from the world of products---microwaves, cars, assembly lines---and in many ways video game testing is no different. A tester's job is to poke, prod, and play the hell out of a game until all the kinks are gone, like a factory-worker smoothing out the latest toy.
There's no game industry standard when it comes to QA procedure---every game is different and every company has its own process---but a tester typically spends months playing builds of the same game over and over again in all sorts of different ways. The more bugs a tester finds, the higher his or her perceived value to the company. This is one hell of a challenge, of course: video games are complicated sets of interlocking systems that require careful, meticulous bug-testing, which can involve testing the same level repeatedly with slight variations---using a new character, wielding a different weapon, following an alternate pathway---and recording everything that happens.
Take Grand Theft Auto V, for example. On Rockstar's giant open-world game, QA testers had to divide and conquer. "You would have individual testers assigned certain missions, or tasks, mini games etc.," said one person who helped test the game. "Normally starting with the big stuff and working down. So doing story missions in order, then heists, then side missions and random characters until you moved on to testing the strip club and prostitutes."
Sometimes, that tester said, they'd also have to devote tons of time to granular parts of the game, like when Rockstar's designers asked a group of QA staff to test everything players could do with the game's automated taxi service. They quickly found that taking a taxi to a new mission would trigger the mission without properly disposing of the cab, leading to some amusing moments as a taxi drove around and tried to back up during cut-scenes.
"I think working on a project like that is made a lot better by the small moments where something truly stupid happens," the tester said. "Talking pigs randomly standing up like a person and walking away, randomly being shot out of the sky in a plane by an ambient pedestrian whose physics had fired him into space. Trevor pulling his trousers down then never animating to pull them back up and spending the rest of the entire game with his trousers around his ankles. Franklin's dog used to instantly die if he touched water... he'd just fall into a pool and sink to the bottom like a rock as soon as his paws got wet."
Finding bugs is just the first step---the second, significantly harder process is trying to reproduce glitches so the company's engineers can zap them. A tester can't just write down something like "Trevor's pants won't stay on" and send it to the programming team; what could engineers possibly do with that information? In order to track down, isolate, and fix a bug, coders need to know exactly how it happens, which can be a tricky riddle to solve given how many variables are in video games. Good QA testers quickly learn to keep track of every action they take---from big to small---so they can at least try to reproduce any bug they stumble upon. "I love that working QA is often like being paid to solve puzzles," said veteran tester Rob Hodgson, who's done the job for eight years now. "Figuring out how to reproduce that bizarre error you encountered, step by exacting step, is thrilling to the right kind of person."
A typical day for a QA tester could vary drastically based on the project, the role, and the position. An outsourcer might have to spend 10 hours slamming into every wall in the latest Call of Duty to see what breaks ("collision testing"). An embedded QA tester might have to work with a programmer to find out why the framerate keeps dropping on the Android version of their new mobile game. The varied and monotonous nature of QA can lead to some unexpected challenges; for example, one tester who worked on the rhythm gameRock Band said the clack-clack-clack of plastic drumming could be so maddening, they had to set a rule: no instruments on Tuesdays and Thursdays.
As they play, testers have to type up bug reports, using software like JIRA to explain what happened and how it happened. Programmers---who ideally at this point are no longer working on new content and are exclusively fixing bugs---analyze the reports and respond as necessary, sometimes with questions, problems, and occasionally snarky comments.
When a console game is almost done, it has to go through certification, a process in which the publisher (like, say, EA) will ask the console manufacturer (Sony, Microsoft, or Nintendo) to check the game for game-crashing bugs. During this certification process, a second layer of QA staff called compliance testers must go through the game with their own fine combs, checking to see if everything lives up to expectations. Every console-maker has its own stringent checklist dictating everything from error messages to achievements, and if a game doesn't make the cut, the publisher will have to fix things and try again, deadlines be damned. "Microsoft required all games to be able to access the Xbox 360 menu from literally everywhere in the game," said one tester who worked in compliance for a major game publisher. "Sony required that you couldn't skip the first time you see the publisher/development studio screens at the beginning of the game. Nintendo didn't want any swearing in their games so every text input had an extensive filter we had to try and break."
The Telltale Speedrun
"I didn't play BioShock Infinite for at least two years after its release," an ex-QA tester told me recently. He'd worked for 2K and spent extensive time testing outInfinite, and he wasn't really happy with how the game turned out, noting that it didn't live up to the original.
"The one thing that got me to play the game again was seeing speed runs of BioShock. [We'd spent] a lot of nights speed-running the game. It was interesting to see what the speed-runners used to cut levels."
One night recently, the tester saw this video of a speedrunner breaking the game's world record. When he got to the 11-minute mark, he started freaking out.
"I got so mad because they use a bug to get outside the level and automatically move forward," the tester said in an e-mail. "I SHOULD HAVE CAUGHT IT!"
For a long time the video game industry presented QA testing as a dream job: hey kids, come get paid to play games all day! But in recent years, horror stories have emerged---QA testers tell tales of monotonous work, grueling hours, and poor treatment from companies that see them as replaceable cogs in the development machine.
There's the money problem, for example. The job has low requirements---typical entry-level tester positions don't need much experience or even a college degree---and there's a ton of demand, so the pay is mediocre. A 2014 Gamasutrasalary survey pegged the average QA tester's yearly pay at $54,833, but that only reflected full-time staff; most testers are contractors working either directly with developers or at third-party QA houses that juggle multiple publisher clients. Many of those contractors told me their salaries ranged from $10 to $15 an hour---that averages out to $21-31,000 per year.
Testers also talk of feeling disrespected at the workplace. Many in QA, especially contractors, have told me they weren't allowed to speak directly to the developers, and that their only communication came in the form of written bug reports. "It was also sort of an unspoken rule that temps shouldn't directly contact the devs," one tester told me. "Any communication was typically routed through the full-time QA Leads. As testers, all our interactions with dev were through comments in the bug database, which is far from an ideal form of communication. It was easy to interpret a developer's comment/question on a bug as snarky/irritated when that wasn't necessarily their intention."
This isn't the case at every studio---"When you give your testers benefits, upward mobility, job security, and respect, it attracts the right people for the job," said Ariel Smith, a tester at the MMO studio Cryptic who told me she loves the gig---but disrespect for QA is certainly a widespread trend. Several testers told me they've had to use side entrances to enter their offices and that they weren't allowed to mingle with the rest of the staff. Others said it's common for other developers to screw with them in all sorts of ways; one frequent story, for example, is of the engineer who fixes a bug yet continually sends QA messages like "could not reproduce." In a typical studio, QA is seen as the bottom of the totem pole. This is in part because of the nature of their job---a tester's role is to show other people where they screwed up. That's always going to bruise some egos.
"The QA only care about finding bugs; the developers only care about fixing them," a former tester told me. "They aren't a team and they aren't working together. It's almost like a game of tennis. The testers actually want the build to be broken because it ensures they have work to do. So, the two sides are kind of working against each other, which isn't healthy for production."
At some game companies, higher-ups give testers strict bug quotas and threaten to shorten their contracts if they don't find enough glitches, which can result in a weird sort of tension as testers compete over who's finding the biggest bugs first. Sometimes, QA employees will find creative ways to work more hours so they get paid more and make themselves seem more valuable to the company. "There were some testers who would hold onto bugs to make sure there's overtime," one tester told me. "If there was no OT scheduled for the weekend they would enter a [major] bug Friday afternoon. In some cases this would cause overtime."
QA testers also have to deal with game development's other systemic issues---particularly, mandatory crunchand frequent layoffs. Big developers tend to hire dozens of QA testers toward the end of big projects only to let them all go once the game ships. Instead of partying with the rest of the developers, they're out looking for new jobs.
This all adds up to what would seem to outsiders like an unquestionably heinous gig, but the job does have bright points. A number of current and former testers have told me that despite QA's many challenges, testing video games can be rewarding and educational in a unique way.
"I've enjoyed my time as a tester and I would do it again if I had to," said Obed Navas, a former tester who worked on games like BioShock andCall of Duty. "Even though 'QA Tester' might not be the most glamorous title, and you run the risk of losing the passion to play at home, in the end, being able to see your name in the credits, and having specific items on you that people know you can't find anywhere ...and them asking you where you got it, being able to respond 'I worked on it,' it's a great feeling and I take pride in that."
Lunchtime Woes
Rob Hodgson, a video game tester, was working on an early build of the multiplayer game Fallen Earthduring lunch one day when suddenly the servers crashed. Not long afterwards, they crashed again. Baffled, he tried to reproduce the bug, but he had no luck---neither he nor anyone else on his team could figure out what was causing such a catastrophic error.
"It made no sense," Hodgson told me recently. "We got one of the programmers to watch the logs as it happened, and he all but shrugged and threw up his hands. It seemed like someone had run off the edge of the world, and the servers were choking, trying to track him. But no one on the [quality assurance] team was testing the world edges, and the designers were all out to lunch!"
After scratching their heads for a while, Hodgson and team asked around and eventually found out what was bringing down the servers. "Turns out that the designers going out to lunch was precisely the problem," Hodgson said. "One of them had just been setting his horse on autorun and going out to eat. Some days it would hit a tree and get stuck. Some days it would get into the endless, flat plains beyond, and melt the servers."
From Assassin's Creed Unityto the PC version of Arkham Knight, today's games seem to be shipping with more problems than ever before. It's easy to pin the blame on QA. Sometimes, that very well may be the case---a number of veterans told me that they've worked alongside sloppy, apathetic testers who just didn't care enough to do things right. Two different former testers told me they'd watch colleagues smoke weed during lunchtime just about every day; one said the QA staff would all wear sunglasses in a futile attempt to hide it.
But lots of testers say they're findingmost if not all the bugs that ship in today's games. The problem is that nobody's fixing them.
Most testers operate using a process called triage, where bugs are prioritized based on importance. Top priority are issues that make the game crash---the "showstoppers," as they're called. Other glitches are categorized based on how important testers think they are. Usually, producers and programmers will take the time to fix showstoppers---it'd be hard for any game to make it past certification with any of those. But small and even moderate-sized bugs often stick to games like barnacles, the victims of tight deadlines and programmers who can only do so much in the time they're given.
"We would often find that either the risk of fixing a bug or the time it would take weren't worth it, especially when we were doing something really uncommon or deliberately breaking the game," said one tester who worked on The Elder Scrolls V: Skyrim, a game that's known for being both extraordinarily massive and uncommonly buggy.
"Some of our bugs would get resolved as 'won't fix' or 'post-release.' Post-release meant, 'This would be nice, but it isn't really needed right now. If it becomes a thing people are calling out on the forums or online, maybe we'll reassess this.' I think there's a lot more leeway nowadays than there was maybe 20 years ago, [because] you can say, 'OK we have this nasty bug but it can be fixed in the day-one patch.' That's acceptable behavior now."
As games get bigger and bigger, making leaps in both graphical fidelity and mechanical complexity, they're generating more and more bugs---bugs that might not be so easy to catch or fix. Development studios can't just change a schedule because they're overwhelmed by glitches. Unless the publisher agrees to what could be a costly delay, that holiday release date ain't going anywhere, no matter how broken the game might be. So during testing, several QA staffers told me, they focus just on catching the bugs that would prevent them from getting through console certification. That might mean ignoring some of a game's other, possibly more significant problems.
"The game developers and the publishers had independent QA teams," recalls one compliance tester who worked for a major publisher. "We would submit a bug with as much detail to recreate the condition as possible, the development team would determine if it was really a bug or not, and then they would either invalidate the bug or attempt to fix it. We would receive a revised build with 'X' number of bugs handled, double-check the problem was fixed, and then move on to the next thing. This aspect of the process, from a gamer's perspective, was the most frustrating. Bugs that caused inconvenience for the player were often considered invalid because they wouldn't affect the ability to release the game, and could be addressed later if people got upset."
Other compliance testers say that, due to deadlines, they've had to find loopholes or do the bare minimum necessary to sneak through certification and start printing discs, promising that a day-one patch would come on release day.
"If we were getting down to the wire and multiplayer was super broken, we'd focus test efforts on single-player/offline areas instead," a tester told me over e-mail. "[Console-makers] would often fail games for severe issues in online functionality, but still allow us to release to manufacturing on the condition we provided a day-one title update to fix issue(s) X, Y, and/or Z. There wasn't a way to guarantee all players would get a patch for single player/offline bugs since offline gameplay wouldn't require an internet connection. So we'd tighten up offline content for final certification, knowing there were issues with online content, and then scramble to patch the online issues in time for launch. This was (unfortunately) a rather common process, and was usually due to a rushed & unrealistic timeline rather than dev/QA incompetence."
Sometimes, bugs will make it into a game for reasons that seem baffling to outside observers. Writes one tester who did contract work for Nintendo of America: "The Sky Drop bug in Pokémon Black and Whitewas something that we caught, but the game had already been released in Japan at that point so instead of fixing the issue they purposefully left it in the game to keep parity." The tester added that despite these idiosyncrasies, Nintendo makes an explicit point of recording every single bug and asking testers to include video footage for every report they submit. "I think this contributes greatly to their reputation for having quality games," he said.
One other compliance tester's explanation for the increasing complexity of bug-testing might make gamers angrier than ever at publishers: it's DLC, he says.
"Let's say we have a game with 40 pieces of microtransactions/DLC," the tester said in an e-mail. "Even if some or all these 40 DLC packages are simple unlocks of on-disc content, we have to test every possible combination of these 40 DLC packages, in all game modes, using different combinations of storage media (20 DLC packs on the hard drive, 15 on a USB flash drive, 5 on a memory unit, etc), mixing up what the host has versus what clients have in different online game modes, verifying graceful handling in the event some dipshit has all his DLC on an external storage device and decides to unplug that device, with each Title Update (1.01, 1.2, etc), with different save data, etc., etc.
"It could (and did) get the point where it's literally impossible for humans to physically test all possible combinations/situations. It's something that would get exponentially more difficult as more variables are added---like if you can purchase DLC in-game."
In short, this shit is tough. Multiplayer games are especially difficult for today's QA departments to handle; even hundreds of testers can't successfully replicate what will happen when hundreds of thousands of people are all shooting aliens on live servers. And while some testers say developers are working more closely with QA than they have in the past, nobody can change an unrealistic deadline.
"Milestone schedules nowadays are totally absurd," one former tester told me. "What [a] lot of people probably don't understand is that even for titles that have been in development for, say, three years, only 1.5 of those years is actually full, proper production. And of that, only nine months in QA. And of that, only three months at full QA capacity. By then, we might be at content lock. And so by the time we are able to speak out, the game is reaching Beta. This is a bit of an exaggeration for some larger titles, but reality for many I've worked with. When QA speak, we are heard... there's just no time to react."
'No Hard Feelings'
Writes one former tester:
We were bundled away into bullpens. I was working with people that had been testing [game] for years and years, were so burnt out on it, but were still going because they had no other hope. Out of the probable hundreds of testers they go through a year, I think maybe two or three ever got hired on. They weren't acquiring any new, valuable experience or skills during all that time. We got condescending emails saying "Congratulations! You've just earned eight hours of vacation time!" like we were kids getting stars for good behavior. I don't need that. Just put it on my paystub.
Meanwhile, new cars were showing up in the lot every couple of months. Ultra fancy, decked out cars that probably cost more than four or five testers' annual income. They were throwing parties for themselves. When you're grinding away and can't afford new clothes regardless and you see that sort of thing, it takes a toll. When we launched, there was a massive round of layoffs. I think close to around 100, but I have no idea. All I know is, QA was pretty much wiped to bare bones. A few hours after they were all escorted off the campus, the launch party kicked off.
Things got bad for me in a lot of ways; a girl I was seeing at the time [left me]... I met a lot of good people that knew how terrible the situation was for us temps, and good people were leaving because of it. It frustrated everyone I worked with. I got bumped around from team to team. That last team was small with way too much work, but I learned a lot and they were really good people. When I told them I was leaving, there were no hard feelings. There was almost a hint of encouragement, like I was telling them I was going to quit drinking or something beneficial like that.
Is QA a viable long-term career? Or will it forever be a means to an end, a pathway to other, more interesting parts of game development?
While reporting for this story, I talked to around 60 QA testers, and of those, only four said they saw QA as a long-term career. Many used QA as a way to segue into producer or designer jobs at game companies; others tried it for a few months or years and then gave up, moving on to more lucrative fields. Those who do stay in QA and succeed in the role can find themselves moving into QA management, a field that's more about handling other people than it is helping make video games. That's unsatisfying for some folks, especially given that the stereotype of testers being young, immature, and unkempt is often true.
"As you move into the role of QA Management, you start to have experiences that nobody could ever prepare you for," said one veteran QA lead. "Testers showing up in pajamas for work, testers sexually harassing one another, testers getting caught stealing... Even the most mundane thing like ordering in food can become a challenge. QA is a field that skews very young. Having the average age of a team be 18-19 is not uncommon, and often times this is their first real job. It's a massive challenge to deal with young personalities and hormones."
Several experienced game developers have told me there are few employees more valuable than a senior QA person---someone who knows how to properly write bug reports; who knows to focus on big-picture issues instead of some clipping or graphical issues; who knows how to work with other people to maximize efficiency and avoid dupes. Those people are rare---low pay and poor treatment combine to drive many QA testers away from the field before they get very far.
"Before I left, I sat down with my manager, the associate producer on the game, and asked him [for] advice," said James, a former tester who asked me not to use his last name. "I wanted to know how to make the jump from temp QA to permanent QA in development or anywhere else. He blatantly told me, 'Distance yourself from QA... the sooner you can get out of QA and distance yourself from ever working in it the better.' After about another eight months, I left QA for good and gave up on working in the video game industry. I am now working at a developer of commercial software, and life is a little less fun, but much more stable. The money is also much better."
Based On A True Story
The movie Grandma's Boy, which came out in 2006 and stars Allen Covert as a video game tester, is something of a running joke among QA. People always ask testers: how much of Grandma's Boy is true? The answer is usually "not a lot."
One scene in the movie, in which the egotistical video game designer J.P. looks over a failed prototype and watches his main character's head fall off, is often brought up as indicative of the movie's inaccuracies (skip to 1:31):
Silly, right? Bugs don't really happen like that. Except for when they do.
"It was in a certain Kinect game that involved obstacle courses in a TV show style presentation," a tester for a major publisher told me. "There were quick intros of the show's announcers at the beginning of each level where they'd provide commentary and set the scene. Once, and only once, one of the reporter's head shifted off his neck and into empty space next to his body - then dropped straight down off of the screen. Nobody could ever reproduce it, and trust me, we tried.
"It was surreal - here I'd developed a distaste for Grandma's Boy due to its inaccurate and somewhat trivializing take on my profession ('Did that guy's head just fall off? That shit doesn't HAPPEN in real games!'), and it was being proven right in a small way. I'll never forget it."
Fantasies like Westwood College's are a far cry from the real world of video game testing. In real QA, the pay is low, the work is tough, and many testers find themselves in unenviable positions, outsourcing or doing contract work for game developers who don't even know their names. (Writes one tester: "I was once sent an email from HR asking for input on planning the company's holiday party---I responded with some feedback and was then told that temporary employees weren't invited.")
Even those who found QA to be a satisfying, rewarding job say things could be better. Way better. A widespread push for change could lead not just to healthier work environments but to more experienced, efficient testers who work directly with game developers to ensure that more bugs get fixed.
"Game testing is, in general, a job of unsung and untrained heroes," reads a report by an old QA company called ST Labs. "For some companies, a typical game testing strategy still consists of throwing sheer numbers of testers on a game as it nears release and hoping that they find the big problems. Understanding the games issues and technology and going about game testing in a systematic and thorough manner is still a new concept for most game development companies."
That report, written by Duri Price and Ilya Pearlman, came out in 1997. Eighteen years and four console generations later, not much has changed. The good game testers are still unsung and untrained heroes, doing their best to get games fixed despite unreasonable schedules, unmotivated co-workers, and a job that's unappreciated at best.
There's another way of looking at it, too.
"You *will* get stuck on shitty games, you *will* get stuck with shitty people, and you *will* get stuck with a Producer/PM that thinks QA Testers are *shitty* and beneath them," one veteran said. "You will work long hours, be underappreciated and see your amazing bug that you spent six hours working on be swept aside and labelled Known Shippable. But you will also see your name in the credits, you will also realize that you contributed to something really cool, and you will also move on to another amazing project."