At Solace State, What Does a QA Tester Do?

Solace State is a visual novel that requires a lot of testing, due to its complex narrative, custom 3D text-in-the-environment system, and trippy sci-fi camera transitions. Less than a week before the game’s release, Kas Millard shares the process.

Long time no see! With days until Solace State’s launch date on September 14, we are trying something new with reaching out to folks across social media and even to our mailing list. This includes trying Ghost, a mailing list publishing platform! Let us know what you think about our longform content in the form of these newsletters here, because a few of us on the team love writing and sharing behind-the-scenes content.

Solace State is a game about empowering friends and hacking the system of a hegemonic biotech conglomerate. It’s also a visual novel that doesn’t look like a visual novel. This is because the 3D text system and its trippy sci-fi camera transitions are custom-designed for the game. And this means that we have a lot to quality control and test for, especially for a small team!

Kas Millard (she/they) is Solace State’s QA Tester and has done some heavy lifting to make sure that everything from the right characters show up on screen to that the music gets played at the right volume! For a game with 38 endings, 31 fully illustrated characters, and over 200k words, this is no easy feat! Without any further ado, Kas shares their insights below about what makes QA so important for a game of this scale.

Enjoy,
Tanya Kan
Director & Exec Producer

---

Quality assurance is one of those things that tends to be misconstrued by those not familiar with the work. Every QA tester has, at some point or another, tried to explain their job to someone not in the industry and was inevitably met with the dreaded response of:

“Oh. So… you just play video games all day?”

We’re definitely playing more video games at work than your average nine-to-five, but that’s not saying much. It’s telling that one of the first things told to prospective game developers is, “Don’t go into game development unless you like looking at spreadsheets.” To some extent or another, that’s true. Game development and spreadsheets go together like peanut butter and jam. And that’s true of everything from budgeting to tracking bugs to even keeping our scripts well organised!

QA is not one-size-fits-all

QA isn’t the sort of thing that is a one-size-fits-all solution. Each game studio will have different needs, as will each project, from company-specific protocols and policies to particularly egregious bugs that require testers to completely redefine the way they’ve been doing things. Working in quality assurance means adaptability. QA’s job isn’t to fix bugs, it’s to find and report them, so it’s important that we do our best to ensure that we do our best to work with the rest of the team and what they need from us. Working on Solace State was no different.

While some issues could be easily logged and reported to the team, other issues were not quite that simple. Sometimes issues only occurred under a very specific set of conditions, requiring a lengthy set of instructions on how to reproduce the issue. Occasionally, it even requires video footage of the issue happening in real-time! On occasion, a seemingly new issue would appear, only for it to be a previously-resolved bug that had reappeared elsewhere due to a change to the code. In those instances, it was important to find the old report and link it to the current issue so that the new fix didn’t accidentally undo the previous one.

But Solace State is a visual novel that doesn’t look like a visual novel, and it also doesn’t have routes that completely branch off from the overarching city-wide conflicts and story arcs. With that came unique problems that required equally unique solutions.

When I first started working on Solace State, we were still a long way from what would end up as the final, shipped product. Much of what would end up in the final version of the game had yet to be implemented; from complete visual rehauls of Solace State’s original demo to massive story moments with choices that would affect the whole game. For instance, Sueli – who would quickly become one of my favourite characters – didn’t even have a finished romance route yet! So how do you test a game that isn’t even finished yet?

Testing from big questions to small

Well, my first task was to test the basic functionality of the game – especially the core fundamentals that would require extensive fixing because they don’t work with certain inputs and impact the game regardless of the scene. This was the stage where we focused on making certain that anything that would come next would have a solid base to be built upon. It is so crucial to have testing early on because of this!

This was the time to be asking the big, important questions. Can the game run from start to finish? What happens if I press the button that the game wants me to? What happens if I pressed a button that the game didn’t want me to? That last part was my favourite.

QA often requires you to play the game like no one probably will, but what if someone does play like that and breaks the game? I remember one of the first bugs I discovered was when I repeatedly pressed the interact button while the game was starting up, players would skip over the main menu and immediately start a new game. Our Generalist Developer, Seamus Ly, would later ask me why I was pressing buttons when there was nothing on the screen, and I just laughed and said it was because I got bored while waiting for it to load and that players would likely feel the same.

From there, testing gradually became more specific. Was continuity maintained between scenes, especially when players made choices that altered the story? Did every aspect of every system work as intended? Were the intentional functions of those systems properly communicated to the player? Were those functions communicated well? If this was someone’s first-ever video game, would they know how to play it?

This was also the stage where new content was gradually getting added, which came with a twofold problem. First, all that new content would have to go through all the previous checks. Second, and perhaps most frustratingly, the solutions to the previous issues would have to be checked to ensure that they still worked with the addition of new code. This often meant that the issues we had long considered resolved would reappear once again.

Tracking issues

To help document all these issues, the dev team utilised a project management tool called Trello. With the help of developer Ryan Miller, the dev team adapted Trello plug-ins so when we experienced a bug in-game, we could export out our log debug files as well as a screenshot that would then be uploaded online.

Think of the log debug file like a grocery list of code, with a full written list of things that happened while the game was running as well as any errors. This log debug file is custom written by our development team so that we get the information that we want. We could then look at all the reported issues from the team, and add in any missing details, additional notes, or things like video footage and more screenshots and put them into different categories for different developers to address. Sometimes there would be an issue with a character’s art, so those issues would be filed under the appropriate art category to be fixed by one of our artists. This keeps everything organised and helps to play to every developer’s strengths.

As these bugs and issues got patched out, they’d be sent out to be confirmed as fixed by quality assurance–that’s me!–who would run tests to see if the issue had been fully resolved. If they weren’t resolved, I would make note of how and where the issue was still persisting. On occasion, fixes would be correctly applied but sometimes they wouldn’t work on a practical level.

For instance, a character was too close to the camera in one scene, but the fix moved them too far back and now the perspective of the scene doesn’t work. Or, after making a change to an object, the object would go back to its default state… which happened in this hilarious-looking character art bug of the one and only Alden Aldridge himself.

Scope and constraints on testing

Still, if issues couldn’t be fixed, whether this be because of technical restraints or time, we would have to sit down and consider a new approach. Then it was a matter of rinse and repeat. On paper, it doesn’t seem like too much work, but I think it’s important to remember just how much content Solace State contains.

Some estimates place the average length of a novel at around 50,000 words. If we’re just talking about what’s in the final, shipped version of Solace State at over 200,000 words, we’re looking at over four times that! That’s not counting the in-game codex (it’s a dictionary of names and terms), either! And all the additional documentation? The scripts? All the documents containing all the planning? Every single piece of writing that’s behind the final product? We’re probably looking at double even that number. That’s enough to be giving several multi-installment book series a run for their money. It all adds up very quickly.

For every page of content that ends up in the game, there’s easily three that didn’t and an additional five pages of documentation, planning, and drafts.

It’s very important to take that into consideration with game development. What may seem easy at first may not end up that way. That’s important for every member of the development team, and that was something I had to remind myself of when I was making suggestions to the team about features that just weren’t working in the final product. We had a small team, and that meant limited resources. If I discovered a problem, I would both have to consider how and why that problem occurred, as well as what the solution could be.

While the ideal fix for a problem might have been to rebuild the system from the ground up, sometimes there just isn’t the time or the manpower for that. It’s easy to think of what we could do if we had an infinite amount of time and just as much money, but it’s less easy to be aware of bugs on a technical scale while also managing player expectations, staying within budget, and also making certain members of the team can hit their respective deadlines.

So how can we fix the problem?

Well, let’s look at it a different way.

What is the problem? Is it a problem on a technical level? As in, is the problem actually one that is purely just based on code not working properly? Or is it a problem because it’s working as intended, it’s just that the intended function is unclear or otherwise not communicated well? Can we solve the problem by adding an additional line to the tutorial instead of completely redoing the entire section?

An intro on accessibility

Another thing I had to test for was accessibility, and accessibility is a much broader category than one would think. There’s the traditional accessibility that people tend to think of–audio, visual, and physical impairments–but there’s also accessibility in the sense of the base level requirements necessary for even able-bodied individuals to play comfortably.

This meant that accessibility checks ranged from checking for colour blindness friendly colours options to different fonts options for those with dyslexia to making menus autoscroll so players didn’t have to manually scroll every time. This was an extensive process, and it’s one that we look forward to talking about in a different blog post!

Solace State is multiplatform, and that means taking the technical limitations of each platform into consideration, as well as each input method and the standards and expectations set by the average player on each platform.

In conclusion

So, yeah, as one of many crucial steps, we are playing games for a job. But we are also editing, we’re documenting with clear and technical language, and we’re providing the back-end support so that others can spend more time doing what they’re good at. For a project like Solace State, that’s a lot of work, but it’s work that needs to be done, and I can’t say I didn’t enjoy doing it.

At the end of the day, game development is about picking and choosing your battles carefully.
Sometimes the easiest answer isn’t always the correct one, and sometimes the correct answer isn’t always the easiest one.

That doesn’t mean that the answer still doesn’t need to be found, and for anyone looking at working QA in games, I wish you nothing but the best of luck.

Kas Millard (she/they) is a Toronto-based game designer and freelance artist currently working as a QA tester and Social Media Manager on Solace State. You can see Kas’ website here.

---

Want to read more deep-dive game development content? Subscribe to our newsletter here!

---

Solace State is a cyberpunk visual novel where you play the young hacker Chloe who confronts political plots as she fights for her friends and her neighbors. Your choices in building up relationships and communities can revolutionize into more or less freedoms.

Solace State is available NOW on Steam and Xbox! Please buy or wishlist today!

This devlog post and Solace State development and marketing in 2023 is made with the support of Ontario Creates