Developer Perspective: Test Automation For Multiplayer VR

In virtual reality bugs can create a less than pleasant user experience due to the player being heavily immersed in the game when they occur. Without knowing the root causes of the issues it’s a hassle to attempt to fix them, and figuring out reproduction steps can be very stale and frustrating. This is especially true in multiplayer games since some of the bugs can only be triggered in very specific situations.

While testing a VR game can be challenging for many reasons, it’s also more fatiguing work for testers than traditional games, and running multiple instances simultaneously can be very difficult. This article outlines how I was able to solve VR testing challenges for PokerStars VR by developing an automated testing system, along with some added tips to help you streamline your testing process.

A camera attached to the avatar increases immersion for users, but can become an obstacle when testing in VR since the user needs to wear it to view the VR environment. To create automated tests, the first step is setting up a second camera so you can easily observe what’s happening without wearing a Rift headset.

When the HMD is detected as unmounted, disable the main camera attached to the avatar and enable the 3rd person camera with wider angle, this way, you can see what’s happening in VR space on the monitor without wearing the headset. You can also record a video of a mirrored screen which can be very useful if you decide to run automated tests unsupervised. In PokerStars VR, we set up a bird’s eye view camera which rotated around the poker table while not wearing the Rift, which you can view within the video above.

It’s time to implement your test bot. The most useful test scenarios are the ones that are hard for human testers and need to be done routinely. For example, some bugs require testers to do specific actions while another function is being executed in order to cause conflict. Unexpected disconnection from the network can happen a lot in the real world and it also needs to be tested regularly. Find what types of tests need to be automated for your project. When implementing, do not make changes to the core game logic as it defeats the whole purpose of the test, build a generic base system first, then write specific test scenarios. Down the road, you might find other test scenarios that can really benefit from test automation, but might not be able to do two different tests at the same time. For example, you could decide to do a network stability test one day, then a general gameplay test the next.

I can not overemphasize how important and useful it is to have log files. Logging stack traces and error exceptions complement each other exceptionally well with an automated test system. When you have your automated test system running, it creates a massive amount of useful data and helps analyze errors discovered from recorded video. Knowing what’s under the hood lets you find the root cause of bugs, so make sure your log file includes error exceptions and stack traces. It can also help to include custom messages when you have a bug report, but unsure of the cause. Put in custom logs where you think it’s suspicious, chances are, after running a few hours of automated testing you will have a much better idea.

Another tip I would like to share is to include whatever helps you locate the right messages from a log file. When I first received results of an overnight automated test session, I had log files that are hundreds of thousands of lines long. It was very challenging to locate the corresponding log message. One way to solve the issue is concatenating log messages and timestamps, this lets you immediately identify which log message matches the bug you found within the recording. Set up a development console that displays errors so they can be easily spotted when analyzing recorded videos.

During the early stages of PokerStars VR development, we discovered many game-breaking issues during host migration. Sometimes variables weren’t synced, RPC functions weren’t called on other clients, or objects became frozen in the middle of interpolation. It was very hard to figure out the reproduction steps for these issues; they required the host disconnecting with very specific timing. Sometimes the host migration had to happen within a less than 0.5 second timing window, this was a huge problem especially considering it is a poker game, the game had to be robust. The way I approached solving this issue was automating host migration tests. First, I implemented a host bot and a client bot. When a host bot is in the lobby, it creates a multiplayer poker room. Each client bot periodically fetches existing room lists from the server and searches for one marked as an automated test room. When a test room is found, the client joins the room. After playing for a random amount of time, the host bot leaves the poker room and returns to the lobby causing host migration to occur. After the host migration happens, the next host will repeat this behaviour. Once a bot returns to the lobby it repeats the test loop. During an automated session, if the poker match state has not been changed for a certain period of time, or if one of the bot players has become unresponsive to RPCs, the game is considered broken. If such a game-breaking issue is detected, all bots leave the room and go back to the lobby.

After work hours, we run automated tests before leaving the office, traditionally lasting about 16 hours. When 8-bots participate during these sessions, 7-host migration tests will be performed every time a new multiplayer room is created. This allows us to do a few hundred host migration tests overnight. Everything that happens during a test session gets logged, recorded, analyzed and discussed how to be resolved.

After we started running automated tests, we were able to go from severe, game-breaking issues taking place every host migration to 0 in a matter of a few weeks. The test system has also been serving as a tool that confirms new features are safe to be released. With the automation system generating useful debugging data, PokerStars VR has been freed from relying heavily on manual internal testing as well as external bug reports.

Automated testing can create tremendous value for your game. It allows QA testers to spend less time blindly finding reproduction steps and stay productive on other work streams. It can find issues that would not be easily spotted. It is a great tool for finding existing issues, but also for verifying that changes made to existing code do not cause further issues. On top of that, instead of relying on users sending bug reports, it generates valuable test data all on its own. Testing in VR is challenging, but I hope this post will help you assess the opportunity to save time and effort in the future.

...and be sure to enjoy PokerStars VR for FREE on the Oculus Store!