Meta Developers

Distribute

VR App Testing Guide - Automation and Performance

Testing, automation, and performance are all essential areas of opportunity to minimize iteration time and improve developer efficiency. Each of these concepts overlaps in a number of ways throughout the development lifecycle. In turn, the following guide covers both high-level concepts and specific areas of opportunity to improve your VR development process.

Test scenarios: A primer

Before we start to provide best practices and considerations for performance testing and test automation, we want to ensure that you have the foundational knowledge of the following scenarios:

Unit Tests: The smallest, individual unit of code. Examples include a game object in isolation, a specific function, method, etc. As unit tests focus on a very specific portion of your codebase, they are historically where developers start to find opportunities for automation, as these are generally the simplest form of a test script.

Integration Tests: Two or more modules of the system combined and tested, for example, testing a full environment, scene, specific interaction, etc. These need to be built and integrated into your engine and include multiple systems; hence, they will require more skill and resources to automate than unit tests.

End to End / QA Testing: Typically done manually, this historically refers to a full playthrough of your app, or specific levels/interactions from end to end. QA is typically managed by an internal or third party QA team and includes documenting a full test plan. There are ways to automate these tests, but keep in mind that automating parts of the QA process can require a high degree of skill and resources that you may want to prioritize elsewhere.

Speed up Quest App iteration time w/ Link

To test Quest gameplay and visual style, be sure to deploy your Quest app with a USB 3.0 Link cable that meets recommended specs. Connecting to your PC will save you a great deal of time as you test certain elements of your Quest app, as opposed to relying on your local wi-fi speed to deploy your build from a PC.

...But also test your Quest app in standalone mode

With the previous note in mind, PC mode does not reflect the true, standalone experience. You should not test performance or any elements of your app that are impacted by the CPU/GPU, as this can lead to major challenges, missed bugs, and extensive performance issues. Deploying and testing on your Quest should be a regular part of your testing process.

How and when to prioritize automation

Automating the test process can help you a great deal, especially in the long term, but there are a number of items you should consider before you begin to focus on automation script writing, and the technical requirements that go along with automation. See below for a few key factors you should keep in mind as you design your testing and automation strategy:

Consider time and skill: Be prepared to conduct a cost-benefit analysis of writing a rock-solid test script as opposed to using manual testing. The more complex the test, the more time and effort will be required to write and maintain.
All things considered, automation is recommended: With the above point in mind, if you have the ability to automate a test, whether something as small as a boot test, a complex interaction or full environment, automation can help your team drastically speed up iteration time and minimize the number of bugs/issues that arise in your app over time.
Automate elements that are repeatable and focused: Automate sections of your app that are repeatable and generally less complex. This will maximize efficiency and ensure your script accurately delivers the designed outcome.
Target edge case opportunities: Employ automation in those strange instances where it would require a great deal of logistical work for a single, manual test, or user flows that might be considered “abnormal”.

Testing and automation best practices

After you’ve decided what type of tests you’re looking to automate and where to focus your resources, use the following best practices to move forward with your test automation process.

Test scripts as inspiration to automate: As your team most likely already uses test scripts to drive your QA process, these can be an easy way to find those repeatable areas of your build that could be automated with minimal effort.
Use source control and continuous integration: If you are looking to publish a fully developed Meta Quest VR app, it’s recommended that you leverage a source control + continuous integration system. These come in many different shapes and sizes, and it’s recommended that you scale your system with the size of your team and type of app you’re creating. For source control systems we recommend taking a look at Git and Mercurial, or Perforce if you have a more complex app. For continuous integration systems check out Jenkins, TeamCity, or GitLab.
2D version of your app recommended: Whether you’re using test automation or not, be sure to include a version of your app that can simply be run on a 2D screen. This will save you plenty of time for testing certain features, variables, and properties, and speed up your debugging process.
Leverage telemetry when possible: This sort of data can be extremely powerful in the long term as you work to analyze what may have caused a specific error. When planning on what telemetry data to log, think about the user journey through your app and the user flows that need to be completed for a successful playthrough.
Detect build errors early: Compiling code and running the asset build on your CI system can detect issues as soon as a new change is committed. Proactively warning your team of these errors can ensure that they avoid any conflicts as well as steer clear of any potential blockers these issues might cause.
Highly recommended for performance testing: On-device automation can be especially helpful for monitoring performance. New changes can unintentionally harm performance, and detecting this as soon as possible will be much easier to fix when the cause is known. Monitoring performance over time can also ensure that your updates to optimize performance are effective. See below for more information on testing your app’s performance.
Trigger events on device: Trigger specific events on the VR headset, for example “tell character to go to X world and move character to Y position”. Examine performance characteristics over ADB and be sure to run certain checks throughout the event.

Performance testing + analysis: Tools and considerations

No matter the genre, performance optimization is critical to the success of any app targeting standalone Meta Quest headsets like the Meta Quest and Quest 2. See below for a few important considerations to remember when planning your development process with performance optimization in mind.

Performance budget: As you work to ensure your VR app meets framerate, it’s recommended that you consider documenting the different parts of your system and the performance budget you will allocate. This will help ensure clear communication and help with prioritization throughout your development lifecycle.

Performance KPIs: Along with your budget, it’s also good to set KPIs for your app’s performance, this might include specific benchmarks for time to launch, time required for a specific interactions, UI response, navigation between scenes, etc.

Thread management: While this would be considered an architectural decision, remember to place critical elements in the foreground, while less important elements take place without blocking the main thread.

Performance monitoring and analysis tools

We continue to build and improve our suite of developer tools that will enable you to monitor, analyze, and optimize your app’s performance. See below for a few highlights of these tools, while you can also check out the Mobile Optimization Tools Page for more information.

OVR Metrics Tool: Analyze your mobile app with a real-time graphic overlay which provides app and device data, such as frame rate, heat, GPU/CPU throttling values, and more. See the OVR Metrics Tool Stats Definition Guide for more on each statistic featured in the tool.
Collect VrApi Logs with Logcat: Collect system logs with this command-line, Android SDK tool. Leverage Logcat to retrieve Meta Quest VrApi logs to review app performance and device settings. See the VrApi Stats Definition Guide for more on each statistic featured in the tool.
ovrgpuprofiler: Access real-time GPU pipeline metrics and render stage tracing with this low level CLI tool. ovrgpuprofiler is included with the Meta Quest and Quest 2 runtime and lives on each device.
Perfetto: Perfetto is an engine-agnostic performance tracing tool with some major advantages over built-in engine profilers, such as the inclusion of a wide view of system processes and additional metrics on the same timeline as the app’s performance profile. It maps functions calling into OVRPlugin to the OS directly and gets insight into various GPU counters and metrics at a high level to better isolate performance issues. It can also generate GPU render stage traces.
RenderDoc: A graphics debugger tool that supports multiple graphics APIs and development platforms, it’s an ideal solution for frame capture and analysis.
RenderDoc for Oculus: A fork of RenderDoc maintained by Meta Quest for use with the Quest Platform. In addition to RenderDoc’s normal graphics debugging capabilities, this version provides access to low-level GPU profiling, specifically tile renderer data from the Meta Quest.
Unity Profiler: Build and run this tool on your Meta Quest to collect even more performance information directly from the Unity environment. See the Official Unity Profiler Documentation for more detail.

Get started with device-based testing

Whether you have a few headsets available to use for continuous automated testing, or you want to get creative and leverage a spare headset to run tests overnight, starting a device lab of any size can help you find bugs and performance issues earlier in your development process. See below for a few of the considerations and best practices for using spare devices for automated testing:

It’s all dependent on your dev resources: While you will need the hardware to get up and running, be sure to know that, first and foremost, your ability to manage a device lab of any kind will ultimately depend on your engineering team, and their ability to write automated test scripts that saves your team time and energy with each iteration.
More automation = more devices: Once you have a few automation scripts written, start testing on available devices during downtime, overnight, etc. All of this is an iterative process, once you have enough automation written, you can begin to justify more hardware to use as dedicated testing units.
Remember to consider bandwidth post test: When you start to justify dedicated testing units, remember that you will need team members to analyze and report on these findings. If this bandwidth doesn’t exist, you may want to reconsider dedicated hardware.
2D testing: As you may have more available PCs and laptops than Meta Quest headsets, you can always start by running your automation on 2D devices. These are more than sufficient for testing gameplay logic, look and feel (initial textures), scripted gameplay, etc.

More resources for testing and automation

While the world is full of helpful resources for testing, automation, performance optimization, and tooling to help drive all of the above, here are a few highlights that will help you get started.

Unity Test Runner Documentation: The Test Runner Tool from Unity can be an extremely powerful asset to your testing toolkit. The tool is especially helpful in managing and running unit tests for elements that wouldn’t otherwise be possible to test as you run your app.
[Video] Unity + Meta Quest Courseware - Testing Unit: In this unit of the free, Meta Quest + Unity video courseware, the team provides a number of insights and best practices for VR app testing.
Meta Quest VR Playtesting Guide: Be sure to review our guide on the many facets of VR playtesting, including planning/research, script writing, remote facilitation and more.
[Video] Automated Testing at Scale in Sea of Thieves: Unreal Fest Europe 2019: An insightful presentation that provides a number of foundational insights for game testing automation, as well as example techniques for implementation.

Did you find this page helpful?