While automation is common throughout most types of software engineering, it has yet to become a common practice for VR app and game development. We want to help you get started with using automation, or at a minimum, understand how to assess, get started, and scale your automation and testing efforts.
This post consists of a series of recommendations and best practices to keep in mind as you prioritize and implement automation. It is an excerpt from a larger guide we recently published in a series focusing on Minimizing Iteration Time and Maximizing Developer Efficiency. The larger guide includes sections on General Considerations and Cross Platform Development, Project Planning and File/Folder Management, Automation and Performance Optimization, as well as Best Practices specifically for Unity development. Check out each of these guides as they include insights gathered from interviews with cross functional VR subject matter experts throughout Facebook Reality Labs.
For those new to software automation, we especially recommend this guide on automation, as it includes a primer on the different testing scenarios that will help you prioritize where you focus your automation resources.
Automating the test process can help you a great deal, especially in the long term, but there are a number of items to consider before you begin to focus on automation script writing and the technical requirements that go along with it. See below for a few key factors you should keep in mind as you design your testing and automation strategy:
Be prepared to analyze the cost benefit analysis of writing a rock solid test script as opposed to using manual testing. The more complex the test, the more time and effort will be required to write and maintain.
With the above point in mind, if you have the ability to automate any of your processes, whether something as small as a boot test, a complex interaction, or full environment, automation can help your team drastically speed up iteration time and minimize the number of bugs/issues that arise in your app over time.
Automate sections of your app that are repeatable and generally less complex. This will maximize efficiency and ensure your script accurately delivers the designed outcome.
Employ automation in those strange instances where it would require a great deal of logistical work for a single, manual test, or user flows that might be considered “abnormal.”
After you’ve decided what type of tests you’re looking to automate and where to focus your resources, use the following best practices to move forward with your test automation process.
As your team most likely already uses test scripts to drive your QA process, these can be an easy way to find those repeatable areas of your build that could be automated with minimal effort.
If you are looking to publish a fully developed Oculus VR app, it’s recommended that you leverage a source control and continuous integration system. These come in many different shapes and sizes, and it’s recommended you scale your system with the size of your team and type of app you’re creating. For source control systems we recommend taking a look at Git and Mercurial, or Perforce if you have a more complex app. For continuous integration systems check out Jenkins, TeamCity, or GitLab.
Whether you’re using test automation or not, be sure to include a version of your app that can simply be run on a 2D screen. This will save you plenty of time for testing certain features, variables, and properties, and speed up your debugging process.
This sort of data can be extremely powerful in the long term as you work to analyze what may have caused a specific error. When planning on what telemetry data to log, think about the user journey through your app and the user flows that need to be completed for a successful playthrough.
Compiling code and running the asset build on your CI system can detect issues as soon as a new change is committed. Proactively warning your team of these errors can ensure that they avoid any conflicts as well as steer clear of any potential blockers these issues might cause.
On-device automation can be especially helpful for monitoring performance. New changes can unintentionally harm performance, and detecting this as soon as possible will be much easier to fix when the cause is known. Monitoring performance over time can also ensure that your updates to optimize performance are effective. See below for more information on testing your app’s performance.
Trigger specific events on the VR headset, for example “tell character to go to X world and move character to Y position”. Examine performance characteristics over ADB and be sure to run certain checks throughout the event.
If you’re looking to learn even more about VR development automation, performance optimization, and automating your testing process be sure to check out the links below.
And if you haven’t already visited the Learn section of our developer center, here are a few recently updated guides for you to further improve your processes and elevate your skillset.
Feel free to leave any comments or questions in the comments section below, and keep up the great work!