Like most VR software and hardware, the technical requirements and processes to implement mixed reality capture (MRC) have come a long way over the past few years. In the guide below we provide all of the necessary links, downloads, tips and watch outs for building your app, integrating MRC tools, and producing high quality MRC experiences.
See below for all of the existing documentation to setup, install and get started with the technical side of MRC. You can also check out the video below which features a step-by-step guide to setting up MRC, calibrating the camera, and loading Open Broadcaster Software (OBS).
Download the latest version: First and foremost, we recommend using the most up to date version of the engine integration, and Platform OS. This will ensure that you’re getting the most recent set of MRC tools to help you implement MRC with minimal effort.
Update your integration per the OVRMrcLib 2.0 update: The OVRMrcLib library is now included within the Oculus Quest runtime. As of version 2, released on 04/20/2020, future updates to the package will be included within the OS. This includes the UE4 and Unity integration which have the most up-to-date OVRMrcLib, however, if you are using an old version of this library, be sure to upgrade to the latest version (which is the shim loader of libOVRMrcLib.so), and remove libOVRMrcLib.so from your APK.
Rift S recommended for trailer production: For maximum fidelity and streamlined debugging from within the game engine, we recommend leveraging the Rift S for producing high quality, video marketing assets.
Prioritize first two pairs during calibration: During camera calibration, if you’re going to get any of the matches perfectly captured, focus on the accuracy of the first two pairs, these are the most important for proper calibration.
Adjust horizontal FOV settings first: When you make detailed adjustments to the calibration via the system UI, be sure to adjust the horizontal FOV first, followed by the translation.
When in doubt, reboot: Like so many other technologies, if you run into technical difficulties it is recommended to first reboot your system + the VR hardware as a first attempt to fix the issue.
Steps to save a 2nd set of recalibration data on your Rift S: The current system can only save calibration data from your first calibration. If you want to clear this data and save a new set of calibration settings, simply follow one of the following steps:
Minimal to moderate light intensity recommended during calibration: Accurate calibration relies on precise directional recording of the controller determined by the LED lights within the controller hardware. Be sure to use minimal to moderate light intensity during calibration, as these can interfere with the LED lights and negatively impact the accuracy of the digital camera.
Additional markers placed on greens screen for enhanced tracking accuracy: As shown in the photo below, we recommend creating unique forms with a few pieces of tape on your green screen, this further grounds the third person camera for improved capture accuracy.
Keep level transitions + game state changes in mind for custom builds: If you decide to do a custom build for MRC, it can help streamline the production process to include a user friendly way to jump into specific sections of the game, potentially informed by your storyboards.
Be sure that your system meets the PC spec and internet requirements: While it may go without saying, be sure that you do not skip on our recommendations for PC system and wireless router specs (5Ghz). We also highly recommend ensuring that you work from a stable network. MRC capture can become challenging if you try to push forward with low spec hardware, while unstable networks may cause the need for recalibration.
Enabling MRC and the third person camera sets your game at ½ the frame rate, this is why you should fully optimize your app wherever possible. We recommend that your game run effectively at 60FPS or better without MRC enabled.
While there are numerous ways to effectively optimize for performance, shaders and effects are the low hanging fruit. Be sure to take into account the spectator’s experience and ask yourself whether those effects that have a significant impact on your performance cost are really necessary. For more information on performance optimization, check out any of the following docs or video presentations:
Lighting plays a massive role in MRC and truly separates the polished, high quality footage from the mediocre files. Be sure to keep in mind the following best practices before and during your MRC shoot:
Along with all of the points above, be sure to recognize that the quality of your camera, lens, and lighting equipment will impact your MRC assets. If you’re new to video production, talk with a teammate or professional peer that is familiar with these sorts of projects, or you may want to hire third party support.
Storyboards recommended + example doc: Like many video projects, it’s recommended that you build a storyboard prior to the day of production. It’s good to have the POV and different angles in mind as you tell your story, especially as each adjustment will require you to recalibrate the MRC tool.
Mixing in-game, first person footage and MRC footage can be a great way to share your experience with a certain level of energy, while you should also consider your use of wide versus long camera lenses per each shot to direct the viewer’s attention in a subtle, more cinematic way.
Here’s an example storyboard created in partnership with the teams at SUPERHOT + We Are Royale.
Wardrobe selection: While it’s generally recommended to select a wardrobe that will resonate with your target audience and game style, it’s also important to note that the system can sometimes confuse clothing for the green screen if they are too similar. Select clothing with colors that are on opposite sides of the color spectrum as your green screen, while solid, darker colors have the lowest risk of being mistaken for your backdrop.
Previous VR / green screen experience recommended for actors: If you are selecting actors for your trailer shoot, it’s always recommended to select those that have VR/green screen experience as this will generally streamline the shooting process.
In-headset actor performance: Whether you’re able to hire an actor or not, be sure to direct the individual who is in-headset to be expressive throughout the shoot. Depending on your app and target audience, you will want to convey a certain emotion whether it’s a sense of calm, humor, warmth, intensity, fun or excitement.
Foreground opportunities with casting (no actor): If you are simply capturing the experience with a third person camera, you have the opportunity to be creative with what is featured in the foreground. Will you display an avatar of some sort? What angles will add even more value to your production? Any great ideas for the third person camera can also be leveraged in the casting experience you share to your consumers.
If you haven’t already, be sure to review the first guide in this series on MRC, which covers app design best practices including locomotion, environmental design and more.
If you’re interested in learning more about how these tools deliver MRC for developers, streamers, and VR users, check out the following OC6 presentation, featuring members of our MRC engineering team.
We hope you find this guide insightful, and that MRC + casting help you to grow your audience, increase engagement, and app sales. If you have any questions or feedback regarding the tooling this guide feel free to reach out to mrcfeedback[at]fb.com, and keep up the great work!