Mixed Reality Capture

Mixed reality capture for Oculus Quest places real-world people and objects in VR. This guide will review how to configure support for mixed reality capture in your Oculus Quest app.

This feature is in development. Implementation and functionality may change.

Introduction

Mixed reality capture places real-world video footage of a user to be composited with the output from a game to create combined video that shows the player in a virtual scene.

Information about device and camera setup can be found in the Quest Mixed Reality Capture Setup Guide.

Once an application is built to use mixed reality capture, users can launch applications in Mixed Reality mode and use the Oculus MRC Plugin for OBS.

Preparation

Mixed reality capture may be used by any application built on the Oculus Integration 1.38, or later, and uses an instance of OVRManager and OVRCameraRig.

you don’t need to do anything if your application uses OVRManager and OVRCameraRig. Mixed reality capture mode is automatically activated if there is a valid camera configuration file in the app data folder before application startup (see Camera Calibration), and has an active connection from OBS (see Composite the Scene using OBS).

Manual Integration

If your project hasn’t spawned the OVRManager and/or OVRCameraRig, you can add the following code to your presistent system component’s Awake() event:

    void Awake() {
            // ...
            OVRManager manager = gameObject.GetComponent<OVRManager>();
            if (manager == null)
            {
                manager = gameObject.AddComponent<OVRManager>();
                manager.trackingOriginType = OVRManager.TrackingOrigin.FloorLevel;
                Debug.Log("OVRManager injected");
            }

            OVRCameraRig cameraRig = gameObject.GetComponent<OVRCameraRig>();
            if (cameraRig == null)
            {
                cameraRig = gameObject.AddComponent<OVRCameraRig>();
                cameraRig.disableEyeAnchorCameras = true;  // this line disables the cameras on the CameraRig, and let the project uses the original camera for VR rendering
                Debug.Log("OVRCameraRig injected");
            }
            // ...
        }

Hide unnecessary objects from Mixed Reality Capture

There are two ways you can remove the unnecessary objects or visual effects from Mixed Reality Capture cameras, but keep them visibility under the headset:

• Click “Show Properties” in the “Mixed Reality Capture” section of OVRManager, add include the layers you want to hide into “extraHiddenLayers”
• If there are image effects or other script components which doesn’t apply to the Mixed Reality Capture camera, you can also detect the camera through the game object name, and destroy those components:

	void Start() {
		// disable the component for OculusMRC cameras
		if (gameObject.name.StartsWith("OculusMRC_"))
		{
			Object.Destroy(this);
			return;
		}

Mixed Reality Capture Performance

Rendering from the additional 3rd-person camera will impact the application performance. The CPU/GPU frequency of the Oculus Quest would be boosted temporarily to provide the extra performance when mixed reality capture is activated.

The video stream sent to OBS is encoded and transmitted every other frame, so the streamed video will be at 1/2 the framerate of the application running on the Oculus Quest. For a quality capture, ensure that the application runs at 60 fps or higher on the Oculus Quest when mixed reality capture is activated.

Compositing the Scene

Mixed reality capture for Oculus Quest supports external composition using the Oculus-provided OBS plugin. The Oculus Quest will stream the foreground and background scenes to a PC that is capturing the real-world image. The OBS plugin will combine the scenes together for the final Mixed Reality recording. Information about using the OBS plugin can be found in the Quest Mixed Reality Capture Setup Guide.