Meta Developers

Health & Safety Recommendation: While building mixed reality experiences with the Passthrough API, we highly recommend that developers evaluate their content to support a comfortable and safe experience for users. Refer to the Insight SDK Tips and Tricks guide for more information.

Passthrough provides real-time and perceptually comfortable 3D visualization of the physical world in the Meta Quest headsets. The Passthrough API allows developers to integrate the passthrough visualization with their virtual experiences. Its functionality is offered by the OpenXR extensions XR_FB_passthrough and XR_FB_triangle_mesh.

For a complete API reference as part of the OpenXR spec, read XR_FB_passthrough and XR_FB_triangle_mesh.

How Does This Work?

Passthrough is rendered by a dedicated service into a separate layer, which is submitted directly to the XR Compositor. An app cannot access images or videos of a user’s physical environment. Instead, the application submits a special placeholder layer. This layer gets replaced with the actual passthrough rendition by the XR Compositor. There are a few ways to customise passthrough - composite layering and styling:

Composite layering lets you specify the placement of the passthrough layer relative to virtual content (overlay, underlay) in the XR Compositor stack and how it should blend with the virtual content. Using alpha masking, you can specify where on the screen passthrough shows up. Currently, there isn’t a way to specify blending based on passthrough depth.
Styling lets you colourize the passthrough feed and apply visual effects like edge rendering.
Surface-Projected Passthrough lets you define the geometry onto which passthrough images are projected. This approach leads to a more stable rendition of passthrough in cases where parts of the user’s environment are known to the application.

Components

XR Session is the OpenXR session object (represented by its handle), as defined by the OpenXR spec. Passthrough objects are scoped to an XrSession instance, and therefore a session must be provided to most “create” calls. For details, read Creating Instances and Sessions.

Passthrough Feature enables passthrough functionality and is responsible for providing the data necessary for that functionality to work.

Passthrough Layers provide the ability to render content. They are produced by the system and submitted to the system compositor on behalf of the application. The application can specify where and how the content is rendered. Where corresponds to the position in the composition stack, for example, on top of or beneath the application’s projection layer). How corresponds to type (projected or reconstruction) and style (color, outlines).

Meshes and geometry instances allow an application to specify the geometry onto which the passthrough images are projected (instead of an automatic environment depth reconstruction). When activated, passthrough is only visible on the geometries selected for surface-projected passthrough. Passthrough behaves like a texture for those geometries. This approach leads to a more stable rendition of passthrough in cases where parts of the user’s environment are known to the application.

Prerequisites

Before you begin with passthrough, make sure to use the latest version of the Meta Quest operating system and the Meta XR Core SDK, which is available individually or as part of the Meta XR All-in-One SDK. Do the following:

Ensure the headset is connected to the internet and has the latest version installed. To verify the latest version, in the headset, go to Settings > System > Software Update and update if necessary.
The app must use 64-bit builds as 32-bit apps are not supported.

If you would like to run your projects directly from within Unity Editor with a device connected via Link, please take the additional prerequisites for Passthrough over Link into account.

Project Setup

Refer to Get Started with Mobile SDK and OpenXR Support for Meta Quest and Quest 2 for information about how to set up a generic OpenXR project for Meta Quest and Quest 2.
Add the passthrough feature flag in the AndroidManifest.xml file: <uses-feature android:name="com.oculus.feature.PASSTHROUGH" android:required="true" />

Note: The passthrough OpenXR extensions will not enable if this feature flag is not set in the manifest file.

Create an Instance and Session

To implement the Passthrough API, XR_FB_passthrough and XR_FB_triangle_mesh (used only for surface-projected Passthrough) extensions need to be declared as required extensions when the OpenXR instance is created:

std::vector<const char*> requiredExtensionNames{
  XR_FB_PASSTHROUGH_EXTENSION_NAME,
  XR_FB_TRIANGLE_MESH_EXTENSION_NAME // optional
};

XrInstanceCreateInfo instanceCreateInfo = {XR_TYPE_INSTANCE_CREATE_INFO};
instanceCreateInfo.createFlags = 0;
instanceCreateInfo.applicationInfo = appInfo;
instanceCreateInfo.enabledApiLayerCount = 0;
instanceCreateInfo.enabledApiLayerNames = NULL;
instanceCreateInfo.enabledExtensionCount = requiredExtensionNames.size();
instanceCreateInfo.enabledExtensionNames = requiredExtensionNames.data();

XrInstance instance;
XrResult result = xrCreateInstance(&instanceCreateInfo, &instance);
if (XR_FAILED(result)) {
  LOG("Failed to create XR instance: %d.\n", initResult);
  exit(1);
}

Then proceed to create the XrSession as you normally would. For a complete sample for setting up and managing an OpenXR instance and session, refer to the Native Samples.

Instance creation fails in case some of the prerequisites mentioned above are not met. You can verify the availability of the extensions before creating the XrInstance by calling xrEnumerateInstanceExtensionProperties and checking that XR_FB_passthrough and XR_FB_triangle_mesh are present in the list of extensions returned.

Ensure that the System Supports Passthrough

Before attempting to use passthrough in your native app, you must ensure that the system supports it. To do so, call xrGetSystemProperties and check the supportsPassthrough flag in the XrSystemPassthroughProperties2FB struct:

static bool SystemSupportsPassthrough(XrInstance instance, XRFormFactor formFactor)
{
    XrSystemPassthroughProperties2FB passthroughSystemProperties{XR_TYPE_SYSTEM_PASSTHROUGH_PROPERTIES2_FB};
    XrSystemProperties systemProperties{XR_TYPE_SYSTEM_PROPERTIES, &passthroughSystemProperties};

    XrSystemGetInfo systemGetInfo{XR_TYPE_SYSTEM_GET_INFO};
    systemGetInfo.formFactor = formFactor;

    XrSystemId systemId = XR_NULL_SYSTEM_ID;
    xrGetSystem(instance, &systemGetInfo, &systemId);
    xrGetSystemProperties(instance, systemId, &systemProperties);

    return (passthroughSystemProperties.capabilities & XR_PASSTHROUGH_CAPABILITY_BIT_FB) == XR_PASSTHROUGH_CAPABILITY_BIT_FB;
}

Similarly, you can check if the current device supports color passthrough by testing whether the XR_PASSTHROUGH_CAPABILITY_COLOR_BIT_FB capability flag is true.

Acquire Function Pointers

To use the functions defined in the passthrough extensions, you need to declare and obtain function pointers.

PFN_xrCreatePassthroughFB pfnXrCreatePassthroughFBX = nullptr;
XrResult result = xrGetInstanceProcAddr(_instance, "xrCreatePassthroughFB", (PFN_xrVoidFunction*)(&pfnXrCreatePassthroughFBX));

if (XR_FAILED(result)) {
  LOG("Failed to obtain the function pointer for xrCreatePassthroughFB.\n");
}

You can find a complete list of function pointers in the OpenXR passthrough sample.

Create and Start the Passthrough Feature

Before any layers of passthrough can be added to the application, the app needs to create a passthrough feature. The passthrough feature controls the basic system resources and processing needed to display passthrough. The application must only create a single passthrough feature in any session.

// Create the feature
XrPassthroughFB passthroughFeature = XR_NULL_HANDLE;

XrPassthroughCreateInfoFB passthroughCreateInfo = {XR_TYPE_PASSTHROUGH_CREATE_INFO_FB};
passthroughCreateInfo.flags = XR_PASSTHROUGH_IS_RUNNING_AT_CREATION_BIT_FB;

XrResult result = pfnXrCreatePassthroughFBX(GetSession(), &passthroughCreateInfo, &passthroughFeature);
if (XR_FAILED(result)) {
  LOG("Failed to create the passthrough feature.\n");
}

By specifying the XR_PASSTHROUGH_IS_RUNNING_AT_CREATION_BIT_FB flag in XrPassthroughCreateInfoFB, the passthrough feature starts immediately after creation. This means that the system may immediately start performing environment reconstruction and other background tasks needed to display passthrough. If you would like to separate creation and starting, for example, because the application doesn’t want to display passthrough immediately when running, leave flags unset and start the feature manually at a later point using the following code:

// Start the feature manually
XrResult result = pfnXrPassthroughStartFB(passthroughFeature);
if (XR_FAILED(result)) {
  LOG("Failed to start a passthrough feature.\n");
}

Create and Start a Passthrough Layer

The passthrough feature prepares the system to generate passthrough images, but to make passthrough appear in your application, the application needs to create and start one or more passthrough layers. Passthrough layers serve two purposes:

Specify how passthrough is composited with your application’s other layers. For example, the main projection layer.
Customize passthrough appearance and behavior individually.

An application can create and start multiple passthrough layers. However, a maximum of three passthrough layers can be created at any given time.

// Create and run passthrough layer
XrPassthroughLayerFB passthroughLayer = XR_NULL_HANDLE;

XrPassthroughLayerCreateInfoFB layerCreateInfo = {XR_TYPE_PASSTHROUGH_LAYER_CREATE_INFO_FB};
layerCreateInfo.passthrough = passthroughFeature;
layerCreateInfo.purpose = XR_PASSTHROUGH_LAYER_PURPOSE_RECONSTRUCTION_FB;
layerCreateInfo.flags = XR_PASSTHROUGH_IS_RUNNING_AT_CREATION_BIT_FB;

XrResult result = pfnXrCreatePassthroughLayerFBX(Session, &layerCreateInfo, &passthroughLayer);
if (XR_FAILED(result)) {
  LOG("Failed to create and start a passthrough layer");
}

Similar to the passthrough feature, you can choose to create a passthrough layer in paused state by omitting the XR_PASSTHROUGH_IS_RUNNING_AT_CREATION_BIT_FB flag, and start it later by using xrPassthroughLayerResumeFB.

The purpose field specifies the type of passthrough layer. In most cases, XR_PASSTHROUGH_LAYER_PURPOSE_RECONSTRUCTION_FB is the appropriate choice. See Surface-Projected Passthrough for an explanation of the alternative.

Submit Passthrough Layers to the Compositor

To instruct the XR compositor in which order a passthrough layer should be blended with other layers submitted by the application, a compositor layer of type XrCompositionLayerPassthroughFB needs to be included in the call to xrEndFrame for each passthrough layer.

In a simple scenario where the application has one projection layer and one passthrough layer, this enables the application to choose whether the passthrough layer should appear on top of or beneath the application’s projection layer. In a complex scenario with more than two layers, any compositing order is achievable.

Passthrough layers for which no proxy layer is submitted may be discarded by the compositor. However, not submitting a proxy layer should not be used to hide a passthrough layer. Applications should pause passthrough layers using xrPassthroughLayerPauseFB to preserve system resources and inform the system that passthrough is inactive. Proxy layers should always be submitted for running passthrough layers and never for paused ones.

The following snippet shows the simple scenario where a passthrough layer is composited as an overlay on the application’s projection layer.

XrCompositionLayerPassthroughFB passthroughCompLayer = {XR_TYPE_COMPOSITION_LAYER_PASSTHROUGH_FB};
passthroughCompLayer.layerHandle = passthroughLayer;
passthroughCompLayer.flags = XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT;
passthroughCompLayer.space = XR_NULL_HANDLE;

const kLayerCount = 2;
const XrCompositionLayerBaseHeader* layers[kLayerCount] = {
  (const XrCompositionLayerBaseHeader*) applicationCompLayer,
  (const XrCompositionLayerBaseHeader*) passthroughCompLayer
};

XrFrameEndInfo endFrameInfo = {XR_TYPE_FRAME_END_INFO};
endFrameInfo.displayTime = predictedDisplayTime;
endFrameInfo.environmentBlendMode = XR_ENVIRONMENT_BLEND_MODE_OPAQUE;
endFrameInfo.layerCount = kLayerCount;
endFrameInfo.layers = layers;

xrEndFrame(session, &endFrameInfo);

Notes on the structure contents:

Since the passthrough composition layer doesn’t contain any image data, XrCompositionLayerPassthroughFB::space is irrelevant and must be set to XR_NULL_HANDLE.
The only layer flag that is relevant for passthrough composition layers is XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT. Other layer flags are ignored.
- The passthrough layers generated by the system may contain non-opaque alpha values when styles or surface-projected Passthrough are used. Those alpha values are ignored if XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT is not set. It is recommended to set the flag unless an application specifically needs to ignore alpha values.
The Meta Quest OpenXR implementation requires XrFrameEndInfo::environmentBlendMode to be set to XR_ENVIRONMENT_BLEND_MODE_OPAQUE, even when the passthrough is used. To display passthrough as background, submit a passthrough layer before the application’s projection layer, that is, changing the order in which applicationCompLayer and passthroughCompLayer are inserted in the XrCompositionLayerBaseHeader array.
The following two extensions are supported in the next chain of a XrCompositionLayerPassthroughFB instance. The properties of these extensions are respected by the passthrough layer upon compositing.
- XR_FB_composition_layer_alpha_blend: specify the blend factors of a passthrough layer, for example, to achieve additive blending, or to mask a passthrough layer with the alpha channel of a different layer (see Compositing and Masking for Mixed Reality).
- XR_KHR_composition_layer_color_scale_bias: modify the color and alpha of a passthrough layer upon compositing.

Enable Based on System Recommendation

In your app, you may want to show either a VR or Passthrough background based on user choice. Our system already provides users with this choice in the home environment, and your app can leverage the user’s home environment preference. More generally speaking, our system provides a recommendation for apps to default to MR or VR based on user preferences. This recommendation is made available by the extension XR_META_passthrough_preferences:

Call xrGetPassthroughPreferences.
If the flag XR_PASSTHROUGH_PREFERENCE_DEFAULT_TO_ACTIVE_BIT_META is set, the system’s recommendation is to show Passthrough in your app. This is currently the case when the user has selected Passthrough as background in their home environment.

Did you find this page helpful?