Meta Developers

Health and Safety Recommendation: While building mixed reality experiences, we highly recommend that you evaluate your content to offer your users a comfortable and safe experience. Please refer to the Health and Safety and Design guidelines before designing and developing your app using Depth.

Overview

What is Depth API?

The Depth API provides real-time depth maps that apps can use to sense the environment. Primarily, it enhances mixed reality (MR) by allowing virtual objects to be occluded by real-world objects and surfaces, which makes them appear integrated into the actual environment. Occlusion is crucial because it prevents virtual content from appearing as a layer over the real world, which can disrupt immersion.

Why use Depth API?

The Scene Model enables the creation of room-scale, mixed reality experiences featuring realistic occlusion. However, it cannot handle occlusion for objects that are dynamically moving within the user’s view, such as hands, limbs, other people, and pets. To achieve realistic occlusion with these dynamic elements, you must also use the Depth API.

Use cases	Depth API	Scene API
Static Occlusion: the occluding real-world environment remains immobile (static) throughout the lifetime of the app, i.e. no moving objects such as hands, people or chairs.	✔	✔
Dynamic Occlusion: the occluding real-world environment contains mobile (dynamic) elements, e.g. the users hands, other people, pets.	✔	✖
Raycasting: Computing intersection of a ray and real-world surfaces. Supports use cases like content placement.	✔	✔
Physics/Collisions: interactions between virtual content and the real-world, like a virtual ball bouncing off of a physical couch.	✖	✔

Requesting user permission

An app that wants to use Depth API needs to request and be granted spatial data permission (com.oculus.permission.USE_SCENE) before accessing any of the functions in the XR_META_environment_depth extension. See the Spatial Data Permission guide for how to set up the app and how to request this permission.

Enabling the extension

The Depth API is defined in the XR_META_environment_depth OpenXR extension. All extensions should be explicitly listed when creating an XrInstance. Include XR_META_ENVIRONMENT_DEPTH_EXTENSION_NAME in this list to enable the Depth API extension:

std::vector<const char*> extensions = {XR_META_ENVIRONMENT_DEPTH_EXTENSION_NAME};

XrInstanceCreateInfo instanceCreateInfo = {XR_TYPE_INSTANCE_CREATE_INFO};
instanceCreateInfo.enabledExtensionCount = extensions.size();
instanceCreateInfo.enabledExtensionNames = extensions.data();

XrInstance instance = XR_NULL_HANDLE;
xrCreateInstance(&instanceCreateInfo, &instance);

For more details, see SampleXrFramework/Src/XrApp.cpp.

Note that this feature is not supported by all runtimes and devices so the app must first call xrGetSystemProperties to query for a XrSystemEnvironmentDepthPropertiesMETA struct and check that supportsEnvironmentDepth is true.

Getting the extension function pointers

You must retrieve pointers to all the functions in the extension before calling them. For more details, see xrGetInstanceProcAddr in the OpenXR spec.

The following snippet initializes all the available functions:

PFN_xrCreateEnvironmentDepthProviderMETA xrCreateEnvironmentDepthProviderMETA = nullptr;
PFN_xrDestroyEnvironmentDepthProviderMETA xrDestroyEnvironmentDepthProviderMETA = nullptr;
PFN_xrStartEnvironmentDepthProviderMETA xrStartEnvironmentDepthProviderMETA = nullptr;
PFN_xrStopEnvironmentDepthProviderMETA xrStopEnvironmentDepthProviderMETA = nullptr;
PFN_xrCreateEnvironmentDepthSwapchainMETA xrCreateEnvironmentDepthSwapchainMETA = nullptr;
PFN_xrDestroyEnvironmentDepthSwapchainMETA xrDestroyEnvironmentDepthSwapchainMETA = nullptr;
PFN_xrEnumerateEnvironmentDepthSwapchainImagesMETA xrEnumerateEnvironmentDepthSwapchainImagesMETA = nullptr;
PFN_xrGetEnvironmentDepthSwapchainStateMETA xrGetEnvironmentDepthSwapchainStateMETA = nullptr;
PFN_xrAcquireEnvironmentDepthImageMETA xrAcquireEnvironmentDepthImageMETA = nullptr;
PFN_xrSetEnvironmentDepthHandRemovalMETA xrSetEnvironmentDepthHandRemovalMETA = nullptr;

xrGetInstanceProcAddr(
  instance,
  "xrCreateEnvironmentDepthProviderMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrCreateEnvironmentDepthProviderMETA));
xrGetInstanceProcAddr(
  instance,
  "xrDestroyEnvironmentDepthProviderMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrDestroyEnvironmentDepthProviderMETA));
xrGetInstanceProcAddr(
  instance,
  "xrStartEnvironmentDepthProviderMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrStartEnvironmentDepthProviderMETA));
xrGetInstanceProcAddr(
  instance,
  "xrStopEnvironmentDepthProviderMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrStopEnvironmentDepthProviderMETA));
xrGetInstanceProcAddr(
  instance,
  "xrCreateEnvironmentDepthSwapchainMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrCreateEnvironmentDepthSwapchainMETA));
xrGetInstanceProcAddr(
  instance,
  "xrDestroyEnvironmentDepthSwapchainMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrDestroyEnvironmentDepthSwapchainMETA));
xrGetInstanceProcAddr(
  instance,
  "xrEnumerateEnvironmentDepthSwapchainImagesMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrEnumerateEnvironmentDepthSwapchainImagesMETA));
xrGetInstanceProcAddr(
  instance,
  "xrGetEnvironmentDepthSwapchainStateMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrGetEnvironmentDepthSwapchainStateMETA));
xrGetInstanceProcAddr(
  instance,
  "xrAcquireEnvironmentDepthImageMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrAcquireEnvironmentDepthImageMETA));
xrGetInstanceProcAddr(
  instance,
  "xrSetEnvironmentDepthHandRemovalMETA",
  reinterpret_cast<PFN_xrVoidFunction*>(&xrSetEnvironmentDepthHandRemovalMETA));

Creating a depth provider

You can create a depth provider by using the xrCreateEnvironmentDepthProviderMETA function. This function creates and returns a XrEnvironmentDepthProvider handle to a depth provider instance. Maximum one depth provider is allowed to exist per app at any given time. The handle is unique per process and cannot be shared across processes.

The definition of the xrCreateEnvironmentDepthProviderMETA function is as follows:

XrResult xrCreateEnvironmentDepthProviderMETA(
    XrSession session,
    const XrEnvironmentDepthProviderCreateInfoMETA* createInfo,
    XrEnvironmentDepthProviderMETA* environmentDepthProvider);

To call this function, you must pass a XrEnvironmentDepthProviderCreateInfoMETA struct containing creation flags.

The struct’s definition is as follows:

typedef struct XrEnvironmentDepthProviderCreateInfoMETA {
    XrStructureType                                type;
    const void* XR_MAY_ALIAS                       next;
    XrEnvironmentDepthProviderCreateFlagsMETA      createFlags;
} XrEnvironmentDepthProviderCreateInfoMETA;

Currently createFlags must be zero, but it might be extended in the future.

To free up all the resources used by the depth provider, you can destroy it by calling xrDestroyEnvironmentDepthProviderMETA:

XrResult xrDestroyEnvironmentDepthProviderMETA(
    XrEnvironmentDepthProviderMETA environmentDepthProvider);

Toggling hand removal

The Depth API supports removing hands from depth maps and replacing them with estimated background depth. Removing hands can be useful for example if the app prefers to use Tracked Hands for hand occlusions instead of the depth maps.

Hand removal can be enabled or disabled at any time by calling the xrSetEnvironmentDepthHandRemovalMETA function:

XrResult xrSetEnvironmentDepthHandRemovalMETA(
    XrEnvironmentDepthProviderMETA environmentDepthProvider,
    const XrEnvironmentDepthHandRemovalSetInfoMETA* setInfo);

The function takes a XrEnvironmentDepthHandRemovalSetInfoMETA argument which is defined as:

typedef struct XrEnvironmentDepthHandRemovalSetInfoMETA {
    XrStructureType             type;
    const void* XR_MAY_ALIAS    next;
    XrBool32                    enabled;
} XrEnvironmentDepthHandRemovalSetInfoMETA;

Creating and enumerating a depth swapchain

Depth maps are provided to the app through a special “readable swapchain” type XrEnvironmentDepthSwapchainMETA. This type is similar to XrSwapchain but supports a different set of operations and is intended to be read instead of written to by the app.

Create the swapchain by calling xrCreateEnvironmentDepthSwapchainMETA:

XrResult xrCreateEnvironmentDepthSwapchainMETA(
    XrEnvironmentDepthProviderMETA environmentDepthProvider,
    const XrEnvironmentDepthSwapchainCreateInfoMETA* createInfo,
    XrEnvironmentDepthSwapchainMETA* swapchain);

This function takes a XrEnvironmentDepthSwapchainCreateInfoMETA struct specifying options for creating the swapchain:

typedef struct XrEnvironmentDepthSwapchainCreateInfoMETA {
    XrStructureType                                 type;
    const void* XR_MAY_ALIAS                        next;
    XrEnvironmentDepthSwapchainCreateFlagsMETA      createFlags;
} XrEnvironmentDepthSwapchainCreateInfoMETA;

Currently createFlags must be zero, but it might be extended in the future.

Once the swapchain is created the resolution can be queried by calling xrGetEnvironmentDepthSwapchainStateMETA:

XrResult xrGetEnvironmentDepthSwapchainStateMETA(
    XrEnvironmentDepthSwapchainMETA swapchain,
    XrEnvironmentDepthSwapchainStateMETA* state);

The returned XrEnvironmentDepthSwapchainStateMETA is defined as:

typedef struct XrEnvironmentDepthSwapchainStateMETA {
    XrStructureType       type;
    void* XR_MAY_ALIAS    next;
    uint32_t              width;
    uint32_t              height;
} XrEnvironmentDepthSwapchainStateMETA;

In the same way as for a regular XrSwapchain, the XrEnvironmentDepthSwapchainMETA needs to be “enumerated” into a graphics API specific array of texture handles. This is done by calling xrEnumerateEnvironmentDepthSwapchainImagesMETA that has the same semantics as xrEnumerateSwapchainImages:

XrResult xrEnumerateEnvironmentDepthSwapchainImagesMETA(
    XrEnvironmentDepthSwapchainMETA swapchain,
    uint32_t imageCapacityInput,
    uint32_t* imageCountOutput,
    XrSwapchainImageBaseHeader* images);

To free up all the resources used by the swapchain, you can destroy it by calling xrDestroyEnvironmentDepthSwapchainMETA:

XrResult xrDestroyEnvironmentDepthSwapchainMETA(
    XrEnvironmentDepthSwapchainMETA swapchain);

Starting and stopping the depth provider

The depth provider is created in a stopped state by default. To start the asynchronous generation of depth maps, you need to call xrStartEnvironmentDepthProviderMETA:

XrResult xrStartEnvironmentDepthProviderMETA(
    XrEnvironmentDepthProviderMETA environmentDepthProvider);

To stop the asynchronous generation of depth maps, call xrStopEnvironmentDepthProviderMETA:

XrResult xrStopEnvironmentDepthProviderMETA(
    XrEnvironmentDepthProviderMETA environmentDepthProvider);

Acquiring depth maps

Depth maps should only be accessed between the xrBeginFrame and xrEndFrame calls. To acquire a lock on the latest available depth map in the swapchain, as well as retrieve metadata needed to parse the depth, apps need to call xrAcquireEnvironmentDepthImageMETA. Once a depth swapchain has been acquired it is locked for the entire duration of the frame. You shouldn’t call xrAcquireEnvironmentDepthImageMETA more than once per frame.

XrResult xrAcquireEnvironmentDepthImageMETA(
    XrEnvironmentDepthProviderMETA environmentDepthProvider,
    const XrEnvironmentDepthImageAcquireInfoMETA* acquireInfo,
    XrEnvironmentDepthImageMETA* environmentDepthImage);

This function takes an XrEnvironmentDepthImageAcquireInfoMETA struct that needs to be populated with some required parameters:

typedef struct XrEnvironmentDepthImageAcquireInfoMETA {
    XrStructureType             type;
    const void* XR_MAY_ALIAS    next;
    XrSpace                     space;
    XrTime                      displayTime;
} XrEnvironmentDepthImageAcquireInfoMETA;

The space field should be set to the XrSpace you want the space to be of the returned pose which the depth map was rendered from. The displayTime field should be set to the displayTime of the current rendered frame as it’s used to compute the pose in case it is time dependent.

The information about the now locked swapchain image is returned the XrEnvironmentDepthImageMETA struct defined as:

typedef struct XrEnvironmentDepthImageMETA {
    XrStructureType                    type;
    const void* XR_MAY_ALIAS           next;
    uint32_t                           swapchainIndex;
    float                              nearZ;
    float                              farZ;
    XrEnvironmentDepthImageViewMETA    views[2];
} XrEnvironmentDepthImageMETA;

typedef struct XrEnvironmentDepthImageViewMETA {
    XrStructureType             type;
    const void* XR_MAY_ALIAS    next;
    XrFovf                      fov;
    XrPosef                     pose;
} XrEnvironmentDepthImageViewMETA;

The nearZ and farZ are the near and far planes defined in an OpenGL projection matrix, and are needed to convert the depth map’s pixel values into metric distances. The format and convention is the same as for regular OpenGL depth textures.

There is a special case you should be aware of: when farZ=inf, you can use an infinite projection matrix as described in Tightening the Precision of Perspective Rendering (see section 3.2: Infinite projection).

When farZ=inf, the bottom-right 2x2 quadrant of the 4x4 projection matrix is defined as:

[ -1   -2*nearZ ]
[ -1    0       ]

Building a projection matrix using common OpenGL helpers and ignoring this special case will introduce NaN values on its coefficients.

Note that the display time and pose of the acquired depth map is likely not the same as the estimated display time and pose for the your app’s frame. To compute the depth of your rendered fragments, you must therefore project the fragments 3D coordinates into the depth map using the provided pose and field-of-view (fov). For an example, see the XrSamples/XrPassthroughOcclusion sample where this method is used to render a scene with Depth API based occlusions.

Did you find this page helpful?