Meta Developers

Health and Safety Recommendation: While building mixed reality experiences, we highly recommend evaluating your content to offer your users a comfortable and safe experience. Please refer to the Health and Safety and Design guidelines before designing and developing your app.

Scene empowers you to quickly build complex and environment aware experiences with rich interactions in the user’s physical environment. Combined with Passthrough and Spatial Anchors, Scene capabilities enable you to build Mixed Reality experiences and create new possibilities for social connections, entertainment, productivity, and more.

For a complete API reference as part of the OpenXR spec, read XR_FB_scene.

How Does This Work?

Scene includes two important concepts: Space Setup and Scene Model.

Space setup

Space setup is a system flow that lets users walk around and capture their scene to generate a Scene Model. Users have complete control over the capture experience and decide what they want to share about their environment. An app can query the system to check whether a Scene Model of the user’s space exists or invoke the Space setup if needed.

Scene Model

Scene Model is a single, comprehensive, up-to-date representation of the real physical world that is easy to index and query. Scene Model provides a geometric and semantic representation of the user’s space so you can build room-scale mixed reality experiences. For example, attach a virtual screen to the user’s wall or have a virtual character navigate on the floor with realistic occlusion. In addition, you can bring physical world objects into virtual reality. For example, users can see their real desk or couch in the virtual world to play or work more comfortably.

The fundamental elements of a Scene Model are Scene Anchors, with each anchor attached with geometric components and semantic labels. For example, the system organizes a user’s living room around individual anchors with semantic labels, such as the floor, ceiling, walls, desk, and couch. Each anchor is also associated with a simple geometric representation, 2D boundary, or 3D bounding box.

Scene Model is managed and persisted by the operating system. All apps can access Scene Model. Scene Model makes it easy for you to understand the environment around the user and blend virtual content into that environment.

Multiple Spaces

Starting from v62, Space Setup allows the user to scan and maintain multiple rooms (spaces) instead of a single room. The user can scan a new room without erasing a previous room. The OS maintains up to 15 rooms, and might be able to locate some or all of the rooms depending on the current user’s location.

Supporting Multiple Rooms and Application Compatibility

Our SDKs for all platforms (OpenXR, Unity, Unreal) have been designed to support multiple rooms. However, Space Setup before v62 provided only a single room, so your existing applications might not behave as expected when more than one room is available. Existing applications will not have multiple rooms returned, starting from v65, new SDK versions will support multiple rooms being returned by default.

Build Mixed Reality Using Scene

Before you begin with Scene, make sure to use the latest version of the Meta Quest operating system and the the Meta XR Core SDK, which is available individually or as part of the Meta XR All-in-One SDK. Do the following:

Scene functionality is supported on the Meta Quest 2, Meta Quest Pro or Meta Quest 3 operating software version 40 or later. Ensure the headset is connected to the internet and has the latest version installed. To verify the latest version, in the Meta Quest headset, go to Quick Settings > About and check the latest version, which should begin with the number 40 or higher.

Project Setup

Follow the OpenXR Support for Meta Quest and Meta Quest 2 instructions.

In the Android manifest, add the following element to enable Scene and Spatial Anchors capabilities:

<uses-permission android:name="com.oculus.permission.USE_ANCHOR_API" />

Then, you’ll need to include the extension headers specific to Scene and Spatial Anchors in your source code. Set up the following header #includes in your code:

#include <openxr/openxr.h>

Creating an Instance and Session

To use Scene through the FB OpenXR Extensions, you must create an OpenXR instance by enumerating the extension headers in the previous code block. For details on creating OpenXR instances, read Creating Instances and Sessions. For example:

const char* const requiredExtensionNames[] = {
    XR_KHR_OPENGL_ES_ENABLE_EXTENSION_NAME,
    XR_EXT_PERFORMANCE_SETTINGS_EXTENSION_NAME,
    XR_KHR_ANDROID_THREAD_SETTINGS_EXTENSION_NAME,
    XR_FB_SPATIAL_ENTITY_EXTENSION_NAME,
    XR_FB_SPATIAL_ENTITY_QUERY_EXTENSION_NAME,
    XR_FB_SPATIAL_ENTITY_STORAGE_EXTENSION_NAME,
    XR_FB_SPATIAL_ENTITY_CONTAINER_EXTENSION_NAME
    XR_FB_SCENE_EXTENSION_NAME,
    XR_FB_SCENE_CAPTURE_EXTENSION_NAME
};

const uint32_t numRequiredExtensions =
    sizeof(requiredExtensionNames) / sizeof(requiredExtensionNames[0]);

This example code creates the instance:

XrInstanceCreateInfo instanceCreateInfo = {XR_TYPE_INSTANCE_CREATE_INFO};
instanceCreateInfo.createFlags = 0;
instanceCreateInfo.applicationInfo = appInfo;
instanceCreateInfo.enabledApiLayerCount = 0;
instanceCreateInfo.enabledApiLayerNames = NULL;
instanceCreateInfo.enabledExtensionCount = numRequiredExtensions;
instanceCreateInfo.enabledExtensionNames = requiredExtensionNames;

XrResult initResult;
OXR(initResult = xrCreateInstance(&instanceCreateInfo, &instance));
if (initResult != XR_SUCCESS) {
    ALOGE("Failed to create XR instance: %d.", initResult);
    exit(1);
}

This example code creates the OpenXR session (please note we are using OpenGLES instead of Vulkan):

XrGraphicsBindingOpenGLESAndroidKHR graphicsBindingAndroidGLES = {XR_TYPE_GRAPHICS_BINDING_OPENGL_ES_ANDROID_KHR};
    graphicsBindingAndroidGLES.display = app.Egl.Display;
    graphicsBindingAndroidGLES.config = app.Egl.Config;
    graphicsBindingAndroidGLES.context = app.Egl.Context;

    XrSessionCreateInfo sessionCreateInfo = {XR_TYPE_SESSION_CREATE_INFO};
    sessionCreateInfo.next = &graphicsBindingAndroidGLES;
    sessionCreateInfo.createFlags = 0;
    sessionCreateInfo.systemId = app.SystemId;

    OXR(initResult = xrCreateSession(instance, &sessionCreateInfo, &app.Session));
    if (initResult != XR_SUCCESS) {
        ALOGE("Failed to create XR session: %d.", initResult);
        exit(1);
    }

See the android_main function in the attached XrSceneModel sample app for more information.

Acquiring Function Pointers

To use the functions defined in the extension headers above, you will need to connect the function pointers with their implementations. For example:

OXR(xrGetInstanceProcAddr(
    instance,
    "xrEnumerateSpaceSupportedComponentsFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrEnumerateSpaceSupportedComponentsFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceComponentStatusFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceComponentStatusFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrSetSpaceComponentStatusFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrSetSpaceComponentStatusFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceUuidFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceUuidFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrQuerySpacesFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrQuerySpacesFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrRetrieveSpaceQueryResultsFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrRetrieveSpaceQueryResultsFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceBoundingBox2DFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceBoundingBox2DFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceBoundingBox3DFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceBoundingBox3DFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceSemanticLabelsFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceSemanticLabelsFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceBoundary2DFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceBoundary2DFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceRoomLayoutFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceRoomLayoutFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrGetSpaceContainerFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrGetSpaceContainerFB)));
OXR(xrGetInstanceProcAddr(
    instance,
    "xrRequestSceneCaptureFB",
    (PFN_xrVoidFunction*)(&app.FunPtrs.xrRequestSceneCaptureFB)));

In this example, the app is of type ovrApp, a struct defined in the attached XrSceneModel sample app. The FunPtrs field in the sample app is a struct defined as follows:

struct ovrExtensionFunctionPointers {
   PFN_xrEnumerateSpaceSupportedComponentsFB xrEnumerateSpaceSupportedComponentsFB = nullptr;
   PFN_xrGetSpaceComponentStatusFB xrGetSpaceComponentStatusFB = nullptr;
   PFN_xrSetSpaceComponentStatusFB xrSetSpaceComponentStatusFB
 = nullptr;
   PFN_xrGetSpaceUuidFB xrGetSpaceUuidFB = nullptr;
   PFN_xrQuerySpacesFB xrQuerySpacesFB = nullptr;
   PFN_xrRetrieveSpaceQueryResultsFB xrRetrieveSpaceQueryResultsFB = nullptr;
   PFN_xrGetSpaceBoundingBox2DFB xrGetSpaceBoundingBox2DFB = nullptr;
   PFN_xrGetSpaceBoundingBox3DFB xrGetSpaceBoundingBox3DFB = nullptr;
   PFN_xrGetSpaceSemanticLabelsFB xrGetSpaceSemanticLabelsFB = nullptr;
   PFN_xrGetSpaceBoundary2DFB xrGetSpaceBoundary2DFB = nullptr;
   PFN_xrGetSpaceRoomLayoutFB xrGetSpaceRoomLayoutFB = nullptr;
   PFN_xrGetSpaceContainerFB xrGetSpaceContainerFB = nullptr;
   PFN_xrRequestSceneCaptureFB xrRequestSceneCaptureFB = nullptr;
};

Did you find this page helpful?