In
First Encounters, an MR experience that comes pre-loaded on every Meta Quest 3, we focused on exploring additional ways we could make use of Scene Mesh’s (
Unity |
Unreal) capabilities to design an even more interactive experience. Extending ideas we previously developed in
Discover and
Phanto using Scene Anchors and Scene Mesh, we intended to wow players with a set piece and core mechanic in which puffians would smash through players’ walls, opening their room to another world.
Developing this functionality allowed us to take away a number of insights that can help you build similar MR experiences. Keep reading to learn about our experience using and extending Scene Anchors and Scene Mesh in First Encounters so you can accelerate your journey with these capabilities.
Setting the Scene
First Encounters was created to showcase the type of experiences MR can unlock on Quest 3. While we would be introducing a number of fantasy elements into the player's space, we wanted to make sure that every part of the experience felt physical, grounded, and had realistic interactions. For the game to feel seamless and engaging, it was very important for any new elements to become a part of the player’s world.
In order to achieve this, we made extensive use of the Scene API (
Unity |
Unreal |
OpenXR |
WebXR).
Scene Anchors provide the ability to understand a user’s space in terms of labeled primitives, the most fundamental being walls, floors, and ceilings. These three primitives can be represented through this API as simple planes that can be queried to understand a quick and basic breakdown of a user’s space. Scene Mesh extends our understanding further by providing a scanned triangle mesh of the user’s room that does a better job of approximating the physical space.
By applying a Passthrough shader to the material of the Scene Mesh, the user would see their room as normal—until we start punching holes in the mesh.
Prototyping in Unity
To allow room to experiment with mechanics while ensuring consistency across them, we decided to punch real holes in the Scene Mesh rather than fake holes visually using a decal or shader. This was extremely quick to prototype and demonstrate using a custom naïve hole punching algorithm:
- Cast a ray to determine the point on the Scene Mesh around which to punch a hole
- Collect all triangles with vertices within a radius
- Squish those triangles vertices to a fixed position outside of the world
This naïve approach worked extremely well for prototyping purposes, but of course had significant limitations. First, the visual quality of the holes produced did not match what we were looking for. Due to a combination of inconsistent triangle size in the Scene Mesh and our choice of how to select triangles, this approach would sometimes lead to large and jagged holes being created. Using a tessellator and decimator gave us one solution for balancing performance and visual consistency when shattering walls.
The hole punching approach avoids creating extra work that would be created when resizing the mesh, but it still triggers Unity/PhysX to cook the physics mesh again. Cooking a concave mesh with a potentially large number of triangles can take a long time. The default Mesh APIs make this a stop-the-world process in Unity, but it can be parallelized and made asynchronous with the Jobs system.
As the cooking process is an engine-side feature within Unity, reducing the latency between modification time and being able to query and display the resulting mesh would require either pre-processing to move that cost upfront, or reducing the mesh size.
Evaluating Existing Solutions
As this feature was core to our game's design but provided a significant unknown in terms of technical effort, we started multiple investigations into existing Unity packages and libraries for mesh destruction. After assessing a number of packages and libraries, we ended up dismissing them for this project for a number of reasons:
- Mesh destruction libraries typically required convex meshes or preferred them for performance reasons. Scene Mesh provides a concave mesh.
- Libraries typically assume that a mesh is a filled volume rather than a hollow one. Scene Mesh provides a hollow mesh which when smashed apart with these libraries, would suddenly become filled if not pre-processed and split apart.
- Libraries often expect a low level of detail and triangle count for meshes to be broken apart, and performance quickly degrades for high triangle count meshes.
- Any library that required the mesh to be available for pre-processing within the Unity Editor could not be used as the Scene Mesh would be unique to each headset at runtime.
- Most libraries were designed for desktop systems and not designed for the tight performance requirements necessary for an MR game on Quest 3. Even if the above issues could be fixed or worked around, core performance issues in the libraries would have prevented shipping a smooth experience.
Fast Code to Break Things
In order to meet our requirements, it became obvious that we needed a custom system. The new DestructibleMesh system would approach the problem by using Voronoi partitioning to sort the Scene Mesh triangles into submesh chunks, which could be given their own GameObjects within Unity and individually disabled as they were destroyed. This allowed us to move the processing cost to the start of the game.
The DestructibleMesh system was written to use the Unity Jobs System for parallelization and the Burst compiler for fast native performance and SIMD vector instructions. In order to sort triangles, the DestructibleMesh system performs the following tasks:
- The game provides the system with an array of points defining the Voronoi cell positions on the Scene API planes. These points are uniformly distributed and simplex noise is used to add some randomness while maintaining a relatively consistent distance from point to point. (Note that distributing points completely randomly would lead to chunks that could be very large, very small, or empty)
- The system first checks to see which Voronoi cell each triangle belongs to by comparing the squared distance between the triangle centroid and the array of points defining the Voronoi cell positions. The index of each Voronoi cell position is used as a submesh ID and the system proceeds to label each triangle with a submesh which should own the triangle based on this partitioning scheme.
- The system then overwrites the calculated submesh owner ID if the triangle is within an area that should be reserved for Health & Safety. The reserved area algorithm determines if removing the triangle could lead to a player bumping into their environment and assigns a special owner ID to sort these triangles into their own mesh. This algorithm is described in more detail below.
- With the task of understanding where to sort each triangle complete, iterate over each triangle, incrementing the respective submesh's triangle count and add each vertex to a bitmap that keeps track of which vertices to copy to the final vertex buffer. Because vertices may be shared between multiple triangles and these triangles may not both be in the destination submesh, we need to make copies to the final vertex buffers only where necessary.
- With our number of triangles and number of vertices per submesh known, our destination vertex and index buffers can be reserved for each submesh. At this point, we walk through our vertex map and populate each submesh's vertex buffer.
- Copy and update each triangle's indices for their respective submesh's index buffers.
- Asynchronously wait for all of the above jobs to complete on the Jobs system.
- Commit and finalize the changes we prepared to the Unity Meshes using the relevant Mesh API for concurrent Jobs system Mesh access. Meshes may be used for rendering at this point.
- Cook the PhysX Meshes on the Job System.
- Asynchronously wait for cooking to complete. Meshes may be used for Physics from this point forward without the risk of incurring a last minute stop-the-world processing step.
Note that throughout these steps, the system makes heavy use of parallelization, SIMD vector instructions, cache-efficient data access patterns, and reduced or processor predictable branching for efficient performance. These exact implementation details are glossed over for simplicity's sake in this high-level explanation.
Reserving a Safe Mixed Reality Play Space
In order to reserve an area of the player's room for Health & Safety, we needed an approach that was carefully thought out. Our goal was a solution that was fast and universally effective across a large variety of room layouts and clutter to help prevent Scene Mesh destruction from creating accidental tripping and collision hazards. The algorithm works simply:
Diagram A (Reserved Bounds in Blue)
- Take the Boundary of our room as a top-down, closed-loop 2D polygon. This can be retrieved from the Scene API through the floor’s OVRScenePlane component.
- Deflate the polygon by pushing each line segment along its inward orthogonal vector by a tuned distance of 0.3m.
- From 0.3m below the ceiling, extend this polygon downward and any triangles with centroids within the resulting bounds are reserved as indestructible.
- If any triangle centroid is below a fixed reserved height plane at 0.75m, always reserve that triangle to provide an indicator of where walls & obstacles along walls continue to exist.
Future Work
Now that First Encounters has shipped, the DestructibleMesh system is being improved to support extensible operations that sort and modify triangles both as pre-processing and as real-time steps at runtime. Operations can be queued by the user, and multiple meshes will be able to be queued to be modified on the Jobs system. In addition to making Scene Mesh modification and Health & Safety features easy to implement for future projects, this will also allow experimentation with different VFX and more precise shattering in the room destruction visuals for future updates of First Encounters.
Thinking Outside the Box
Scene Mesh and Scene Anchors offer many new possibilities to drastically change how users see and interact with their space, but despite the benefits of these new capabilities, there are currently some limitations to keep in mind while working with them.
While developing First Encounters, we were careful in how we designed our wall destruction mechanics, knowing that Scene Mesh does not offer texture data for the mesh. This means that we could not deform the mesh by having it bash inwards or outwards, as we would not be able to deform objects in Passthrough. When chunks of a user’s room are shattered, visuals for the broken pieces of walls and ceilings use a generic texture, but this is hidden by the combination of various VFX particles and keeping these pieces short lived.
It is also worth quickly noting that within Unity, certain queries such as Collider.ClosestPoint() should be avoided with Scene Mesh. Behind the scenes, this method in particular uses
Physics.ClosestPoint(), and neither method works with concave meshes. When it comes to concave MeshColliders, these methods will fail silently, returning the original point passed in. For certain use cases,
Physics.ComputePenetration() offers a potential alternative approach when working with Scene Mesh.
Blast Off
There is a lot of potential for you to subvert and expand what people can expect from MR. APIs for Scene Understanding, such as Scene Anchors and Scene Mesh, provide new ways to tightly integrate virtual elements into the real world in ways that feel natural. I hope that sharing some of the insights gained from working with these new capabilities on First Encounters helps developers expand the definition of what MR can be.
