Use Simultaneous Hands and Controllers (Multimodal)
Updated: Jul 15, 2024
Multimodal input provides simultaneous tracking of both hands and controllers. It also indicates whether the controller(s) are in hand or not. Multimodal input allows users to enjoy the benefits of both worlds: they can use hands for immersion, and controllers for accuracy and haptics. When enabled, Multimodal input overrides other transition methods, including auto transitions and double tap.
- Hand and controller gameplay: Use a controller in one hand as a tool or weapon (high accuracy, haptics, wide FoV), while keeping your other hand free to interact with objects or cast spells (high immersion).
- Single controller gameplay: If your experience only requires one controller for most interactions, allow the users to only pick up one controller and use their other hand for immersion and casual interactions when needed (for example, use a controller as a racket and use your free hand to pick up the ball and interact with menus).
- Use hands and controllers interchangeably for comfort: Use one hand for direct interactions with controls, while the other hand is holding a controller as a “remote control” for low effort indirect interactions.
- Instant transition between hands and controllers: With hand tracking active, we can instantly identify when the controllers are no longer in the hand. This minimizes the need for manual transitions and solves the problems of slow or failed auto transitions.
- Find my controllers: With controllers now tracked when using hands, the app can show the user where the controllers are when they want to pick them up. This allows smoother transition back to controllers without having to break immersion by turning on passthrough/ taking off the headset.
- The ‘in hand’ signal is based on various imperfect signals including hand and controller pose and controller signals. As a result, the system may indicate that the controller not in hand in certain scenarios where tracking is lost or inaccurate, or controllers are held still for some time. It is recommended to design with that limitation in mind (for example, avoid dropping objects from a hand due to false short transitions from controllers to hands).
- When the pause function is called, the application will switch back into the “non-simultaneous” mode that traditional hands+controllers apps run in, where the user can use either hands or controllers at a given time. The tracking system may take a few seconds to recognize and decide on the correct input to enable, depending on whether the user is holding controllers when this happens.
- When using Link on PC, pose data for controllers is unavailable when you’re not actively using them (such as when they’re lying on a table).
- Quest 2
- Quest Pro
- Quest 3
- Quest 3S
- Unity version 2022.3.15f1 and above (Unity 6+ is recommended)
- Meta XR Core SDK v62 and above
- Multimodal input is incompatible with Inside-Out Body Tracking (IOBT) and Full Body Synthesis (FBS). You shouldn’t enable them together.
- Multimodal input cannot be enabled together with Fast Motion Mode (FMM). If both are enabled together, Multimodal will take precedence. As FMM is defined in the manifest, you may enable FMM at the app level, and then turn on multimodal only in specific experiences where FMM is less important.
- On Quest 2, Multimodal cannot be enabled together with LipSync
- Passthrough, Multimodal input, and Wide Motion Mode (WMM) cannot be enabled together. If they all are turned on together, the system will disable WMM.
- Full compatibility with capsense hands.
- Full compatibility with haptics.
Multimodal can be programmatically interacted with via its two OpenXR extensions, XR_META_simultaneous_hands_and_controllers and XR_META_detached_controllers.
The XR_META_simultaneous_hands_and_controllers extension lets you programmatically toggle multimodal. The extension has two methods.
xrResumeSimultaneousHandsAndControllersTrackingMETA(XrSession session, const XrSimultaneousHandsAndControllersTrackingResumeInfoMETA* resumeInfo);
This method turns on simultaneous hands and controllers mode.
xrPauseSimultaneousHandsAndControllersTrackingMETA(XrSession session, const XrSimultaneousHandsAndControllersTrackingPauseInfoMETA* resumeInfo);
This method turns off simultaneous hands and controllers mode.
The extension also defines a System properties struct, XrSystemSimultaneousHandsAndControllersPropertiesMETA. The struct checks if the multimodal extension is supported. The struct has these fields.
- XrStructureType type;
- void* XR_MAY_ALIAS next;
- XrBool32 supportsSimultaneousHandsAndControllers;
Here’s an example of how you’d create an instance of the struct and then start multimodal.
XrSimultaneousHandsAndControllersTrackingResumeInfoMETA resumeInfo = { XR_TYPE_SIMULTANEOUS_HANDS_AND_CONTROLLERS_TRACKING_RESUME_INFO_META};
OXR(xrResumeSimultaneousHandsAndControllersTrackingMETA_(GetSession(), &resumeInfo));
The XR_META_detached_controllers extension lets you track
controllers when they are not being held. It does this by introducing two new paths.
/user/detached_controller_meta/left/user/detached_controller_meta/right
These paths may have interaction profile bindings suggested using any
interaction profile, (excluding those for hand tracking), that are accepted
for /user/hand/left and /user/hand/right, respectively.
Furthermore, the component paths for these two new paths must match those
available for their corresponding top level paths.
The same interaction profile can be used to suggest bindings for both
/user/hand/ and /user/detached_controller_meta paths, but
the runtime must only populate one of these at any given time. For example,
/user/hand/left or /user/detached_controller_meta/left,
and either /user/hand/right or
/user/detached_controller_meta/right.
Actions suggested for these top level user paths will become active when
their corresponding action set is active and the corresponding controller is
detected to be not held in the user’s hand.
In-hand detection is left to the implementing runtime.
The runtime may use proximity data, orientation data, or any other data for
both hands and both controllers for in-hand detection. Controllers with handedness should be considered in hand when held by their
corresponding hand (a left controller held in the right hand should be
considered detached, for example).
This is done individually such that any combination of held or not held
controllers may be supported (both or neither controller held, or one
controller held in one hand for left or right controllers).
The runtime must return the currently active interaction profile from
xrGetCurrentInteractionProfile, for these new top-level paths, and
must send an INTERACTION_PROFILE_CHANGED event when a change happens to the
current interaction profile.
An application can use this event and value from
xrGetCurrentInteractionProfile to determine when the user put down or
picked up a controller.
Interaction Profile Additions When this extension is active, a runtime must support binding these new
paths for all interaction profiles that support /user/hand/left/grip and
/user/hand/right/grip, including those interaction profiles enabled through
extensions.
Actions bound to the detached_controller_meta input component paths must
behave as though those paths were listed in the original definition of an
interaction profile.
Detecting hand<->controller modularity change per-hand:
// In event polling code
XrResult result = xrPollEvent(instance, &event);
if (result == XR_SUCCESS) {
// ...
switch (event.type) {
case XR_TYPE_EVENT_DATA_INTERACTION_PROFILE_CHANGED: {
for (int handIndex = 0; handIndex < NUM_HANDS; ++handIndex) {
XrInteractionProfileState profileState;
xrGetCurrentInteractionProfile(session, path, &profileState);
// Inspect profileState to understand if this hand is now holding a
// controller, or if the controller has been placed somewhere
if(currentState[handIndex] != profileState.interactionProfile){
LOG("Detected state change in {} from {} to {}", handIndex,
currentState[handIndex], profileState.interactionProfile);
// Do whatever is required when state changed
transitionHandState(handIndex, profileState.interactionProfile);
}
}
break;
}
}
}
How can I confirm multimodal is running on my headset? Confirm that the detached controller action path is providing data.
Can I evaluate the feature on my headset without changing my code? No.
Switching between controllers and hands doesn’t work in the sample. Ensure that you’re running on Quest 2 (with Meta Quest Pro controllers paired to your headset), Quest 3, or Quest Pro.
