Object Interaction Part 1: Common Interaction Patterns

This is the first in a series of four articles that review common usage patterns, technical issues, trade-offs, and pitfalls that are important to consider when implementing a VR interaction system. Our first post covers common interactions like pushing buttons, pulling levers, carrying objects, firing weapons, and opening doors.

The fundamental goal of any interaction system is to be as easy to learn as possible, with consistent rules for users so they can anticipate how to interact with anything in the virtual world, regardless of any special behavior an object might have. When an accurate physical simulation is combined with logic that coordinates input, animation, and scripted behavior, the result is an experience where objects consistently behave as expected during player interactions.

Common Interaction Patterns
Interacting with a VR environment generally involves grabbing, pushing, carrying, or moving objects. Held objects like tools and weapons typically require specific orientations and low-latency hand tracking. Environmental objects are usually pushed or pulled while constrained in some way, like levers or buttons on a control console. Many interactable objects have some kind of side effect or activation behavior. We’ll describe some of these common behaviors below, but the first thing to consider is how the interaction should happen in the first place.

First Contact: How to Begin Interaction with an Object
Physical interaction with hand colliders can be a very satisfying experience and is encouraged when possible. Even if an object has no specific purpose, if it looks like something that can be grabbed or moved, people will expect to be able to do so in VR, even if it just means sweeping objects off a table or throwing them around. It’s easy to imagine how motion controllers should interact with objects—we move the controller into an object, and either the object moves in response to hand collisions, or an object-specific interaction will begin. To make content more accessible to people with smaller play areas or mobility limitations, add features for picking up or activating objects in the distance. In some cases, it may be preferred to move the content to fit the play area defined by the user.

Another thing to consider is whether or not interaction should require deliberate activation, or if activation can occur because of collisions with arbitrary objects that might be moving around in the scene unpredictably. This is important because using trigger volumes that detect hand colliders can be more efficient and predictable than responding to collisions with arbitrary objects. For some titles, it will make sense for every interactable object to respond to physics. For others, it can provide a more consistent experience if reactions only occur in direct response to the user’s intentions. In either case, it’s important to consider which interactions should require an explicit action by the user because this decision can have a significant impact on both the design and performance of the app.

Interactable Objects
Common interactions include pushing buttons, pulling levers, carrying objects, firing weapons, and opening doors. Some will trigger a one-time animation, others might move objects to match the position of a controller, and others might move an object to match a specific orientation relative to the controller. Interaction may trigger dependent gameplay logic, like when activating a weapon. Some interaction systems will provide an abstraction layer responsible for connecting standard inputs like grab, activate, and release to a set of interfaces that objects can respond to in a way that makes sense. Abstraction layers for input are helpful—not only do they make it easier to support multiple platforms, they also make it much easier to remap buttons based on changes in gameplay context or an evolving design.

Generally, interactable objects have collision primitives that let users push them around with their hands and trigger volumes that enable logic to pick the object up when a button is pressed. Hand poses and animations that are specific to objects or grips shared among objects can improve the quality of the experience. In some cases, it may be useful to have an inverse kinematic system that moves the fingers individually to the surface of arbitrary-held objects so that fewer hand poses and animations need to be prepared in advance.

This image shows a simple toggle switch that activates when something hits one of the colliders. When activated, the ToggleSwitchTrigger script changes the rotation of the Toggle object in the hierarchy to the right value for each mode of the switch. Despite being a very simple arrangement with no joints or physics to tune, it may be all some projects need for a toggle switch.

Next, we’ll look at many of the common types of interactable objects and the behaviors they typically express when triggered. For larger objects like doors where physical simulation makes sense, attaching them to the player’s hands and relying on the physics system to constrain their positions can result in a natural behavior. Some objects, like small buttons, may not need to be physically simulated and might instead rely on trigger volumes for activation and procedural animation to move them as needed.

It’s important to consider what should happen when the user pulls an interactable object through their avatar’s collision volume. For example, if the player pulls a door into the player’s collision volume, this could cause unpredictable physical response by either the door or the player. If the player happens to be standing in the path of the door’s motion, the physics engine could behave unpredictably, potentially pushing the player out of the map or causing other side effects. The easiest way to avoid this problem is to disable collisions with objects currently held by the user, but this might not be compatible with the overall design. The ideal solution is to tune the physics so that the player’s physics don’t move in reaction to held objects.

Buttons
One of the first decisions to make when setting up a button is whether it should use physical simulation with rigid bodies and joints or trigger volumes with procedural animations. Physically simulated buttons are more complicated to set up than triggered buttons, but the extra effort might be worthwhile, depending on the context.

Trigger volumes are often a good fit for smaller buttons that activate immediately on contact. The small movement these require can be set up with a procedural animation very easily, which is generally less work than setting up and tuning a physical behavior. Whenever something is in the trigger volume, the button activates and animates to the pressed position. The trigger volume might limit detection to the user’s hands in order to prevent accidental activation from environmental objects. Filtering to an extended finger can provide more precise control than the entire hand, which can be useful for a virtual keyboard where many buttons are right next to each other. This is likely to require training the user because of the subtlety of the gesture.

Here’s an example of a small keypad, where each button has a spherical trigger volume the user can activate with their extended index finger.

Because of the small size, there’s no need for precise colliders or physically simulated movement. The script will trigger a quick animation of the button using the Push Curve shown in the screenshot when the user triggers it. This example also demonstrates the value of splitting the animations into two parts—one for the activation, and another for the deactivation when the user moves off the button. In this configuration, the button pops back up before the user moves their finger away. Having the button deactivate while the finger is still there isn’t as realistic as it could be with a little more work, but it’s still less work than physically simulating the button.

Physical simulation is perhaps most useful with larger buttons where the button moves a larger distance and the user is more aware of its movement. If it’s important to exclude unintended activation by objects in the environment, physical simulation may not be the best choice because it’s more complicated to have buttons selectively disable collision response during the physical simulation than it is to disregard a trigger volume event. Further complicating the implementation is that a physically simulated button will require additional logic to prevent activating gameplay logic until it’s been at least partly depressed.

Doors
Swinging doors are often physically simulated objects on a hinge joint, possibly configured with angle limits if there’s no environmental geometry to limit their motion. While a standard collision response can be used to open doors, it may be preferable to have a door that remains latched, requiring the user to grab a handle or perform some unrelated gameplay action before the door will open. There are several ways to prevent the user from opening a latched door; the simplest is to change it to a kinematic object. If necessary, the door can use springs on the hinge joint to close automatically when the player releases it.

Sliding doors tend to be simpler than swinging doors, with the obvious difference in how they move between two points. It’s common for sliding doors to open and close with an animation based on trigger volumes that guarantee the door will never push an object. If something is in the way, they simply retract into the wall or ceiling. Many games use sliding doors because of how easy they are to set up and how effective they are at avoiding collision problems. While sliding doors usually have an automatic mechanism, the game design might require a physical simulation. If this is the case, they’ll behave virtually the same as a drawer, explained below.

Drawers
Moving along an axis between two points, drawers are functionally similar to sliding doors. They may require a specific grip to open, include springs to close them automatically, or respond to hand collisions. Physical simulation isn’t necessarily required—moving them to match the attached hand position can work as long as collision behavior is not a concern.

Control Dials
The user will grab and rotate control dials that usually drive content-specific scripted behavior. Dials may snap to fixed positions when released to support a fixed number of states. Typical control panel dials don’t benefit from physical simulation; they simply match the change in orientation of the attached motion controllers. It’s important to note that the dials don’t change state when the interaction begins—values change only when the relative orientation of the attachment changes. Application logic can query the current value of the dial or subscribe to events that will be raised when the value changes.

Levers
There are many types of levers, including spring-return, linear or stepped positions along a line, and more complex shapes as seen in manual transmission gear-shifters. Levers provide data to the application in much the same way that dials do, with a current state value and state change events.

Sliders
The following screenshot shows a simple slider configured with a script that will move the slider along the housing between SliderRange0 and SliderRange1 as long as the user’s extended fingertip is within the sphere collider. This control is small, so it makes sense for fingertips resting on top of the slider to be able to move the slider. Another variety of this technique is to use two colliders, one above and another below the slider, which would limit movement to the push direction instead of following the finger.

Coming Up Next
Stay tuned for Part 2, “Locomotion and Held Objects,” where we’ll cover patterns related to movement and carrying objects.