A hand pose is defined by shapes and transforms. Shapes are boolean conditions about the required position of the hand’s finger joints. Transforms are boolean conditions about the required orientation of the hand in the world space. A pose is detected when the tracked hand matches that pose’s required shapes and transforms.

This topic explains poses, shape and transform recognition, and the criteria used to determine if a pose is detected. It also describes debug components you can use to visually debug poses. To make your own custom hand pose, see Build a Custom Hand Pose.

Interaction SDK includes six ready-to-use example pose prefabs. You can define your own poses using the patterns defined in these prefabs. You can experiment with pose detection using these pose prefabs in the PoseExamples sample scene.

Each pose prefab has these components:

Poses include one or more shapes. A shape is a set of boolean conditions about the position of one or more fingers. The conditions are defined using Finger Features states. If a tracked hand meets these conditions, the shape becomes active. If all the shapes in the pose are active and the transform is also active, the pose is detected. A pose’s shapes are stored as ShapeRecognizer assets in the pose’s ShapeRecognizerActiveState component.

ShapeRecognizerActiveState checks the shapes that make up a pose against the state of every finger. If they all match, the ShapeRecognizerActiveState becomes active.

ShapeRecognizer is a ScriptableObject that defines a shape. To define a shape, it uses a set of rules called Feature Configs. Feature Configs specify a required position (state) for each of the five fingers. It defines state using at least one of the finger features: curl, flexion, abduction, and opposition. ShapeRecognizer is referenced by the ShapeRecognizerActiveState component to determine if a pose is active.

FingerFeatureStateProvider provides the finger states of the tracked hands and contains the state transition thresholds for each finger. It’s referenced by the ShapeRecognizerActiveState component.

FingerFeatureStateThresholds is a ScriptableObject that defines the state thresholds for each finger feature. A state threshold is a set of boundaries that determine when a finger has transitioned between states. For example, the curl feature has 3 states: open, neutral, and closed. The state thresholds for curl use an angle in degrees to define when the finger’s state has changed from open to neutral, neutral to closed, or vice-versa.

Interaction SDK provides four sets of default state thresholds, which are under DefaultSettings/PoseDetection:

Given the transition between two states, A <> B:

If the current state is “A”, to transition up to “B” then the angle must rise above the midpoint for that pairing by at least (width / 2.0) for “Min Time In State” seconds.

If the current state is “B”, to transition down to “A” then the angle must drop below the midpoint for that pairing by at least (width / 2.0) for “Min Time In State” seconds.

So for Curl, to transition:

Finger Features are specific finger positions that let you define a shape. There are four features:

Represents how bent the top two joints of the finger or thumb are. This feature doesn’t take the Proximal (knuckle) joint into consideration.

States:

The extent that the Proximal (knuckle) joint is bent relative to the palm. Flexion is only reliable on the four fingers. It can provide false positives on the thumb.

States:

Abduction is the angle between two adjacent fingers, measured at the base of those two fingers. Meaning the angle between the given finger and the adjacent finger that’s closer to the pinky. For example, Abduction for the index finger is the angle between the index and middle finger.

States:

Note: Abduction on Pinkie is not supported.

How close a given fingertip is to the thumb tip. Can only be used on index, middle, ring, and pinky fingers.

States:

Poses consist of one or more transforms. The transform of the hand only represents the orientation and position. The orientation is only evaluated relative to the WristUp, WristDown, PalmDown, PalmUp, PalmTowardsFace, PalmsAwayFromFace, FingersUp, FingersDown, and PinchClear transforms.

A pose’s required transforms are listed in the pose’s TransformRecognizerActiveState component. During hand tracking, the hand’s transforms are compared to the pose’s transforms. If both sets of transforms match and all the shapes in the pose are active, then the pose is detected.

TransformRecognizerActiveState takes a hand, the current state of the hand’s transforms, and a list of transform feature configs, and a transform config. To get the current state of the hand’s transforms, it uses the GetHandAspect method to retrieve the TransformFeatureStateProvider component. That component reads the raw feature values and quantizes them into TransformFeatureStates using the TransformConfig you provide in this component.

TransformFeatureStateProvider provides the transform states of the tracked hand. It’s referenced by the TransformRecognizerActiveState component.

Note: Once you register a specific configuration, the RegisterConfig method can then query the state of each state tracked for configuration. It leverages FeatureStateProvider to drive state-changing logic.

Velocity recognition components detect motion, whereas shape recognition only detects static poses. For example, shape recognition can detect a hand in a thumbs-up pose, but cannot determine if the hand is moving upward while in that pose.

There are two velocity recognition components. Both components get joint data from the JointDeltaProvider component.

IActiveState components can be chained together using the Sequence component. Because Sequences can recognize a series of IActiveStates over time, they can be used to compose complex gestures. For examples of complex gestures, see the GestureExamples sample scene.

Sequence takes a list of ActivationSteps and iterates through them as they become active. Each ActivationStep consists of an IActiveState, a minimum active time, and a maximum step time. These steps function as follows:

Once the final ActivationStep in the Sequence has completed, the Sequence becomes active. If an optional KeepActiveWhile IActiveState has been provided, the Sequence will remain active as long as KeepActiveWhile is active.

The last phase of a Sequence is the optional cooldown phase, the duration of which can be set in the RemainActiveCooldown field. A Sequence that is deactivating will wait for this cooldown timer to elapse before finally becoming inactive.

SequenceActiveState is an IActiveState wrapper for a Sequence, and can either report active once the Sequence has started, once the Sequence steps have finished, or both.

The classes described here implement the underlying functionality, but will not need to be modified in order to use Pose Recognition.

FeatureStateProvider is a generic helper class that keeps track of the current state of features, and uses thresholds to determine state changes. The class uses buffers to state changes to ensure that features do not switch rapidly between states.

A generic editor class that defines the look and feel of dependent editor classes such as FingerFeatureStateThresholdsEditor and TransformStatesThresholdsEditor.

Lets you define editor-visible descriptions and hints and values to aid users in setting thresholds for features.

IFeatureStateThreshold is a generic interface that defines the functionality of all thresholds used in hand pose detection.

IFeatureStateThresholds defines a collection of IFeatureStateThresholds, specific to a feature type, whether that is finger features or transform features.

IFeatureThresholds provides an interface to a collection of IFeatureStateThresholds as well as MinTimeInState (i.e. a minimum threshold of time a feature can be in a certain state before transitioning to another state).

An IActiveState that tests to see if a hand joint is inside a collider. If a SphereCollider is specified then its radius is used for checking, otherwise the script relies on the collider’s bounds. This class is useful in case a developer wishes to see if the hand joint exists within a certain volume when a pose is active.

Can be attached to an object that needs to be anchored from the center eye. Position and rotation offsets can be specified, along with options to toggle the roll, pitch and yaw of the latter. One can combine this with a ColliderContainsHandJointActiveState to position a collider relative to the center eye.