Oculus Hand Interaction: Pose detection
Previously, we saw how to set up hand tracking, and we learned about grab interaction, poke interaction and ray interaction. If you haven’t checked out the first part of this series, we highly recommend checking it. In this blog, we’ll learn how to create our hand pose and use the pose detection component to detect it. Upon detection, we’ll perform some action.
Before we start:
- Make sure you have a Unity project with version 39 of the Oculus Integration package imported.
- A scene with the OculusInteractionRig setup for hand tracking.
Prerequisite
You need to have a good understanding of how the hand pose is detected. You can check out the documentation here. To summarize:
There are different hand poses that we can do, like thumbs up, thumbs down, rock, paper scissors, etc. If you have tried out the sample scene that comes along with the SDK you might have seen it already. But, if you want to create your own you need to understand the basic concept of how the poses are detected. The hand pose detection happens in two steps one is shape recognition and the other is transform recognition.
The shape recognizer component considers the state of the finger joints and the transform recognizer component considers the hand’s orientation in 3D space. Each finger is configured by listing the desired state of one or more Finger Features: Curl, Flexion, Abduction, and Opposition. To learn more you can check this documentation shape recognition.

The transform recognizer component checks the position and orientation of the hands. For example, if the palm is facing up, down, facing the face or facing away from the face. It recognizes other parts of the hand too, for example, the wrist and finger. To learn more you can check this documentation on transform recognition.

Creating Custom Pose
There are various hand poses we can create and for this blog I am choosing the Spiderman hand pose. Creating a pose is simple, it’s a scriptable object that comes along with the Oculus SDK, all we need to do is configure it to match the pose we require. So to create a pose:
-
Right-click on the Project window and click on
Create
→Oculus
→Interaction
→SDK
→Pose Detection
→Shape
. You can name the shape as SpiderManPose. -
Let’s now configure each of the fingers to match the SpiderManPose,
- The thumb can be fully opened or slightly bent but not fully closed, it should not be fully bent around the knuckle and there has to be a gap between the thumb and the index finger.
Thus, the thumb feature needs three types of configuration. Those are curl, flexion and abduction. Where the CurlIs Not
Closed
, FlexionIs Not
Closed
and AbductionIs
Open
- The index finger can be fully opened or slightly bent but not fully closed, it should not be fully bent around the knuckle and the gap between the index finger and the middle finger doesn’t matter.
So, the index feature needs two types of configuration and those are curl and flexion. Where the CurlIs Not
Closed
and FlexionIs Not
Closed
.
- The middle finger can be fully closed or slightly bent but not fully open, it should be fully bent around the knuckle and the gap between the middle finger and the ring finger doesn’t matter.
So, the middle feature needs two types of configuration, curl and flexion. Where the CurlIs Not
Open
and FlexionIs
Closed
.
- The ring finger is the same as the middle finger. So the ring feature will also need two types of configuration, curl and flexion. Where the Curl
Is Not
Open
and FlexionIs
Closed
.
- The pinky finger is the same as the index finger. So, the pinky feature will also need two types of configuration and those are curl and flexion. Where the Curl
Is Not
Closed
and FlexionIs Not
Closed
.
- The thumb can be fully opened or slightly bent but not fully closed, it should not be fully bent around the knuckle and there has to be a gap between the thumb and the index finger.
-
We’ll also provide a name for the shape which is the same as the scriptable object SpiderManPose.
With that we have created a pose that can be detected, next we’ll add the components that are required for pose detection.
Pose Detection
For the pose detection to work correctly, we’ll have to make sure the right components are added and the correct parameters are referenced. So, ensure that you’ll follow the exact steps given below to add pose detection to the scene.
-
Create an empty GameObject by right-clicking on the Hierarchy window and selecting
Create Empty
. Name it as PoseDetection. -
Let’s create the pose detection for the left hand. So, create an empty GameObject as a child of PoseDetection GameObejct by right-clicking on it and selecting
Create Empty
. Name it as SpideyPoseLeft. -
Select the SpideyPoseLeft GameObejct and add the following components:
Hand Ref
: This component will take either the left hand or the right hand as a reference.Shape Recognizer Active State
: This component will take a reference hand and a Shape (custom pose) as input. When the pose is detected on the referenced hand it will output true or else false.Transform Recognizer Active State
: This component will take a reference hand as input and check if the hand’s transform matches the set parameters. If the hand transform is within a predefined threshold then the output will be true or else false.Active State Group
: This component will be used to check the output of both theShape Recognizer Active State
andTransform Recognizer Active State
components. It acts as a logic gate.Active State Selector
: This component will take the referenced active state as input and trigger events when it’s true or false.Selector Unity Event Wrapper
: This component will take the selector as input and allow us to perform some action when the events are triggered.
-
Now let’s add the parameters to each of the components
-
Open the OculusInteractionSampleRig prefab to view its children → open the InputOVR to view its children → open the Hands GameObejct to view its children.
Drag and drop the LeftHand GameObject into the Hand parameter of theHand Ref
component. -
Drag and drop the
Hand Ref
component into the Hand parameter on theShape Recognizer Active State
component. Also, drag and drop the custom pose we had created earlier into the Shapes list parameter. -
Drag and drop the
Hand Ref
component into the Hand parameter on theTransform Recognizer Active State
component → selectPalm Up
as the parameter for Transform Feature Configs, also make sure the Palm Up parameter isTrue
→ for the parameter Feature Thresholds select theDefaultTransformFeatureStateThresholds
. -
Drag and drop the
Shape Recognizer Active State
component and theTransform Recognizer Active State
component into the Active States parameter of theActive State Group
component. Make sure the Logic Operator parameter isAND
. -
Drag and drop the
Active State Group
component into the Active State parameter on theActive State Selector
component. -
Drag and drop the
Active State Selector
component into the Selector parameter on theSelector Unity Event Wrapper
component.
-
-
If you want to create the same pose detection for righthand as well, all you need to do is:
- Duplicate the SpideyPoseLeft GameObejct.
- Rename it as SpideyPoseRight.
- Drag and drop the Right Hand GameObejct into the
Hand Ref
component
Before we use the events to carry out some tasks, let's add debug visualizers to see if the pose detection is happening correctly.
Debugging Hand Pose
The SDK comes with a few debug visualizer prefabs which we can make use of to visualize the hand pose. The visual cue will give us a better understanding of how the hand pose is getting recognized. This will allow us to quickly identify the problem and debug it.
-
Create an empty GameObject by right-clicking on the Hierarchy window and selecting
Create Empty
. Name it as PoseDebug. -
Debugging shape recognizer:
- In the Project window search for
HandShapeSkeletalDebugVisual
→ drag and drop the prefab inside the PoseDebug GameObejct. - Drag and drop SpideyPoseLeft GameObject into the Shape Recognizer Active State parameter of the
Hand Shape Skeletal Debug Visual
component. - This prefab will use a line rendered to show the skeletal view of the hand. A green line renderer indicates that the bone of the finger is detected in the correct configuration as defined. Whereas a red line renderer would indicate otherwise.
- You can now hit play and see the debug visualizer at work. You will be able to see that the entire hand skeleton turns green only when the correct hand pose is detected.
- In the Project window search for
-
Debugging transform recognizer:
- In the Project window search for
TransformRecognizerDebugVisual
→ drag and drop the prefab inside the PoseDebug GameObejct. - Drag and drop the LeftHand GameObejct into the Hand parameter of the
Hand Shape Skeletal Debug Visual
component. Also, drag and drop SpideyPoseLeft GameObject into the Transform Recognizer Active State parameter. - Move the prefab such that it’s slightly ahead of the camera.
- This prefab uses a cube to show whether the hand orientation is right or wrong. A green cube would indicate the hand is facing the direction as defined and a red cube would indicate that the hand is not facing the direction defined.
- When you hit play, you’ll be able to observe that the cube turns green when the plan is facing up and red otherwise.
- In the Project window search for
Now that we have verified that the hand pose works correctly, disable the PoseDebug GameObejct and move on to the next section.
Assigning Task
We can make use of the Unity events from the Selector Unity Event Wrapper
component to carry out some tasks. For now, we can disable the Cube GameObject when the pose is detected and enable it when the pose is not detected. So, to do that:
- Select the SpideyPoseLeft GameObejct → under the
Selector Unity Event Wrapper
component and click on the plus sign ➕ at both the event calls. - Drag and drop the Cube GameObejct into that Object parameter.
- Under the When Selected() parameter select
GameObject
→SetActive (bool)
→ make sure the box is unchecked. - Similarly, under the When Unselected() parameter select
GameObject
→SetActive (bool)
→ make sure to check the box ✅
With that, we have finished setting up or scene with pose detection and the task that has to be carried out when the pose is detected.
Testing
Remember that we have added the task to disable and enable GameObject only for the left hand on pose detection. So when the pose is detected on the left hand the cube should disappear and when the pose is no longer detected it should reappear. You can hit the play button and test it out.

Conclusion
In this blog, we saw how to create our pose and use the events to perform some tasks. There are various ways you can use pose detection, it can be used to open menus, locomote, a simple game, learn sign languages and many more. Make sure to create your pose and create some amazing experiences.
__________________
Thank you
Thanks for reading this blog post. 🧡 If you are interested in creating your own AR and VR apps, you can learn more about it here on immersive insiders. Also, if you have any questions, don't hesitate to reach out! We're always happy to help.