Skip to main content

Hand Skeleton and Hand Gestures

Hand skeleton feature allows you to detect and render hand skeleton 2D.

How to add and use Hand Skeleton feature​

  1. Enable Hand skeleton feature with BanubaSDKBridge.bnb_recognizer_insert_feature function.
// var recognizer = BanubaSDKManager.instance.Recognizer
// ...
var featuresId = BanubaSDKBridge.bnb_recognizer_get_features_id();
BanubaSDKBridge.bnb_recognizer_insert_feature(recognizer, featureId.hand_skeleton, out var error);
  1. Get the detected hand with BanubaSDKBridge.bnb_frame_data_get_hand function.
// frameData = BanubaSDKBridge.bnb_recognizer_process_frame_data(..., frameData, ...)
/// ...
var error = IntPtr.Zero;
var hand = BanubaSDKBridge.bnb_frame_data_get_hand(frameData, Screen.height, Screen.height, BanubaSDKBri, bnb_rect_fit_mode_t, bnb_fit_height, out error);
Utils.CheckError(error);
  1. bnb_hand_data_t contains landmarks for hand skeleton, transformation for landmarks and current detected gesture(For gesture detecting see Hand Gestures section). Apply that landmarks as vertices of you mesh.
[StructLayout(LayoutKind.Sequential)]
public struct bnb_hand_data_t
{
    public bnb_hand_gesture_t gesture;
    public int vertices_count;
    public IntPtr vertices;

    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 16)]
    public float[] transform;
};

How to add and use Hand gestures feature​

With Unity Face AR Hand Gestures feature you can get and use hand gestures triggers in your app.

At the the moment our algorithms are able to recognize 5 different hand gestures:

  • Like 👍
  • Ok 👌
  • Palm ✋
  • Rock 🤘
  • Victory/Peace ✌️
  1. Enable Hand gestures feature with BanubaSDKBridge.bnb_recognizer_insert_feature function.
// var recognizer = BanubaSDKManager.instance.Recognizer
// ...
var featuresId = BanubaSDKBridge.bnb_recognizer_get_features_id();
BanubaSDKBridge.bnb_recognizer_insert_feature(recognizer, featureId.hand_gestures, out var error);

NOTE: featureId.hand_gestures always enables featureId.hand_skeleton.

  1. Get this frame's detected gesture with BanubaSDKBridge.bnb_frame_data_get_gesture function.
// frameData = BanubaSDKBridge.bnb_recognizer_process_frame_data(..., frameData, ...)
/// ...
var gesture = BanubaSDKBridge.bnb_frame_data_get_gesture(frameData, out var error);

Example prefab and script​

You can find HandSkeleton.prefab example prefab under Assets -> BanubaFaceAR -> Hands -> Prefabs. It has an attached HandSkeleton.cs component that contains all needed logic to enable and use hand skeleton and gestures.

  1. Drop the prefab into scene. image

  2. Add render camera reference for the HandSkeleton.prefab. image

  3. Hit Play to see how it works!