How to use Hand gestures feature

This tutorial will guide you on how to get and use hand gestures triggers using Banuba API. The API provides ready-made documented methods making it easy for developers to call AR features in their apps.

At the the moment our algorithms are able to recognize 5 different hand gestures:

• Like 👍
• Ok 👌
• Palm ✋
• Rock 🤘
• Victory/Peace ✌️

Download example

note

Hand gestures tracking feature requires a Hand gestures tracking neural network. Please, make sure your licence plan includes one by contacting your sales manager or fill in the website form to request it.

Integrating hand gestures tracking feature into your app

1. Build your project with Banuba SDK.
2. Use the test_gestures effect or your version of this effect to enable hand gestures feature. If you are using SDK demo app it's enought to copy (if it's not already there) test_gestures effect folder to effects folder and load it on demand using code from below:

Swift

// private BanubaSdkManager mSdkManager;
// ...
mSdkManager.loadEffect("test_gestures")

Java

// private let sdkManager = BanubaSdkManager()
// ...
sdkManager.loadEffect("test_gestures")

note

Hand gestures tracking with trigger function being called in config.js may be added to any of your effects. It can be done by simply adding "hand_gestures" option to "recognizer" property array in config.json file.

3. Process hand gestures tracking data we recieve in the Effect. There is a trigger function being called in JS by SDK every frame while any of the hand gestures being recognized:

   • setGesture(json) - this function recieves a json with an idx of a currently recognized gesture, which possible values are:
     • 0 - none,
     • 1 - like,
     • 2 - ok,
     • 3 - palm,
     • 4 - rock,
     • 5 - victory/peace.

   You may extend this trigger function behaviour by adding your custom logic to config.js from the "test_gestures" effect folder. E.g. play guitar solo audio every time someone shows the "rock":

   function setGesture(json){
       var gestureInfo = JSON.parse(json);
       switch (gestureInfo.idx) {
           case 4:
               Api.playSound("rockme.ogg", false, 1);
               break;
           default:
               break;
       }
   }

   Only your creativity is the limit!

4. Now you can run your application to test hand gestures tracking and your custom logic applied.