View as MarkdownOpen in ChatGPTOpen in Claude

How to use hand gesture recognition triggers in the Banuba Face AR SDK

The Banuba Face AR SDK recognizes five hand gestures (like, ok, palm, rock, victory) in real time. Your effect can read the current gesture on every frame using hand.getGesture() in an onUpdate listener. Use it to fire an animation, a sound or a touchless action in a filter. This guide covers platforms, performance and adding hand gesture recognition to your own effect.

Download example

note

Hand gesture tracking requires the Hand gesture tracking neural network. Please, make sure your licence plan includes one by contacting your sales manager or fill in the website form to request it.

Which hand gestures the Banuba Face AR SDK recognizes

Gesture	Value returned by hand.getGesture()	bnb.GestureType
None (no gesture in frame)	0	NO_GESTURE
Like 👍	1	LIKE
Ok 👌	2	OK
Palm ✋	3	PALM
Rock 🤘	4	ROCK
Victory / Peace ✌️	5	VICTORY

The Banuba Face AR SDK recognizes five hand gestures. Each one reaches your effect as a value returned by hand.getGesture(). Compare it with the named bnb.GestureType constants in your config.js logic. Hand gesture recognition requires the hand gesture tracking neural network listed on the SDK features page, so make sure your license plan includes it by contacting your sales manager or filling in the website form to request it.

Supported platforms

Platform	Hand gestures	Integration guide
iOS	Supported (CPU)	iOS integration
Android	Supported (CPU)	Android integration
Web	Supported (CPU)	Web integration
macOS and Windows	Supported (CPU)	Desktop integration
Unity	Supported through the recognizer API	Unity overview

Hand gesture recognition runs on the CPU and is supported on iOS, Android, Web, macOS and Windows, and on Unity through its own API. On iOS, Android, Web and desktop you load an effect that has the hand gesture recognizer enabled; on Unity you enable the feature in code (see the Unity section below). Hand gesture recognition has shipped in the Banuba Face AR SDK since version 0.38.0 on iOS and Android and since version 1.2.0 on all platforms. See the full per-platform feature matrix.

Real-time performance

Device tier	Real-time FPS
iOS Mid	30
iOS High	30
Android Low	30
Android High	30

Hand gesture recognition in the Banuba Face AR SDK runs at 30 FPS in real time on every device tier Banuba tests, with a maximum recognition distance of 2.5 m. The figures were measured under fixed lab conditions, so treat them as a reference; device, camera resolution and the other features running in the same effect all affect the result. See how the device tiers are defined and the full metric set in the technical specification.

Integrating hand gesture recognition into your app

Follow these steps to get the sample effect running; the next section shows how to move the gesture handling into your own effect.

1. Build your project with the Banuba Face AR SDK.
2. Load the test_gestures effect, or your own effect with hand gestures enabled. In the SDK demo app, copy the test_gestures effect folder into the effects folder if it is not already there, then load it on demand:

Swift

// private BanubaSdkManager mSdkManager;
// ...
mSdkManager.loadEffect("test_gestures", false);

Java

// private let sdkManager = BanubaSdkManager()
// ...
sdkManager.loadEffect("test_gestures")

3. Run the app, show one of the five gestures to the camera and the sample effect reacts. Then edit its config.js as shown in the next section.

Adding hand gesture recognition to your own effect

Start from the test_gestures sample, which includes the hand asset and tracking setup in config.json. When extending an existing effect, merge the sample's hand asset and hand tracker configuration into your effect as well. Its hand asset is named hand_skelet.

Enable recognition in config.js with bnb.scene.enableRecognizerFeature(bnb.FeatureID.HAND_GESTURES), obtain the hand asset, then read its gesture in an onUpdate listener. Extend the listener with your own logic. For example, emit an effect event when someone starts showing the rock gesture:

bnb.scene.enableRecognizerFeature(bnb.FeatureID.HAND_GESTURES)

const hand = bnb.scene.getAssetManager().findHand("hand_skelet")
let previousGesture = bnb.GestureType.NO_GESTURE

bnb.eventListener.on("onUpdate", () => {
    const gesture = hand.getGesture()
    if (gesture === bnb.GestureType.ROCK && gesture !== previousGesture) {
        bnb.scene.addEffectEvent("gesture", { type: "Rock" })
        // Add your animation, sound, or other custom action here.
    }
    previousGesture = gesture
})

The listener runs every frame; comparing with the previous value prevents the action from repeating on every frame while the gesture is held. The sample also emits a gesture effect event when its displayed gesture changes.

The effect folder layout and the config.json fields are described in the effects overview.

Hand gestures in the Banuba Face AR SDK for Unity

On Unity you can enable hand gestures directly on the recognizer and read the gesture from the frame data, without loading a gesture effect. Enabling hand gestures also enables hand skeleton tracking.

1. Enable the feature with BanubaSDKBridge.bnb_recognizer_insert_feature:

var featuresId = BanubaSDKBridge.bnb_recognizer_get_features_id();
BanubaSDKBridge.bnb_recognizer_insert_feature(
    BanubaSDKManager.instance.Recognizer, featuresId.hand_gestures, out var error);
Utils.CheckError(error);

2. In your recognition-result handler, get the hand and read its gesture:

private void OnRecognitionResult(FrameData frameData)
{
    var hand = BanubaSDKBridge.bnb_frame_data_get_hand(
        frameData, Screen.width, Screen.height, BanubaSDKManager.FitMode,
        Application.isMobilePlatform, out var error);
    Utils.CheckError(error);
    if (hand.vertices_count == 0) return;

    var gesture = hand.gesture;
    // Use the gesture in your custom logic.
}

The HandSkeleton.prefab under Assets > BanubaFaceAR > Hands > Prefabs carries a HandSkeleton.cs component with the logic to enable and use hand skeleton and gestures. It subscribes to BanubaSDKManager.instance.onRecognitionResult, exposes the detected Gesture, and removes the feature and listener when destroyed. Full Unity guide.

Other useful resources

• Sample effect: hand gestures detection example using hand.getGesture() in config.js.
• All demo effects: test_gestures and test_HandSkelet are listed under hands.
• Support: having trouble with integration? Ask us for help.

FAQ

Does the Banuba Face AR SDK recognize hand gestures in the browser?

Yes. Web is one of the platforms where hand gesture recognition in the Banuba Face AR SDK is supported, on the CPU, alongside iOS, Android, macOS, Windows and Unity. See the per-platform feature matrix.

Does hand gesture recognition in the Banuba Face AR SDK run in real time on low-end Android phones?

Yes. Under Banuba lab conditions hand gesture recognition runs at 30 FPS on the Android Low tier as well as on Android High, iOS Mid and iOS High, up to 2.5 m from the camera. Actual performance depends on the device, camera resolution and other enabled features. See the metrics and tier definitions in the technical specification.

Can I add hand gesture triggers to my own Banuba Face AR SDK effect?

Yes. Include the hand asset and tracking setup, enable bnb.FeatureID.HAND_GESTURES in config.js, and read hand.getGesture() in an onUpdate listener. Start from the test_gestures sample.

Does the Banuba Face AR SDK also track the hand skeleton, not only gestures?

Yes. Hand skeleton (2D hand landmarks) is a separate Hand AR feature of the Banuba Face AR SDK: on Unity enabling hand gestures also enables hand skeleton tracking, and the test_HandSkelet demo effect displays the skeleton model on the other platforms. See the demo effects.

What do I need in my Banuba license to use hand gesture recognition?

Hand gesture recognition in the Banuba Face AR SDK requires the hand gesture tracking neural network, so check that your license plan includes it with your sales manager or request it through the form on the hand tracking and gesture recognition page.