Skip to main content

Banuba SDK Web AR Integration tutorials

Banuba WebAR SDK depends on BanubaSDK.data and BanubaSDK.wasm (or BanubaSDK.simd.wasm if you are targeting SIMD) files. By default the SDK expects these files to be accessible from the application root i.e. by the /BanubaSDK.data, /BanubaSDK.wasm (/BanubaSDK.simd.wasm) links. It must be taken into consideration when working with application bundlers like Vite, Rollup or Webpack.

Generally speaking one should be able to put BanubaSDK.data, BanubaSDK.wasm and BanubaSDK.simd.wasm files into the application assets folder (usually public/) and get BanubaSDK properly loading these files.

But you may want to place the files somewhere else, that case the locateFile property of the Player.create() method should help you to set-up BanubaSDK properly.

Bundlers​

Ordinary bundlers have no issues with the Banuba WebAR SDK and it should be easy to set up the SDK with them.

Vite​

import { Player, Module /* ... */ } from "@banuba/webar"
// vite uses special ?url syntax to import files as URLs
import data from "@banuba/webar/BanubaSDK.data?url"
import wasm from "@banuba/webar/BanubaSDK.wasm?url"
import simd from "@banuba/webar/BanubaSDK.simd.wasm?url"

import FaceTracker from "@banuba/webar/face_tracker.zip?url"
import Background from "@banuba/webar/background.zip?url"

// ...

const player = await Player.create({
  clientToken: "xxx-xxx-xxx",
  // point BanubaSDK where to find these vital files
  locateFile: {
    "BanubaSDK.data": data,
    "BanubaSDK.wasm": wasm,
    "BanubaSDK.simd.wasm": simd,
  },
})

await player.addModule(new Module(FaceTracker), new Module(Background))

// ...

See Vite Explicit URL imports docs for details.

Rollup​

import { Player, Module /* ... */ } from "@banuba/webar"
// you need to set-up @rollup/plugin-url for the import syntax to work
import data from "@banuba/webar/BanubaSDK.data"
import wasm from "@banuba/webar/BanubaSDK.wasm"
import simd from "@banuba/webar/BanubaSDK.simd.wasm"

import FaceTracker from "@banuba/webar/face_tracker.zip"
import Background from "@banuba/webar/background.zip"

// ...

const player = await Player.create({
  clientToken: "xxx-xxx-xxx",
  // point BanubaSDK where to find these vital files
  locateFile: {
    "BanubaSDK.data": data,
    "BanubaSDK.wasm": wasm,
    "BanubaSDK.simd.wasm": simd,
  },
})

await player.addModule(new Module(FaceTracker), new Module(Background))

// ...

See @rollup/plugin-url docs for details.

Webpack​

Depending on the version of Webpack used, you may have to add following rule to the module.rules section of the webpack.config.js:

module.exports = {
  module: {
    rules: [
      // ...
      {
        test: /\.wasm$/,
        type: 'javascript/auto',
        loader: 'file-loader',
      },
      // ...
    ],
    },
  },
}

Now import of .wasm files as URLs should work properly:

import { Player, Module /* ... */ } from "@banuba/webar"
import data from "@banuba/webar/BanubaSDK.data"
import wasm from "@banuba/webar/BanubaSDK.wasm"
import simd from "@banuba/webar/BanubaSDK.simd.wasm"

import FaceTracker from "@banuba/webar/face_tracker.zip"
import Background from "@banuba/webar/background.zip"

// ...

const player = await Player.create({
  clientToken: "xxx-xxx-xxx",
  // point BanubaSDK where to find these vital files
  locateFile: {
    "BanubaSDK.data": data,
    "BanubaSDK.wasm": wasm,
    "BanubaSDK.simd.wasm": simd,
  },
})

await player.addModule(new Module(FaceTracker), new Module(Background))

// ...

See the related Webpack issue for details.

React​

Connect Banuba WebAR SDK as any other library:

import React, { useEffect } from "react"
import data from "@banuba/webar/BanubaSDK.data"
import wasm from "@banuba/webar/BanubaSDK.wasm"
import simd from "@banuba/webar/BanubaSDK.simd.wasm"
import FaceTracker from "@banuba/webar/face_tracker.zip"
import Glasses from "/path/to/Glasses.zip"
import { Webcam, Player, Module, Effect, Dom } from "@banuba/webar"

export default function WebARComponent() {
  // componentDidMount
  useEffect(() => {
    let webcam

    Player.create({
      clientToken: "xxx-xxx-xxx",
      locateFile: {
        "BanubaSDK.data": data,
        "BanubaSDK.wasm": wasm,
        "BanubaSDK.simd.wasm": simd,
      },
    }).then(async (player) => {
      await player.addModule(new Module(FaceTracker))
      await player.use(webcam = new Webcam())
      player.applyEffect(new Effect(Glasses))
      Dom.render(player, "#webar")
    })

    // componentWillUnmount
    return () => {
      webcam?.stop()
      Dom.unmount("#webar")
    }
  })

  return <div id="webar"></div>
}

See Bundlers for notes about specific bundlers and locateFile usage.

See Banuba React demo app for more code examples.

Real Time Video communication​

WebRTC​

Considering the Fireship WebRTC demo:

import {
  MediaStream as BanubaMediaStream,
  Player,
  Module,
  Effect,
  MediaStreamCapture,
} from "https://cdn.jsdelivr.net/npm/@banuba/webar/dist/BanubaSDK.browser.esm.js"

// ...

webcamButton.onclick = async () => {
  localStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true })
  remoteStream = new MediaStream()

  const player = await Player.create({ clientToken: "xxx-xxx-xxx" })
  await player.addModule(new Module("https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/background.zip"))

  const webar = new MediaStreamCapture(player)

  await player.use(new BanubaMediaStream(localStream))
  player.applyEffect(new Effect("BackgroundBlur.zip"))
  player.play()

  // original audio
  const audio = localStream.getAudioTracks()[0]
  // webar processed video
  const video = webar.getVideoTracks()[0]

  localStream = new MediaStream([audio, video])

  // Push tracks from local stream to peer connection
  localStream.getTracks().forEach((track) => {
    pc.addTrack(track, localStream)
  })

  // ...
}

See Capturing processed video > MediaStream for details.

Agora​

Check the official Banuba Agora extension for Web - @banuba/agora-extension.

Or if you need a more fine-granted control, you may use the Banuba WebAR directly:

import "https://download.agora.io/sdk/web/AgoraRTC_N-4.1.0.js"
import { MediaStream, Player, Module, Effect, MediaStreamCapture } from "https://cdn.jsdelivr.net/npm/@banuba/webar"

// ...

const camera = await navigator.mediaDevices.getUserMedia({ audio: true, video: true })
const player = await Player.create({ clientToken: "xxx-xxx-xxx" })
await player.addModule(new Module("https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/background.zip"))

const webar = new MediaStreamCapture(player)

await player.use(new MediaStream(camera))
player.applyEffect(new Effect("BackgroundBlur.zip"))
player.play()

// original audio
const audio = await AgoraRTC.createMicrophoneAudioTrack()
// webar processed video
const video = await AgoraRTC.createCustomVideoTrack({ mediaStreamTrack: webar.getVideoTrack() })

const client = AgoraRTC.createClient({ mode: "live", role: "host", codec: "h264" })
await client.join("AGORA APP ID", "CHANNEL NAME", null)
await client.publish([audio, video])

// ...

See AgoraWebSDK NG API docs for details.

See Banuba Video call demo app for more code examples.

OpenTok (TokBox)​

import "https://cdn.jsdelivr.net/npm/@opentok/client"
import { MediaStream, Player, Module Effect, MediaStreamCapture } from "https://cdn.jsdelivr.net/npm/@banuba/webar/dist/BanubaSDK.browser.esm.js"

// ...

const camera = await navigator.mediaDevices.getUserMedia({ audio: true, video: true })
const player = await Player.create({ clientToken: "xxx-xxx-xxx" })
await player.addModule(new Module("https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/background.zip"))

const webar = new MediaStreamCapture(player)

await player.use(new MediaStream(camera))
player.applyEffect(new Effect("BackgroundBlur.zip"))
player.play()

// original audio
const audio = camera.getAudioTracks()[0]
// webar processed video
const video = webar.getVideoTracks()[0]

const session = OT.initSession("OT API KEY", "OT SESSION ID")
session.connect("OT SESSION TOKEN", async () => {
  const publisher = await OT.initPublisher(
    "publisher",
    {
      insertMode: "append",
      audioSource: audio,
      videoSource: video,
      width: "100%",
      height: "100%",
    },
    () => {},
  )

  session.publish(publisher, () => {})
})

// ...

See TokBox Video API docs for details.

See Banuba Video call (TokBox) demo app for more code examples.