EffectPlayer module is the main interface for all low-level functionality of Banuba SDK.
As described in the Introduction page,
EffectPlayer is responsible
for consuming camera frames, running various recognition pipelines on it, sending the recognized data
to the effect playback engine, and drawing the final image on the provided OpenGL surface.
For most use cases of Banuba SDK, it's better to use platform-specific APIs and language bindings as described in the Platforms section. The
EffectPlayerAPI should be used only for non-trivial cases not covered by Platform APIs.
To work with the
EffectPlayer application, you should create an appropriate execution environment. The two most
essential components of this environment are the camera and render surface. The camera pushes the image frames to
The render surface creates the OpenGL context with target framebuffer where the final image is rendered.
The camera or any other image source should periodically push the frames into EffectPlayer. Usually, it's done in the dedicated thread called Camera thread.
EffectPlayer periodically runs pipeline with recognition technologies to retrieve data from an input image. The component with the pipeline is called
Recognizer, and it runs its work cycle in a dedicated thread incapsulated in
Before recognition, the input images from the camera are placed into
InputBuffer where they are waiting for the next recognition cycle. During processing a lot of information is attached to the image, e.g. face positions, segmentation masks from neural networks etc. In the end of the recognition process, these data compose the final result of the recognition which is stored in a separate data structure called
FrameData is placed in
OutputBuffer from where it is sent to an Effect and drawn to the render surface.
During the drawing cycle
FrameData is popped from
OutputBuffer and consumed by the Effect engine. On this stage, the Effect engine executes all effect's logic, e.g. sends all recognized data to the script engine or to the renderer.
Drawing cycles are usually initiated from an app or platform modules. The platform module creates a dedicated Render thread to manage this task.
This module ensures that the OpenGL context and target framebuffer exist during the drawing calls.
As descibed in the Overview section,
EffectPlayer expects its methods to be called from the three treads: the UI thread, render thread or camera thread. See the reference of EffectPlayer for more details about each method.
Note It's important to call each method of
EffectPlayeronly from its dediceted thread. Calling methods from the wrong threads can cause unexpected app crashes and halts.
EffectPlayer supports different operational modes:
- offline high-quality processing of an individual images or video frames
- online processing and visualization of real-time camera input
Realtime camera preview
The Camera and Render thread work continuously, and the user sees the processed image on the screen in real time. It's the most commonly used operational mode.
To enable this mode, call the following methods:
- in a loop:
The user can apply effects to the static photo uploaded from the gallery and see dynamic changes in real time.
- in a loop:
The video is processed synchronously frame by frame with results returned as a series of images in system memory.
- In a loop:
Note Realtime processing methods can't be used during video processing (
push_frame, etc.). Please, use the methods descibed above.
One photo is processed synchronously with results returned as an image in system memory.
- In a loop (if processing of multiple images is needed):
Note The state of face tracking isn't saved between
process_imagemethod is slow but accurate and not intended for realtime usage.