Package com.banuba.sdk.effect_player

Interface EffectPlayer

All Known Implementing Classes:
EffectPlayer.CppProxy
public interface EffectPlayer
The EffectPlayer class provides ability to play AR effects on set of images or video.

Lifecycle

In order to support an application lifecycle EffectPlayer provides special methods to control it during application states like pause or losing focus.

Effect playback

Initial playback state of newly created Effect Player instance is `active`. With playback control methods described below, the player can be launched/resumed or paused. State of effect player becomes stopped/inactive at the moment of surface loss or by calling the EffectPlayer::playback_stop method. One can use next three methods to control effect playback:
- `EffectPlayer.playback_play`. Switching to active state attempt. Possible from paused or stopped state and has no effect if effect playback is already active. Playback resumes from the position saved before `EffectPlayer.playbackPause` call.
- `EffectPlayer.playback_pause`. Suspend current playback attempt. The recognizer thread is stopped and all the video textures and audio units playback is stopped as well. Effect player doesn't react on EffectPlayer::pushFrame calls in suspended state except asynchronous-inconsistent mode.
- `EffectPlayer.playbackStop`. Switch to inactive state. In addition to pause clears recognizer's buffer. The next switch to active state will result in total rerun of active effect which means that it will be started from the very beginning.

Android usage example

If you want an application to be active on windowed mode and if the focus is on another application you can switch EffectPlayer into active state when `onStart` activity callback occurs and switch to paused when `onStop` occurs. Otherwise you can suspend `EffectPlayer` on `onPause` and resume on `onResume` callback to make an instance of the `EffectPlayer` inactive while activity losing focus.

Context/surface control

There are two methods that have to be used to ensure correct operating of the EffectPlayer. Losing context without notifying the EffectPlayer leads to application crash. Methods for context changes notification:
- `EffectPlayer.surfaceCreated`
- `EffectPlayer.surfaceDestroyed`

Multi-Threading performance and safety. Listeners.

EffectPlayer allows to call some of its methods from several threads (simultaneously) for convenience and increased performance. This is allowed only during normal operation, all invocations from other threads MUST finish before you start destroying the EffectPlayer instance. Most methods belong to one of the two groups:
- MUST be called from the "Main" (also "Render") thread ("Not Reentrant, Not Thread-safe")
- MAY be called from "Any" thread. Most are "Thread-safe" Listener callbacks can be called back from any thread. Generally for best performance and responsiveness you'll have 3 threads:
- Main(render) thread for drawing-related operations
- "Camera" thread for handling and push()-ing frames into EffectPlayer
- UI Thread for handling user interactions and other tasks

  • Method Details

    • addFrameDurationListener

      void addFrameDurationListener(@Nullable FrameDurationListener frameDurationListener)
      Add callback to receive FPS information. Thread-safe. May be called from any thread

    • removeFrameDurationListener

      void removeFrameDurationListener(@Nullable FrameDurationListener frameDurationListener)
      Remove callback to receive FPS information. Thread-safe. May be called from any thread

    • addFaceNumberListener

      void addFaceNumberListener(@Nullable FaceNumberListener faceNumberListener)
      Add callback to receive faces count in frame. Thread-safe. May be called from any thread

    • removeFaceNumberListener

      void removeFaceNumberListener(@Nullable FaceNumberListener faceNumberListener)
      Remove callback to receive faces count in frame. Thread-safe. May be called from any thread

    • addFrameDataListener

      void addFrameDataListener(@Nullable FrameDataListener frameDataListener)
      Add callback to receive frame data right after processing in recognizer. Thread-safe. May be called from any thread

    • removeFrameDataListener

      void removeFrameDataListener(@Nullable FrameDataListener frameDataListener)
      Remove callback to receive frame data right after processing in recognizer. Thread-safe. May be called from any thread

    • addCameraPoiListener

      void addCameraPoiListener(@Nullable CameraPoiListener cameraPoiListener)
      Add callback to receive center of the face in frame relative to top-left corner in [0, 1] space. Thread-safe. May be called from any thread

    • removeCameraPoiListener

      void removeCameraPoiListener(@Nullable CameraPoiListener cameraPoiListener)
      Remove callback to receive center of the face in frame relative to top-left Thread-safe. May be called from any thread

    • addEffectInfoListener

      void addEffectInfoListener(@Nullable EffectInfoListener effectInfoListener)
      Add callback to receive current effect info from Effect Player. Thread-safe. May be called from any thread

    • removeEffectInfoListener

      void removeEffectInfoListener(@Nullable EffectInfoListener effectInfoListener)
      Remove callback to receive current effect info from Effect Player. Thread-safe. May be called from any thread

    • addEffectActivationCompletionListener

      void addEffectActivationCompletionListener(@Nullable EffectActivationCompletionListener effectActivationCompletionListener)
      Add callback to receive current effect activation notification from Effect Player. Thread-safe. May be called from any thread

    • removeEffectActivationCompletionListener

      void removeEffectActivationCompletionListener(@Nullable EffectActivationCompletionListener effectActivationCompletionListener)
      Remove callback to receive current effect activation notification from Effect Player. Thread-safe. May be called from any thread

    • setMaxFaces

      void setMaxFaces(int maxFaces)
      Sets maximum allowed face results, if face tracking feature is present. Thread-safe. May be called from any thread This option is relevant only for GLFX effects.

    • setExternalCameraTexture

      void setExternalCameraTexture(int name, int width, int height)
      Set OpenGL name of input external texture. https://developer.android.com/reference/android/graphics/SurfaceTexture . MUST be called from the render thread
      Parameters:
      name - Texture name
      width - Textute width
      height - Texture height
      See Also:

    • setUseExtCamTex

      void setUseExtCamTex(boolean value)
      Use external texture as input source. MUST be called from the render thread
      See Also:

    • surfaceCreated

      void surfaceCreated(int width, int height)
      Use to notify the EffectPlayer that the surface exists and effect can be played. 0, 0 for width and height are valid values. MUST be called from the render thread

    • surfaceChanged

      void surfaceChanged(int width, int height)
      Notify about rendering surface being resized. MUST be called from the render thread

    • surfaceDestroyed

      void surfaceDestroyed()
      This method should be called right before an active context will become invalid. Switches playback state to inactive state. If it's not done an application will be crashed on next draw iteration. After losing the surface effect playback can't be resumed from last position. MUST be called from the render thread

    • draw

      long draw()
      Draw the current effect into the current OpenGL framebuffer. Uses internal frame_data object obtained from latest push_frame recognition result. Return current frame number if drawing was performed and caller should swap buffers otherwise `DRAW_SKIPPED`(-1) MUST be called from the render thread

    • drawWithExternalFrameData

      long drawWithExternalFrameData(@Nullable FrameData frameData)
      Draw the current effect into the current OpenGL framebuffer. Uses externally provided frameData object instead of internal one obtained from latest push_frame recognition result. Return frame number from provided frameData if drawing was performed and caller should swap buffers otherwise `DRAW_SKIPPED`(-1) MUST be called from the render thread

    • captureBlit

      void captureBlit(int captureWidth, int captureHeight)
      Record last rendering ("draw_()") result into current OpenGL framebuffer. Content is cropped to maintain effect_size() aspect ratio MUST be called from the render thread

    • readPixels

      @NonNull PixelBuffer readPixels()
      Receive the last rendered frame in binary form.
      Returns:
      4-byte per pixel data, size is `fx_width * fx_height * 4` MUST be called from the render thread

    • setRenderConsistencyMode

      void setRenderConsistencyMode(@NonNull ConsistencyMode value)
      Thread-safe. May be called from any thread

    • processImage

      @NonNull Data processImage(@NonNull FullImageData inputImage, @NonNull PixelFormat outputPixelFormat)
      Process an image with current effect. Must be called from the render thread.
      Parameters:
      inputImage - to avoid conversion recommended to use YUV image
      outputPixelFormat - to avoid conversion recommended to use RGBA
      params - extra image processing arguments

    • processImageFrameData

      @NonNull Data processImageFrameData(@Nullable FrameData inputFrameData, @NonNull PixelFormat outputPixelFormat)
      Process an image with current effect. Prefer this method over `processImage` when you have extra input data besides just an input image. Must be called from render thread.
      Parameters:
      inputFrameData - `FrameData` with an image
      outputPixelFormat - to avoid conversion recommended to use RGBA
      params - extra image processing arguments

    • processImageData

      @NonNull byte[] processImageData(@NonNull byte[] inputImage, int width, int height, @NonNull CameraOrientation orientation, boolean isMirrored, @NonNull PixelFormat inputPixelFormat, @NonNull PixelFormat outputPixelFormat)
      Process image with current effect. Must be called from render thread. NOTE: inputImage have copy overhead, preferable to use process_image method MUST be called from the render thread
      Parameters:
      inputImage - input image with `width * height * 4` size
      outputPixelFormat - to avoid conversion recommended to use RGBA

    • pushFrame

      void pushFrame(@NonNull FullImageData fullImage)
      Provides image to process and to play effect. Thread-safe. May be called from any thread

    • pushFrameWithNumber

      void pushFrameWithNumber(@NonNull FullImageData fullImage, long frameNumber)
      Provides image to process and to play effect. Thread-safe. May be called from any thread

    • pushFrameData

      void pushFrameData(@Nullable FrameData frameData)
      Provides FrameData to process and to play effect. Must contain full image. Thread-safe. May be called from any thread

    • pushFrameDataWithNumber

      void pushFrameDataWithNumber(@Nullable FrameData frameData, long frameNumber)
      Provides FrameData with frame number to process and to play effect. Must contain full image. Thread-safe. May be called from any thread

    • playbackPlay

      void playbackPlay()
      MUST be called from the main(render) thread

    • playbackPause

      void playbackPause()

    • playbackStop

      void playbackStop()

    • getPlaybackState

      @NonNull EffectPlayerPlaybackState getPlaybackState()
      Thread-safe. May be called from any thread

    • getInputManager

      @Nullable InputManager getInputManager()
      Get interface to control user iterations. This events will be passed to effect. Thread-safe. May be called from any thread

    • startVideoProcessing

      void startVideoProcessing(long screenWidth, long screenHeight, @NonNull CameraOrientation orientation, boolean resetEffect, boolean offlineMode)
      Initialize video processing. To provide frames use *process_video_frame* methods. Effect audio is recorded as well and can be accessed using *process_recorded_audio* and *write_recorded_audio*. *push_frame* should not be called during processing. MUST be called from the render thread

    • stopVideoProcessing

      void stopVideoProcessing(boolean resetEffect)
      Finish processing and return renderer to normal state. MUST be called from the render thread

    • processVideoFrameData

      @Nullable FrameData processVideoFrameData(@Nullable FrameData inputFrameData, @Nullable Integer recognizerIterations)
      Provide frame data to evaluate video processing. MUST be called from the render thread
      Parameters:
      inputFrameData - Frame Data to process.
      params - Processing params.
      recognizerIterations - Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.
      Returns:
      Frame data with frame processing results.

    • processVideoFrame

      @Nullable FrameData processVideoFrame(@NonNull FullImageData inputImage, @Nullable Integer recognizerIterations)
      Provide frame to evaluate video processing. MUST be called from the render thread
      Parameters:
      inputImage - Image to process.
      params - Processing params.
      recognizerIterations - Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.
      Returns:
      Frame data with frame processing results.

    • drawVideoFrame

      @NonNull Data drawVideoFrame(@Nullable FrameData inputFrameData, long timeNs, @NonNull PixelFormat outputPixelFormat)
      Draw processed frame to image buffer. MUST be called from the render thread
      Parameters:
      inputFrameData - FrameData to draw.
      timeNs - Frame position on timeline.
      outputPixelFormat - Output image format.
      Returns:
      Buffer with processed image in selected format.

    • drawVideoFrameAllocated

      @NonNull Data drawVideoFrameAllocated(@Nullable FrameData inputFrameData, long timeNs, @NonNull PixelFormat outputPixelFormat, @NonNull Data allocatedBuffer)
      The same as *draw_video_frame* but accepts preallocated buffer to avoid recreation for every frame. MUST be called from the render thread

    • writeRecordedAudio

      void writeRecordedAudio(@NonNull String filename, long lengthMs)
      Save recorded audio. Not thread-safe but can be called from any thread.
      Parameters:
      filename - ".wav" extension must be used.
      lengthMs - Track length. Set to '0' to use full duration.

    • onVideoRecordStart

      void onVideoRecordStart(boolean playAudioWhileRecording)
      Trigger record start events in EP and Effect. Records all sounds during recording to be replayed by `processRecordedAudio`. If @param playAudioWhileRecording is true, the audio will continue to play. Otherwise, the audio will be muted. Thread-safe. May be called from any thread

    • onVideoRecordEnd

      void onVideoRecordEnd()
      Triggers record stop events in EP and Effect. Thread-safe. May be called from any thread

    • effectManager

      @Nullable EffectManager effectManager()
      Get effect manager object Thread-safe. May be called from any thread

    • setRecognizerOfflineMode

      void setRecognizerOfflineMode(boolean on)
      Force recognizer offline mode Thread-safe. May be called from any thread

    • setRecognizerUseFutureFilter

      void setRecognizerUseFutureFilter(boolean on)
      Set future frame filtrarion mode. Produce smoother recognition result (anti jitter), however adds inconsistency in push'ed/draw'ed frames (one frame lag) Example: push frame 1 - draw frame 1, push frame 2 - draw frame 1, push frame 3 - draw frame 2, ...

    • setRecognizerUseFutureInterpolate

      void setRecognizerUseFutureInterpolate(boolean on)
      Set future frame interpolation mode. Produce faster recognition result (skip even frames), however adds inconsistency in push'ed/pop'ed frames (one frame lag) Example: push frame 1 - pop frame 1, push frame 2 - pop frame 1, push frame 3 - pop frame 2, ...

    • setFrameProcessor

      void setFrameProcessor(@Nullable FrameProcessor processor)
      Set frame processor as current Thread-safe. May be called from any thread

    • frameProcessor

      @Nullable FrameProcessor frameProcessor()
      Get current frame processor Thread-safe. May be called from any thread

    • create

      @Nullable static EffectPlayer create(@Nullable EffectPlayerConfiguration configuration)

    • setRenderBackend

      static void setRenderBackend(@NonNull RenderBackendType backendType)

    • getCurrentRenderBackendType

      @NonNull static RenderBackendType getCurrentRenderBackendType()