Package com.banuba.sdk.effect_player

Interface EffectPlayer

All Known Implementing Classes:
EffectPlayer.CppProxy
public interface EffectPlayer
The EffectPlayer class provides ability to play AR effects on set of images or video.

Lifecycle

In order to support an application lifecycle EffectPlayer provides special methods to control it during application states like pause or losing focus.

Effect playback

Initial playback state of newly created Effect Player instance is `active`. With playback control methods described below, the player can be launched/resumed or paused. State of effect player becomes stopped/inactive at the moment of surface loss or by calling the EffectPlayer::playback_stop method. One can use next three methods to control effect playback:
- `EffectPlayer.playback_play`. Switching to active state attempt. Possible from paused or stopped state and has no effect if effect playback is already active. Playback resumes from the position saved before `EffectPlayer.playbackPause` call.
- `EffectPlayer.playback_pause`. Suspend current playback attempt. The recognizer thread is stopped and all the video textures and audio units playback is stopped as well. Effect player doesn't react on EffectPlayer::pushFrame calls in suspended state except asynchronous-inconsistent mode.
- `EffectPlayer.playbackStop`. Switch to inactive state. In addition to pause clears recognizer's buffer. The next switch to active state will result in total rerun of active effect which means that it will be started from the very beginning.

Android usage example

If you want an application to be active on windowed mode and if the focus is on another application you can switch EffectPlayer into active state when `onStart` activity callback occurs and switch to paused when `onStop` occurs. Otherwise you can suspend `EffectPlayer` on `onPause` and resume on `onResume` callback to make an instance of the `EffectPlayer` inactive while activity losing focus.

Context/surface control

There are two methods that have to be used to ensure correct operating of the EffectPlayer. Losing context without notifying the EffectPlayer leads to application crash. Methods for context changes notification:
- `EffectPlayer.surfaceCreated`
- `EffectPlayer.surfaceDestroyed`

Multi-Threading performance and safety. Listeners.

EffectPlayer allows to call some of its methods from several threads (simultaneously) for convenience and increased performance. This is allowed only during normal operation, all invocations from other threads MUST finish before you start destroying the EffectPlayer instance. Most methods belong to one of the two groups:
- MUST be called from the "Main" (also "Render") thread ("Not Reentrant, Not Thread-safe")
- MAY be called from "Any" thread. Most are "Thread-safe" Listener callbacks can be called back from any thread. Generally for best performance and responsiveness you'll have 3 threads:
- Main(render) thread for drawing-related operations
- "Camera" thread for handling and push()-ing frames into EffectPlayer
- UI Thread for handling user interactions and other tasks

  • Method Details

    • addFrameDurationListener

      void addFrameDurationListener​(@Nullable FrameDurationListener frameDurationListener)
      Add callback to receive FPS information. Thread-safe. May be called from any thread

    • removeFrameDurationListener

      void removeFrameDurationListener​(@Nullable FrameDurationListener frameDurationListener)
      Remove callback to receive FPS information. Thread-safe. May be called from any thread

    • addFaceNumberListener

      void addFaceNumberListener​(@Nullable FaceNumberListener faceNumberListener)
      Add callback to receive faces count in frame. Thread-safe. May be called from any thread

    • removeFaceNumberListener

      void removeFaceNumberListener​(@Nullable FaceNumberListener faceNumberListener)
      Remove callback to receive faces count in frame. Thread-safe. May be called from any thread

    • addFrameDataListener

      void addFrameDataListener​(@Nullable FrameDataListener frameDataListener)
      Add callback to receive frame data right after processing in recognizer. Thread-safe. May be called from any thread

    • removeFrameDataListener

      void removeFrameDataListener​(@Nullable FrameDataListener frameDataListener)
      Remove callback to receive frame data right after processing in recognizer. Thread-safe. May be called from any thread

    • addLowLightListener

      void addLowLightListener​(@Nullable LowLightListener lowLightListener)
      Add callback to receive low light info Thread-safe. May be called from any thread

    • removeLowLightListener

      void removeLowLightListener​(@Nullable LowLightListener lowLightListener)
      Remove callback to receive low light info Thread-safe. May be called from any thread

    • addCameraPoiListener

      void addCameraPoiListener​(@Nullable CameraPoiListener cameraPoiListener)
      Add callback to receive center of the face in frame relative to top-left corner in [0, 1] space. Thread-safe. May be called from any thread

    • removeCameraPoiListener

      void removeCameraPoiListener​(@Nullable CameraPoiListener cameraPoiListener)
      Remove callback to receive center of the face in frame relative to top-left Thread-safe. May be called from any thread

    • addEffectInfoListener

      void addEffectInfoListener​(@Nullable EffectInfoListener effectInfoListener)
      Add callback to receive current effect info from Effect Player. Thread-safe. May be called from any thread

    • removeEffectInfoListener

      void removeEffectInfoListener​(@Nullable EffectInfoListener effectInfoListener)
      Remove callback to receive current effect info from Effect Player. Thread-safe. May be called from any thread

    • addEffectActivationCompletionListener

      void addEffectActivationCompletionListener​(@Nullable EffectActivationCompletionListener effectActivationCompletionListener)
      Add callback to receive current effect activation notification from Effect Player. Thread-safe. May be called from any thread

    • removeEffectActivationCompletionListener

      void removeEffectActivationCompletionListener​(@Nullable EffectActivationCompletionListener effectActivationCompletionListener)
      Remove callback to receive current effect activation notification from Effect Player. Thread-safe. May be called from any thread

    • setMaxFaces

      void setMaxFaces​(int maxFaces)
      Sets maximum allowed face results, if face tracking feature is present. Thread-safe. May be called from any thread This option is relevant only for GLFX effects.

    • setExternalCameraTexture

      void setExternalCameraTexture​(int name, int width, int height)
      Set OpenGL name of input external texture. https://developer.android.com/reference/android/graphics/SurfaceTexture . MUST be called from the render thread
      Parameters:
      name - Texture name
      width - Textute width
      height - Texture height
      See Also:
      Note: This function is Android only. Note: We use texture size to calculate aspect ratio only.

    • setUseExtCamTex

      void setUseExtCamTex​(boolean value)
      Use external texture as input source. MUST be called from the render thread
      See Also:
      setExternalCameraTexture(int,int,int)

    • surfaceCreated

      void surfaceCreated​(int width, int height)
      Use to notify the EffectPlayer that the surface exists and effect can be played. 0, 0 for width and height are valid values. MUST be called from the render thread

    • surfaceChanged

      void surfaceChanged​(int width, int height)
      Notify about rendering surface being resized. MUST be called from the render thread

    • surfaceDestroyed

      void surfaceDestroyed()
      This method should be called right before an active context will become invalid. Switches playback state to inactive state. If it's not done an application will be crashed on next draw iteration. After losing the surface effect playback can't be resumed from last position. MUST be called from the render thread

    • draw

      long draw()
      Draw the current effect into the current OpenGL framebuffer. Uses internal frame_data object obtained from latest push_frame recognition result. Return current frame number if drawing was performed and caller should swap buffers otherwise `DRAW_SKIPPED`(-1) MUST be called from the render thread

    • drawWithExternalFrameData

      long drawWithExternalFrameData​(@Nullable FrameData frameData)
      Draw the current effect into the current OpenGL framebuffer. Uses externally provided frameData object instead of internal one obtained from latest push_frame recognition result. Return frame number from provided frameData if drawing was performed and caller should swap buffers otherwise `DRAW_SKIPPED`(-1) MUST be called from the render thread

    • captureBlit

      void captureBlit​(int captureWidth, int captureHeight)
      Record last rendering ("draw_()") result into current OpenGL framebuffer. Content is cropped to maintain effect_size() aspect ratio MUST be called from the render thread

    • readPixels

      @NonNull Data readPixels​(int width, int height)
      Receive the last rendered frame in binary form.
      Parameters:
      width - width of the picture to take, must match the values passed to `setEffectSize` or `create`
      height - height of the picture to take, must match the values passed to `setEffectSize` or `create`
      Returns:
      RGBA data, size is `width * height * 4` MUST be called from the render thread

    • setRenderConsistencyMode

      void setRenderConsistencyMode​(@NonNull ConsistencyMode value)
      Thread-safe. May be called from any thread

    • setRenderTransform

      void setRenderTransform​(@NonNull PixelRect imageRect, @NonNull PixelRect viewportRect, boolean xFlip, boolean yFlip)
      Request display of sub-area of the input image into sub-area of the output surface, with optional x,y flips imageRect is fitted inside viewportRect Resets transform to default if either rect has 0 dimensions
      Parameters:
      imageRect - rectangle in input image coordinates(pixels) after applying input rotations and flips
      viewportRect - rectangle in output surface coordinates(pixels) MUST be called from the render thread

    • processImage

      @NonNull Data processImage​(@NonNull FullImageData inputImage, @NonNull PixelFormat outputPixelFormat, @NonNull ProcessImageParams params)
      Process an image with current effect. Must be called from the render thread.
      Parameters:
      inputImage - to avoid conversion recommended to use YUV image
      outputPixelFormat - to avoid conversion recommended to use RGBA
      params - extra image processing arguments

    • processImageFrameData

      @NonNull Data processImageFrameData​(@Nullable FrameData inputFrameData, @NonNull PixelFormat outputPixelFormat, @NonNull ProcessImageParams params)
      Process an image with current effect. Prefer this method over `processImage` when you have extra input data besides just an input image. Must be called from render thread.
      Parameters:
      inputFrameData - `FrameData` with an image
      outputPixelFormat - to avoid conversion recommended to use RGBA
      params - extra image processing arguments

    • processImageData

      @NonNull byte[] processImageData​(@NonNull byte[] inputImage, int width, int height, @NonNull CameraOrientation orientation, boolean isMirrored, @NonNull PixelFormat inputPixelFormat, @NonNull PixelFormat outputPixelFormat, @NonNull ProcessImageParams params)
      Process image with current effect. Must be called from render thread. NOTE: inputImage have copy overhead, preferable to use process_image method MUST be called from the render thread
      Parameters:
      inputImage - input image with `width * height * 4` size
      outputPixelFormat - to avoid conversion recommended to use RGBA

    • pushFrame

      void pushFrame​(@NonNull FullImageData fullImage)
      Provides image to process and to play effect. Thread-safe. May be called from any thread

    • pushFrameWithNumber

      void pushFrameWithNumber​(@NonNull FullImageData fullImage, long frameNumber)
      Provides image to process and to play effect. Thread-safe. May be called from any thread

    • pushFrameData

      void pushFrameData​(@Nullable FrameData frameData)
      Provides FrameData to process and to play effect. Must contain full image. Thread-safe. May be called from any thread

    • playbackPlay

      void playbackPlay()
      MUST be called from the main(render) thread

    • playbackPause

      void playbackPause()

    • playbackStop

      void playbackStop()

    • getPlaybackState

      @NonNull EffectPlayerPlaybackState getPlaybackState()
      Thread-safe. May be called from any thread

    • getCurrentScene

      @Nullable Scene getCurrentScene()
      If currently loaded effect is based on new Scene engine and it is loaded completly returns Scene object from the effect. Otherwise returns null. MUST be called from the render thread Returned Scene object MUST be used only from the render thread and only while the effect is in the loaded state.

    • setEffectVolume

      void setEffectVolume​(float volume)
      Set effect audio volume. Thread-safe. May be called from any thread
      Parameters:
      volume - A value in range `[0, 1]`, where `1` means maximum volume.

    • enableAudio

      void enableAudio​(boolean enable)
      Set audio enabled

    • getInputManager

      @Nullable InputManager getInputManager()
      Get interface to control user iterations. This events will be passed to effect. Thread-safe. May be called from any thread

    • startVideoProcessing

      void startVideoProcessing​(long screenWidth, long screenHeight, @NonNull CameraOrientation orientation, boolean resetEffect, boolean offlineMode)
      Initialize video processing. To provide frames use *process_video_frame* methods. Effect audio is recorded as well and can be accessed using *process_recorded_audio* and *write_recorded_audio*. *push_frame* should not be called during processing. MUST be called from the render thread

    • stopVideoProcessing

      void stopVideoProcessing​(boolean resetEffect)
      Finish processing and return renderer to normal state. MUST be called from the render thread

    • processVideoFrameData

      @Nullable FrameData processVideoFrameData​(@Nullable FrameData inputFrameData, @NonNull ProcessImageParams params, @Nullable java.lang.Integer recognizerIterations)
      Provide frame data to evaluate video processing. MUST be called from the render thread
      Parameters:
      inputFrameData - Frame Data to process.
      params - Processing params.
      recognizerIterations - Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.
      Returns:
      Frame data with frame processing results.

    • processVideoFrame

      @Nullable FrameData processVideoFrame​(@NonNull FullImageData inputImage, @NonNull ProcessImageParams params, @Nullable java.lang.Integer recognizerIterations)
      Provide frame to evaluate video processing. MUST be called from the render thread
      Parameters:
      inputImage - Image to process.
      params - Processing params.
      recognizerIterations - Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.
      Returns:
      Frame data with frame processing results.

    • drawVideoFrame

      @NonNull Data drawVideoFrame​(@Nullable FrameData inputFrameData, long timeNs, @NonNull PixelFormat outputPixelFormat)
      Draw processed frame to image buffer. MUST be called from the render thread
      Parameters:
      inputFrameData - FrameData to draw.
      timeNs - Frame position on timeline.
      outputPixelFormat - Output image format.
      Returns:
      Buffer with processed image in selected format.

    • drawVideoFrameAllocated

      @NonNull Data drawVideoFrameAllocated​(@Nullable FrameData inputFrameData, long timeNs, @NonNull PixelFormat outputPixelFormat, @NonNull Data allocatedBuffer)
      The same as *draw_video_frame* but accepts preallocated buffer to avoid recreation for every frame. MUST be called from the render thread

    • isVoiceChangerConfigured

      boolean isVoiceChangerConfigured()
      Thread-safe. May be called from any thread

    • processRecordedAudio

      void processRecordedAudio​(@NonNull java.lang.String inFilename, @NonNull java.lang.String outFilename, float mixVolume)
      Process recorded voice with voice changer effects mixing with recorded sounds. Copies in->out if no voice changer or effect is active. Not thread-safe but can be called from any thread. Recorded sounds have the same lifetime as the effect(e.g reset after load_effect, etc.).
      Parameters:
      inFilename - Input filename.
      outFilename - Must have ".wav" extension.
      mixVolume - Relative volume for mixing in sounds, [0..1].

    • writeRecordedAudio

      void writeRecordedAudio​(@NonNull java.lang.String filename, long lengthMs)
      Save recorded audio. Not thread-safe but can be called from any thread.
      Parameters:
      filename - ".wav" extension must be used.
      lengthMs - Track length. Set to '0' to use full duration.

    • onVideoRecordStart

      void onVideoRecordStart​(boolean playAudioWhileRecording)
      Trigger record start events in EP and Effect. Records all sounds during recording to be replayed by `processRecordedAudio`. If @param playAudioWhileRecording is true, the audio will continue to play. Otherwise, the audio will be muted. Thread-safe. May be called from any thread

    • onVideoRecordEnd

      void onVideoRecordEnd()
      Triggers record stop events in EP and Effect. Thread-safe. May be called from any thread

    • isDeviceNnCompatible

      boolean isDeviceNnCompatible()
      Check is device compatible with Neural Networks player Thread-safe. May be called from any thread On some platforms (e.g. Android) may require the first invocation to be on the render thread

    • startFramedataCapture

      void startFramedataCapture​(@NonNull java.lang.String folder, @Nullable java.lang.String filename)
      Request to start framedata capture process. Output file is in CBOR format. Thread-safe. May be called from any thread
      Parameters:
      folder - Output folder.
      filename - Output filename. If the value is an empty string, filename is generated based on date and time.

    • stopFramedataCapture

      void stopFramedataCapture()
      See Also:
      startFramedataCapture(java.lang.String,java.lang.String)

    • startAnalyticsCapture

      void startAnalyticsCapture​(@NonNull java.util.HashMap<java.lang.String,​java.lang.String> deviceInfo, @NonNull AnalyticsConfig config, @Nullable AnalyticsListener listener)
      Request to start analytics capture process. Analytics supports two business cases: 1. Capture high resolution photo with face (bigger than 720p). 2. Capture N frames with face (N is configurable, if N=0 capturer will skip this case). Result is a path to CBOR file with images and face recognition results, compressed with zlib. Thread-safe. May be called from any thread
      Parameters:
      deviceInfo - Any user specific information to attach in analytics report.
      config - Setup configurable properties (like frames counter and delay before capture).
      listener - Callback which is fired in one of two above mentioned cases.

    • stopAnalyticsCapture

      void stopAnalyticsCapture()

    • effectManager

      @Nullable EffectManager effectManager()
      Get effect manager object Thread-safe. May be called from any thread

    • debugInterface

      @Nullable DebugInterface debugInterface()
      For internal usage only

    • setRecognizerOfflineMode

      void setRecognizerOfflineMode​(boolean on)
      Force recognizer offline mode Thread-safe. May be called from any thread

    • setFaceSearchMode

      void setFaceSearchMode​(@NonNull FaceSearchMode faceSearch)
      Change face search mode Thread-safe. May be called from any thread

    • create

      @Nullable static EffectPlayer create​(@NonNull EffectPlayerConfiguration configuration)

    • versionMajor

      static int versionMajor()
      Get major version of EffectPlayer. Use this method to filter out breaking changes in implementation of this class.
      See Also:
      EffectPlayer.versionMinor

    • versionMinor

      static int versionMinor()
      Get minor version of EffectPlayer
      See Also:
      EffectPlayer.versionMajor