Package com.banuba.sdk.effect_player

Interface EffectPlayer

  • All Known Implementing Classes:
    EffectPlayer.CppProxy
    public interface EffectPlayer
    The EffectPlayer class provides ability to play AR effects on set of images or video.

    Lifecycle

    In order to support an application lifecycle EffectPlayer provides special methods to control it during application states like pause or losing focus.

    Effect playback

    Initial playback state of newly created Effect Player instance is `active`. With playback control methods described below, the player can be launched/resumed or paused. State of effect player becomes stopped/inactive at the moment of surface loss or by calling the EffectPlayer::playback_stop method. One can use next three methods to control effect playback:
    - `EffectPlayer.playback_play`. Switching to active state attempt. Possible from paused or stopped state and has no effect if effect playback is already active. Playback resumes from the position saved before `EffectPlayer.playbackPause` call.
    - `EffectPlayer.playback_pause`. Suspend current playback attempt. The recognizer thread is stopped and all the video textures and audio units playback is stopped as well. Effect player doesn't react on EffectPlayer::pushFrame calls in suspended state except asynchronous-inconsistent mode.
    - `EffectPlayer.playbackStop`. Switch to inactive state. In addition to pause clears recognizer's buffer. The next switch to active state will result in total rerun of active effect which means that it will be started from the very beginning.

    Android usage example

    If you want an application to be active on windowed mode and if the focus is on another application you can switch EffectPlayer into active state when `onStart` activity callback occurs and switch to paused when `onStop` occurs. Otherwise you can suspend `EffectPlayer` on `onPause` and resume on `onResume` callback to make an instance of the `EffectPlayer` inactive while activity losing focus.

    Context/surface control

    There are two methods that have to be used to ensure correct operating of the EffectPlayer. Losing context without notifying the EffectPlayer leads to application crash. Methods for context changes notification:
    - `EffectPlayer.surfaceCreated`
    - `EffectPlayer.surfaceDestroyed`

    Multi-Threading performance and safety. Listeners.

    EffectPlayer allows to call some of its methods from several threads (simultaneously) for convenience and increased performance. This is allowed only during normal operation, all invocations from other threads MUST finish before you start destroying the EffectPlayer instance. Most methods belong to one of the two groups:
    - MUST be called from the "Main" (also "Render") thread ("Not Reentrant, Not Thread-safe")
    - MAY be called from "Any" thread. Most are "Thread-safe" Listener callbacks can be called back from any thread. Generally for best performance and responsiveness you'll have 3 threads:
    - Main(render) thread for drawing-related operations
    - "Camera" thread for handling and push()-ing frames into EffectPlayer
    - UI Thread for handling user interactions and other tasks

    • Method Detail

      • addFrameDurationListener

        void addFrameDurationListener​(@Nullable
                              FrameDurationListener frameDurationListener)
        Add callback to receive FPS information. Thread-safe. May be called from any thread

      • removeFrameDurationListener

        void removeFrameDurationListener​(@Nullable
                                 FrameDurationListener frameDurationListener)
        Remove callback to receive FPS information. Thread-safe. May be called from any thread

      • addFaceNumberListener

        void addFaceNumberListener​(@Nullable
                           FaceNumberListener faceNumberListener)
        Add callback to receive faces count in frame. Thread-safe. May be called from any thread

      • removeFaceNumberListener

        void removeFaceNumberListener​(@Nullable
                              FaceNumberListener faceNumberListener)
        Remove callback to receive faces count in frame. Thread-safe. May be called from any thread

      • addFrameDataListener

        void addFrameDataListener​(@Nullable
                          FrameDataListener frameDataListener)
        Add callback to receive frame data right after processing in recognizer. Thread-safe. May be called from any thread

      • removeFrameDataListener

        void removeFrameDataListener​(@Nullable
                             FrameDataListener frameDataListener)
        Remove callback to receive frame data right after processing in recognizer. Thread-safe. May be called from any thread

      • addCameraPoiListener

        void addCameraPoiListener​(@Nullable
                          CameraPoiListener cameraPoiListener)
        Add callback to receive center of the face in frame relative to top-left corner in [0, 1] space. Thread-safe. May be called from any thread

      • removeCameraPoiListener

        void removeCameraPoiListener​(@Nullable
                             CameraPoiListener cameraPoiListener)
        Remove callback to receive center of the face in frame relative to top-left Thread-safe. May be called from any thread

      • addEffectInfoListener

        void addEffectInfoListener​(@Nullable
                           EffectInfoListener effectInfoListener)
        Add callback to receive current effect info from Effect Player. Thread-safe. May be called from any thread

      • removeEffectInfoListener

        void removeEffectInfoListener​(@Nullable
                              EffectInfoListener effectInfoListener)
        Remove callback to receive current effect info from Effect Player. Thread-safe. May be called from any thread

      • addEffectActivationCompletionListener

        void addEffectActivationCompletionListener​(@Nullable
                                           EffectActivationCompletionListener effectActivationCompletionListener)
        Add callback to receive current effect activation notification from Effect Player. Thread-safe. May be called from any thread

      • removeEffectActivationCompletionListener

        void removeEffectActivationCompletionListener​(@Nullable
                                              EffectActivationCompletionListener effectActivationCompletionListener)
        Remove callback to receive current effect activation notification from Effect Player. Thread-safe. May be called from any thread

      • setMaxFaces

        void setMaxFaces​(int maxFaces)
        Sets maximum allowed face results, if face tracking feature is present. Thread-safe. May be called from any thread This option is relevant only for GLFX effects.

      • surfaceCreated

        void surfaceCreated​(int width,
                    int height)
        Use to notify the EffectPlayer that the surface exists and effect can be played. 0, 0 for width and height are valid values. MUST be called from the render thread

      • surfaceChanged

        void surfaceChanged​(int width,
                    int height)
        Notify about rendering surface being resized. MUST be called from the render thread

      • surfaceDestroyed

        void surfaceDestroyed()
        This method should be called right before an active context will become invalid. Switches playback state to inactive state. If it's not done an application will be crashed on next draw iteration. After losing the surface effect playback can't be resumed from last position. MUST be called from the render thread

      • draw

        long draw()
        Draw the current effect into the current OpenGL framebuffer. Uses internal frame_data object obtained from latest push_frame recognition result. Return current frame number if drawing was performed and caller should swap buffers otherwise `DRAW_SKIPPED`(-1) MUST be called from the render thread

      • drawWithExternalFrameData

        long drawWithExternalFrameData​(@Nullable
                               FrameData frameData)
        Draw the current effect into the current OpenGL framebuffer. Uses externally provided frameData object instead of internal one obtained from latest push_frame recognition result. Return frame number from provided frameData if drawing was performed and caller should swap buffers otherwise `DRAW_SKIPPED`(-1) MUST be called from the render thread

      • captureBlit

        void captureBlit​(int captureWidth,
                 int captureHeight)
        Record last rendering ("draw_()") result into current OpenGL framebuffer. Content is cropped to maintain effect_size() aspect ratio MUST be called from the render thread

      • readPixels

        @NonNull
PixelBuffer readPixels()
        Receive the last rendered frame in binary form.
        Returns:
        4-byte per pixel data, size is `fx_width * fx_height * 4` MUST be called from the render thread

      • setRenderConsistencyMode

        void setRenderConsistencyMode​(@NonNull
                              ConsistencyMode value)
        Thread-safe. May be called from any thread

      • processImage

        @NonNull
Data processImage​(@NonNull
                  FullImageData inputImage,
                  @NonNull
                  PixelFormat outputPixelFormat)
        Process an image with current effect. Must be called from the render thread.
        Parameters:
        inputImage - to avoid conversion recommended to use YUV image
        outputPixelFormat - to avoid conversion recommended to use RGBA
        params - extra image processing arguments

      • processImageFrameData

        @NonNull
Data processImageFrameData​(@Nullable
                           FrameData inputFrameData,
                           @NonNull
                           PixelFormat outputPixelFormat)
        Process an image with current effect. Prefer this method over `processImage` when you have extra input data besides just an input image. Must be called from render thread.
        Parameters:
        inputFrameData - `FrameData` with an image
        outputPixelFormat - to avoid conversion recommended to use RGBA
        params - extra image processing arguments

      • processImageData

        @NonNull
byte[] processImageData​(@NonNull
                        byte[] inputImage,
                        int width,
                        int height,
                        @NonNull
                        CameraOrientation orientation,
                        boolean isMirrored,
                        @NonNull
                        PixelFormat inputPixelFormat,
                        @NonNull
                        PixelFormat outputPixelFormat)
        Process image with current effect. Must be called from render thread. NOTE: inputImage have copy overhead, preferable to use process_image method MUST be called from the render thread
        Parameters:
        inputImage - input image with `width * height * 4` size
        outputPixelFormat - to avoid conversion recommended to use RGBA

      • pushFrame

        void pushFrame​(@NonNull
               FullImageData fullImage)
        Provides image to process and to play effect. Thread-safe. May be called from any thread

      • pushFrameWithNumber

        void pushFrameWithNumber​(@NonNull
                         FullImageData fullImage,
                         long frameNumber)
        Provides image to process and to play effect. Thread-safe. May be called from any thread

      • pushFrameData

        void pushFrameData​(@Nullable
                   FrameData frameData)
        Provides FrameData to process and to play effect. Must contain full image. Thread-safe. May be called from any thread

      • pushFrameDataWithNumber

        void pushFrameDataWithNumber​(@Nullable
                             FrameData frameData,
                             long frameNumber)
        Provides FrameData with frame number to process and to play effect. Must contain full image. Thread-safe. May be called from any thread

      • playbackPlay

        void playbackPlay()
        MUST be called from the main(render) thread

      • playbackPause

        void playbackPause()

      • playbackStop

        void playbackStop()

      • getInputManager

        @Nullable
InputManager getInputManager()
        Get interface to control user iterations. This events will be passed to effect. Thread-safe. May be called from any thread

      • startVideoProcessing

        void startVideoProcessing​(long screenWidth,
                          long screenHeight,
                          @NonNull
                          CameraOrientation orientation,
                          boolean resetEffect,
                          boolean offlineMode)
        Initialize video processing. To provide frames use *process_video_frame* methods. Effect audio is recorded as well and can be accessed using *process_recorded_audio* and *write_recorded_audio*. *push_frame* should not be called during processing. MUST be called from the render thread

      • stopVideoProcessing

        void stopVideoProcessing​(boolean resetEffect)
        Finish processing and return renderer to normal state. MUST be called from the render thread

      • processVideoFrameData

        @Nullable
FrameData processVideoFrameData​(@Nullable
                                FrameData inputFrameData,
                                @Nullable
                                java.lang.Integer recognizerIterations)
        Provide frame data to evaluate video processing. MUST be called from the render thread
        Parameters:
        inputFrameData - Frame Data to process.
        params - Processing params.
        recognizerIterations - Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.
        Returns:
        Frame data with frame processing results.

      • processVideoFrame

        @Nullable
FrameData processVideoFrame​(@NonNull
                            FullImageData inputImage,
                            @Nullable
                            java.lang.Integer recognizerIterations)
        Provide frame to evaluate video processing. MUST be called from the render thread
        Parameters:
        inputImage - Image to process.
        params - Processing params.
        recognizerIterations - Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.
        Returns:
        Frame data with frame processing results.

      • drawVideoFrame

        @NonNull
Data drawVideoFrame​(@Nullable
                    FrameData inputFrameData,
                    long timeNs,
                    @NonNull
                    PixelFormat outputPixelFormat)
        Draw processed frame to image buffer. MUST be called from the render thread
        Parameters:
        inputFrameData - FrameData to draw.
        timeNs - Frame position on timeline.
        outputPixelFormat - Output image format.
        Returns:
        Buffer with processed image in selected format.

      • drawVideoFrameAllocated

        @NonNull
Data drawVideoFrameAllocated​(@Nullable
                             FrameData inputFrameData,
                             long timeNs,
                             @NonNull
                             PixelFormat outputPixelFormat,
                             @NonNull
                             Data allocatedBuffer)
        The same as *draw_video_frame* but accepts preallocated buffer to avoid recreation for every frame. MUST be called from the render thread

      • writeRecordedAudio

        void writeRecordedAudio​(@NonNull
                        java.lang.String filename,
                        long lengthMs)
        Save recorded audio. Not thread-safe but can be called from any thread.
        Parameters:
        filename - ".wav" extension must be used.
        lengthMs - Track length. Set to '0' to use full duration.

      • onVideoRecordStart

        void onVideoRecordStart​(boolean playAudioWhileRecording)
        Trigger record start events in EP and Effect. Records all sounds during recording to be replayed by `processRecordedAudio`. If @param playAudioWhileRecording is true, the audio will continue to play. Otherwise, the audio will be muted. Thread-safe. May be called from any thread

      • onVideoRecordEnd

        void onVideoRecordEnd()
        Triggers record stop events in EP and Effect. Thread-safe. May be called from any thread

      • effectManager

        @Nullable
EffectManager effectManager()
        Get effect manager object Thread-safe. May be called from any thread

      • setRecognizerOfflineMode

        void setRecognizerOfflineMode​(boolean on)
        Force recognizer offline mode Thread-safe. May be called from any thread

      • setRecognizerUseFutureFilter

        void setRecognizerUseFutureFilter​(boolean on)
        Set future frame filtrarion mode. Produce smoother recognition result (anti jitter), however adds inconsistency in push'ed/draw'ed frames (one frame lag) Example: push frame 1 - draw frame 1, push frame 2 - draw frame 1, push frame 3 - draw frame 2, ...

      • setRecognizerUseFutureInterpolate

        void setRecognizerUseFutureInterpolate​(boolean on)
        Set future frame interpolation mode. Produce faster recognition result (skip even frames), however adds inconsistency in push'ed/pop'ed frames (one frame lag) Example: push frame 1 - pop frame 1, push frame 2 - pop frame 1, push frame 3 - pop frame 2, ...

      • setFrameProcessor

        void setFrameProcessor​(@Nullable
                       FrameProcessor processor)
        Set frame processor as current Thread-safe. May be called from any thread

      • frameProcessor

        @Nullable
FrameProcessor frameProcessor()
        Get current frame processor Thread-safe. May be called from any thread

      • setRenderBackend

        static void setRenderBackend​(@NonNull
                             RenderBackendType backendType)

      • getCurrentRenderBackendType

        @NonNull
static RenderBackendType getCurrentRenderBackendType()