Banuba SDK
|
The EffectPlayer class provides ability to play AR effects on set of images or video. More...
#import <BNBEffectPlayer.h>
Instance Methods | |
(void) | - addFrameDurationListener: |
Add callback to receive FPS information. More... | |
(void) | - removeFrameDurationListener: |
Remove callback to receive FPS information. More... | |
(void) | - addFaceNumberListener: |
Add callback to receive faces count in frame. More... | |
(void) | - removeFaceNumberListener: |
Remove callback to receive faces count in frame. More... | |
(void) | - addFrameDataListener: |
Add callback to receive frame data right after processing in recognizer. More... | |
(void) | - removeFrameDataListener: |
Remove callback to receive frame data right after processing in recognizer. More... | |
(void) | - addLowLightListener: |
Add callback to receive low light info Thread-safe. More... | |
(void) | - removeLowLightListener: |
Remove callback to receive low light info Thread-safe. More... | |
(void) | - addCameraPoiListener: |
Add callback to receive center of the face in frame relative to top-left corner in [0, 1] space. More... | |
(void) | - removeCameraPoiListener: |
Remove callback to receive center of the face in frame relative to top-left Thread-safe. More... | |
(void) | - addEffectInfoListener: |
Add callback to receive current effect info from Effect Player. More... | |
(void) | - removeEffectInfoListener: |
Remove callback to receive current effect info from Effect Player. More... | |
(void) | - addEffectActivationCompletionListener: |
Add callback to receive current effect activation notification from Effect Player. More... | |
(void) | - removeEffectActivationCompletionListener: |
Remove callback to receive current effect activation notification from Effect Player. More... | |
(void) | - setMaxFaces: |
Sets maximum allowed face results, if face tracking feature is present. More... | |
(void) | - setExternalCameraTexture:width:height: |
Set OpenGL name of input external texture. More... | |
(void) | - setUseExtCamTex: |
Use external texture as input source. More... | |
(void) | - surfaceCreated:height: |
Use to notify the EffectPlayer that the surface exists and effect can be played. More... | |
(void) | - surfaceChanged:height: |
Notify about rendering surface being resized. More... | |
(void) | - surfaceDestroyed |
This method should be called right before an active context will become invalid. More... | |
(int64_t) | - draw |
Draw the current effect into the current OpenGL framebuffer. More... | |
(int64_t) | - drawWithExternalFrameData: |
Draw the current effect into the current OpenGL framebuffer. More... | |
(void) | - captureBlit:captureHeight: |
Record last rendering ("draw_()") result into current OpenGL framebuffer. More... | |
(nonnull NSData *) | - readPixels:height: |
Receive the last rendered frame in binary form. More... | |
(void) | - setRenderConsistencyMode: |
Thread-safe. More... | |
(void) | - setRenderTransform:viewportRect:xFlip:yFlip: |
Request display of sub-area of the input image into sub-area of the output surface, with optional x,y flips imageRect is fitted inside viewportRect Resets transform to default if either rect has 0 dimensions. More... | |
(nonnull NSData *) | - processImage:outputPixelFormat:params: |
Process an image with current effect. More... | |
(nonnull NSData *) | - processImageFrameData:outputPixelFormat:params: |
Process an image with current effect. More... | |
(nonnull NSData *) | - processImageData:width:height:orientation:isMirrored:inputPixelFormat:outputPixelFormat:params: |
Process image with current effect. More... | |
(void) | - pushFrame: |
Provides image to process and to play effect. More... | |
(void) | - pushFrameWithNumber:frameNumber: |
Provides image to process and to play effect. More... | |
(void) | - pushFrameData: |
Provides FrameData to process and to play effect. More... | |
(void) | - playbackPlay |
MUST be called from the main(render) thread. More... | |
(void) | - playbackPause |
(void) | - playbackStop |
(BNBEffectPlayerPlaybackState) | - getPlaybackState |
Thread-safe. More... | |
(nullable BNBScene *) | - getCurrentScene |
If currently loaded effect is based on new Scene engine and it is loaded completly returns Scene object from the effect. More... | |
(void) | - setEffectVolume: |
Set effect audio volume. More... | |
(void) | - enableAudio: |
Set audio enabled. More... | |
(nullable BNBInputManager *) | - getInputManager |
Get interface to control user iterations. More... | |
(void) | - startVideoProcessing:screenHeight:orientation:resetEffect:offlineMode: |
Initialize video processing. More... | |
(void) | - stopVideoProcessing: |
Finish processing and return renderer to normal state. More... | |
(nullable BNBFrameData *) | - processVideoFrameData:params:recognizerIterations: |
Provide frame data to evaluate video processing. More... | |
(nullable BNBFrameData *) | - processVideoFrame:params:recognizerIterations: |
Provide frame to evaluate video processing. More... | |
(nonnull NSData *) | - drawVideoFrame:timeNs:outputPixelFormat: |
Draw processed frame to image buffer. More... | |
(nonnull NSData *) | - drawVideoFrameAllocated:timeNs:outputPixelFormat:allocatedBuffer: |
The same as draw_video_frame but accepts preallocated buffer to avoid recreation for every frame. More... | |
(BOOL) | - isVoiceChangerConfigured |
Thread-safe. More... | |
(void) | - processRecordedAudio:outFilename:mixVolume: |
Process recorded voice with voice changer effects mixing with recorded sounds. More... | |
(void) | - writeRecordedAudio:lengthMs: |
Save recorded audio. More... | |
(void) | - onVideoRecordStart: |
Trigger record start events in EP and Effect. More... | |
(void) | - onVideoRecordEnd |
Triggers record stop events in EP and Effect. More... | |
(BOOL) | - isDeviceNnCompatible |
Check is device compatible with Neural Networks player Thread-safe. More... | |
(void) | - startFramedataCapture:filename: |
Request to start framedata capture process. More... | |
(void) | - stopFramedataCapture |
(void) | - startAnalyticsCapture:config:listener: |
Request to start analytics capture process. More... | |
(void) | - stopAnalyticsCapture |
(nullable BNBEffectManager *) | - effectManager |
Get effect manager object Thread-safe. More... | |
(nullable BNBDebugInterface *) | - debugInterface |
For internal usage only. More... | |
(void) | - setRecognizerOfflineMode: |
Force recognizer offline mode Thread-safe. More... | |
(void) | - setFaceSearchMode: |
Change face search mode Thread-safe. More... | |
Class Methods | |
(nullable BNBEffectPlayer *) | + create: |
(int32_t) | + versionMajor |
Get major version of EffectPlayer. More... | |
(int32_t) | + versionMinor |
Get minor version of EffectPlayer. More... | |
The EffectPlayer class provides ability to play AR effects on set of images or video.
In order to support an application lifecycle EffectPlayer provides special methods to control it during application states like pause or losing focus.
Initial playback state of newly created Effect Player instance is active
. With playback control methods described below, the player can be launched/resumed or paused. State of effect player becomes stopped/inactive at the moment of surface loss or by calling the EffectPlayer::playback_stop method. One can use next three methods to control effect playback:
- EffectPlayer.playback_play
. Switching to active state attempt. Possible from paused or stopped state and has no effect if effect playback is already active. Playback resumes from the position saved before EffectPlayer.playbackPause
call.
- EffectPlayer.playback_pause
. Suspend current playback attempt. The recognizer thread is stopped and all the video textures and audio units playback is stopped as well. Effect player doesn't react on EffectPlayer::pushFrame calls in suspended state except asynchronous-inconsistent mode.
- EffectPlayer.playbackStop
. Switch to inactive state. In addition to pause clears recognizer's buffer. The next switch to active state will result in total rerun of active effect which means that it will be started from the very beginning.
If you want an application to be active on windowed mode and if the focus is on another application you can switch EffectPlayer into active state when onStart
activity callback occurs and switch to paused when onStop
occurs. Otherwise you can suspend EffectPlayer
on onPause
and resume on onResume
callback to make an instance of the EffectPlayer
inactive while activity losing focus.
There are two methods that have to be used to ensure correct operating of the EffectPlayer. Losing context without notifying the EffectPlayer leads to application crash. Methods for context changes notification:
- EffectPlayer.surfaceCreated
- EffectPlayer.surfaceDestroyed
EffectPlayer allows to call some of its methods from several threads (simultaneously) for convenience and increased performance. This is allowed only during normal operation, all invocations from other threads MUST finish before you start destroying the EffectPlayer instance. Most methods belong to one of the two groups:
- MUST be called from the "Main" (also "Render") thread ("Not Reentrant, Not Thread-safe")
- MAY be called from "Any" thread. Most are "Thread-safe" Listener callbacks can be called back from any thread. Generally for best performance and responsiveness you'll have 3 threads:
- Main(render) thread for drawing-related operations
- "Camera" thread for handling and push()-ing frames into EffectPlayer
- UI Thread for handling user interactions and other tasks
Definition at line 77 of file BNBEffectPlayer.h.
- (void) addCameraPoiListener: | (nullable id< BNBCameraPoiListener >) | cameraPoiListener |
Add callback to receive center of the face in frame relative to top-left corner in [0, 1] space.
Thread-safe. May be called from any thread
- (void) addEffectActivationCompletionListener: | (nullable id< BNBEffectActivationCompletionListener >) | effectActivationCompletionListener |
Add callback to receive current effect activation notification from Effect Player.
Thread-safe. May be called from any thread
- (void) addEffectInfoListener: | (nullable id< BNBEffectInfoListener >) | effectInfoListener |
Add callback to receive current effect info from Effect Player.
Thread-safe. May be called from any thread
- (void) addFaceNumberListener: | (nullable id< BNBFaceNumberListener >) | faceNumberListener |
Add callback to receive faces count in frame.
Thread-safe. May be called from any thread
- (void) addFrameDataListener: | (nullable id< BNBFrameDataListener >) | frameDataListener |
Add callback to receive frame data right after processing in recognizer.
Thread-safe. May be called from any thread
- (void) addFrameDurationListener: | (nullable id< BNBFrameDurationListener >) | frameDurationListener |
Add callback to receive FPS information.
Thread-safe. May be called from any thread
- (void) addLowLightListener: | (nullable id< BNBLowLightListener >) | lowLightListener |
Add callback to receive low light info Thread-safe.
May be called from any thread
- (void) captureBlit: | (int32_t) | captureWidth | |
captureHeight: | (int32_t) | captureHeight | |
Record last rendering ("draw_()") result into current OpenGL framebuffer.
Content is cropped to maintain effect_size() aspect ratio MUST be called from the render thread
+ (nullable BNBEffectPlayer *) create: | (nonnull BNBEffectPlayerConfiguration *) | configuration |
- (nullable BNBDebugInterface *) debugInterface |
For internal usage only.
- (int64_t) draw |
Draw the current effect into the current OpenGL framebuffer.
Uses internal frame_data object obtained from latest push_frame recognition result.
Return current frame number if drawing was performed and caller should swap buffers otherwise DRAW_SKIPPED
(-1) MUST be called from the render thread
- (nonnull NSData *) drawVideoFrame: | (nullable BNBFrameData *) | inputFrameData | |
timeNs: | (int64_t) | timeNs | |
outputPixelFormat: | (BNBPixelFormat) | outputPixelFormat | |
Draw processed frame to image buffer.
MUST be called from the render thread
inputFrameData | FrameData to draw. |
timeNs | Frame position on timeline. |
outputPixelFormat | Output image format. |
- (nonnull NSData *) drawVideoFrameAllocated: | (nullable BNBFrameData *) | inputFrameData | |
timeNs: | (int64_t) | timeNs | |
outputPixelFormat: | (BNBPixelFormat) | outputPixelFormat | |
allocatedBuffer: | (nonnull NSData *) | allocatedBuffer | |
The same as draw_video_frame but accepts preallocated buffer to avoid recreation for every frame.
MUST be called from the render thread
- (int64_t) drawWithExternalFrameData: | (nullable BNBFrameData *) | frameData |
Draw the current effect into the current OpenGL framebuffer.
Uses externally provided frameData object instead of internal one obtained from latest push_frame recognition result.
Return frame number from provided frameData if drawing was performed and caller should swap buffers otherwise DRAW_SKIPPED
(-1) MUST be called from the render thread
- (nullable BNBEffectManager *) effectManager |
Get effect manager object Thread-safe.
May be called from any thread
- (void) enableAudio: | (BOOL) | enable |
Set audio enabled.
- (nullable BNBScene *) getCurrentScene |
If currently loaded effect is based on new Scene engine and it is loaded completly returns Scene object from the effect.
Otherwise returns null. MUST be called from the render thread Returned Scene object MUST be used only from the render thread and only while the effect is in the loaded state.
- (nullable BNBInputManager *) getInputManager |
Get interface to control user iterations.
This events will be passed to effect. Thread-safe. May be called from any thread
- (BNBEffectPlayerPlaybackState) getPlaybackState |
Thread-safe.
May be called from any thread
- (BOOL) isDeviceNnCompatible |
Check is device compatible with Neural Networks player Thread-safe.
May be called from any thread On some platforms (e.g. Android) may require the first invocation to be on the render thread
- (BOOL) isVoiceChangerConfigured |
Thread-safe.
May be called from any thread
- (void) onVideoRecordEnd |
Triggers record stop events in EP and Effect.
Thread-safe. May be called from any thread
- (void) onVideoRecordStart: | (BOOL) | playAudioWhileRecording |
Trigger record start events in EP and Effect.
Records all sounds during recording to be replayed by processRecordedAudio
. If
playAudioWhileRecording | is true, the audio will continue to play. Otherwise, the audio will be muted. Thread-safe. May be called from any thread |
- (void) playbackPause |
- (void) playbackPlay |
MUST be called from the main(render) thread.
- (void) playbackStop |
- (nonnull NSData *) processImage: | (nonnull BNBFullImageData *) | inputImage | |
outputPixelFormat: | (BNBPixelFormat) | outputPixelFormat | |
params: | (nonnull BNBProcessImageParams *) | params | |
Process an image with current effect.
Must be called from the render thread.
inputImage | to avoid conversion recommended to use YUV image |
outputPixelFormat | to avoid conversion recommended to use RGBA |
params | extra image processing arguments |
- (nonnull NSData *) processImageData: | (nonnull NSData *) | inputImage | |
width: | (int32_t) | width | |
height: | (int32_t) | height | |
orientation: | (BNBCameraOrientation) | orientation | |
isMirrored: | (BOOL) | isMirrored | |
inputPixelFormat: | (BNBPixelFormat) | inputPixelFormat | |
outputPixelFormat: | (BNBPixelFormat) | outputPixelFormat | |
params: | (nonnull BNBProcessImageParams *) | params | |
Process image with current effect.
Must be called from render thread. NOTE: inputImage have copy overhead, preferable to use process_image method MUST be called from the render thread
inputImage | input image with width * height * 4 size |
outputPixelFormat | to avoid conversion recommended to use RGBA |
- (nonnull NSData *) processImageFrameData: | (nullable BNBFrameData *) | inputFrameData | |
outputPixelFormat: | (BNBPixelFormat) | outputPixelFormat | |
params: | (nonnull BNBProcessImageParams *) | params | |
Process an image with current effect.
Prefer this method over processImage
when you have extra input data besides just an input image.
Must be called from render thread.
inputFrameData | FrameData with an image |
outputPixelFormat | to avoid conversion recommended to use RGBA |
params | extra image processing arguments |
- (void) processRecordedAudio: | (nonnull NSString *) | inFilename | |
outFilename: | (nonnull NSString *) | outFilename | |
mixVolume: | (float) | mixVolume | |
Process recorded voice with voice changer effects mixing with recorded sounds.
Copies in->out if no voice changer or effect is active. Not thread-safe but can be called from any thread. Recorded sounds have the same lifetime as the effect(e.g reset after load_effect, etc.).
inFilename | Input filename. |
outFilename | Must have ".wav" extension. |
mixVolume | Relative volume for mixing in sounds, [0..1]. |
- (nullable BNBFrameData *) processVideoFrame: | (nonnull BNBFullImageData *) | inputImage | |
params: | (nonnull BNBProcessImageParams *) | params | |
recognizerIterations: | (nullable NSNumber *) | recognizerIterations | |
Provide frame to evaluate video processing.
MUST be called from the render thread
inputImage | Image to process. |
params | Processing params. |
recognizerIterations | Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number. |
- (nullable BNBFrameData *) processVideoFrameData: | (nullable BNBFrameData *) | inputFrameData | |
params: | (nonnull BNBProcessImageParams *) | params | |
recognizerIterations: | (nullable NSNumber *) | recognizerIterations | |
Provide frame data to evaluate video processing.
MUST be called from the render thread
inputFrameData | Frame Data to process. |
params | Processing params. |
recognizerIterations | Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number. |
- (void) pushFrame: | (nonnull BNBFullImageData *) | fullImage |
Provides image to process and to play effect.
Thread-safe. May be called from any thread
- (void) pushFrameData: | (nullable BNBFrameData *) | frameData |
Provides FrameData to process and to play effect.
Must contain full image. Thread-safe. May be called from any thread
- (void) pushFrameWithNumber: | (nonnull BNBFullImageData *) | fullImage | |
frameNumber: | (int64_t) | frameNumber | |
Provides image to process and to play effect.
Thread-safe. May be called from any thread
- (nonnull NSData *) readPixels: | (int32_t) | width | |
height: | (int32_t) | height | |
Receive the last rendered frame in binary form.
width | width of the picture to take, must match the values passed to setEffectSize or create |
height | height of the picture to take, must match the values passed to setEffectSize or create |
width * height * 4
MUST be called from the render thread - (void) removeCameraPoiListener: | (nullable id< BNBCameraPoiListener >) | cameraPoiListener |
Remove callback to receive center of the face in frame relative to top-left Thread-safe.
May be called from any thread
- (void) removeEffectActivationCompletionListener: | (nullable id< BNBEffectActivationCompletionListener >) | effectActivationCompletionListener |
Remove callback to receive current effect activation notification from Effect Player.
Thread-safe. May be called from any thread
- (void) removeEffectInfoListener: | (nullable id< BNBEffectInfoListener >) | effectInfoListener |
Remove callback to receive current effect info from Effect Player.
Thread-safe. May be called from any thread
- (void) removeFaceNumberListener: | (nullable id< BNBFaceNumberListener >) | faceNumberListener |
Remove callback to receive faces count in frame.
Thread-safe. May be called from any thread
- (void) removeFrameDataListener: | (nullable id< BNBFrameDataListener >) | frameDataListener |
Remove callback to receive frame data right after processing in recognizer.
Thread-safe. May be called from any thread
- (void) removeFrameDurationListener: | (nullable id< BNBFrameDurationListener >) | frameDurationListener |
Remove callback to receive FPS information.
Thread-safe. May be called from any thread
- (void) removeLowLightListener: | (nullable id< BNBLowLightListener >) | lowLightListener |
Remove callback to receive low light info Thread-safe.
May be called from any thread
- (void) setEffectVolume: | (float) | volume |
Set effect audio volume.
Thread-safe. May be called from any thread
volume | A value in range [0, 1] , where 1 means maximum volume. |
- (void) setExternalCameraTexture: | (int32_t) | name | |
width: | (int32_t) | width | |
height: | (int32_t) | height | |
Set OpenGL name of input external texture.
https://developer.android.com/reference/android/graphics/SurfaceTexture . MUST be called from the render thread
name | Texture name |
width | Textute width |
height | Texture height |
Note: This function is Android only.
Note: We use texture size to calculate aspect ratio only.
- (void) setFaceSearchMode: | (BNBFaceSearchMode) | faceSearch |
Change face search mode Thread-safe.
May be called from any thread
- (void) setMaxFaces: | (int32_t) | maxFaces |
Sets maximum allowed face results, if face tracking feature is present.
Thread-safe. May be called from any thread This option is relevant only for GLFX effects.
- (void) setRecognizerOfflineMode: | (BOOL) | on |
Force recognizer offline mode Thread-safe.
May be called from any thread
- (void) setRenderConsistencyMode: | (BNBConsistencyMode) | value |
Thread-safe.
May be called from any thread
- (void) setRenderTransform: | (nonnull BNBPixelRect *) | imageRect | |
viewportRect: | (nonnull BNBPixelRect *) | viewportRect | |
xFlip: | (BOOL) | xFlip | |
yFlip: | (BOOL) | yFlip | |
Request display of sub-area of the input image into sub-area of the output surface, with optional x,y flips imageRect is fitted inside viewportRect Resets transform to default if either rect has 0 dimensions.
imageRect | rectangle in input image coordinates(pixels) after applying input rotations and flips |
viewportRect | rectangle in output surface coordinates(pixels) MUST be called from the render thread |
- (void) setUseExtCamTex: | (BOOL) | value |
Use external texture as input source.
MUST be called from the render thread
- (void) startAnalyticsCapture: | (nonnull NSDictionary< NSString *, NSString * > *) | deviceInfo | |
config: | (nonnull BNBAnalyticsConfig *) | config | |
listener: | (nullable id< BNBAnalyticsListener >) | listener | |
Request to start analytics capture process.
Analytics supports two business cases:
deviceInfo | Any user specific information to attach in analytics report. |
config | Setup configurable properties (like frames counter and delay before capture). |
listener | Callback which is fired in one of two above mentioned cases. |
- (void) startFramedataCapture: | (nonnull NSString *) | folder | |
filename: | (nullable NSString *) | filename | |
Request to start framedata capture process.
Output file is in CBOR format. Thread-safe. May be called from any thread
folder | Output folder. |
filename | Output filename. If the value is an empty string, filename is generated based on date and time. |
- (void) startVideoProcessing: | (int64_t) | screenWidth | |
screenHeight: | (int64_t) | screenHeight | |
orientation: | (BNBCameraOrientation) | orientation | |
resetEffect: | (BOOL) | resetEffect | |
offlineMode: | (BOOL) | offlineMode | |
Initialize video processing.
To provide frames use process_video_frame methods. Effect audio is recorded as well and can be accessed using process_recorded_audio and write_recorded_audio. push_frame should not be called during processing. MUST be called from the render thread
Exception | Must be aborted with stop_video_processing() |
- (void) stopAnalyticsCapture |
- (void) stopFramedataCapture |
- (void) stopVideoProcessing: | (BOOL) | resetEffect |
Finish processing and return renderer to normal state.
MUST be called from the render thread
- (void) surfaceChanged: | (int32_t) | width | |
height: | (int32_t) | height | |
Notify about rendering surface being resized.
MUST be called from the render thread
- (void) surfaceCreated: | (int32_t) | width | |
height: | (int32_t) | height | |
Use to notify the EffectPlayer that the surface exists and effect can be played.
0, 0 for width and height are valid values. MUST be called from the render thread
- (void) surfaceDestroyed |
This method should be called right before an active context will become invalid.
Switches playback state to inactive state. If it's not done an application will be crashed on next draw iteration. After losing the surface effect playback can't be resumed from last position. MUST be called from the render thread
+ (int32_t) versionMajor |
Get major version of EffectPlayer.
Use this method to filter out breaking changes in implementation of this class.
+ (int32_t) versionMinor |
Get minor version of EffectPlayer.
- (void) writeRecordedAudio: | (nonnull NSString *) | filename | |
lengthMs: | (int64_t) | lengthMs | |
Save recorded audio.
Not thread-safe but can be called from any thread.
filename | ".wav" extension must be used. |
lengthMs | Track length. Set to '0' to use full duration. |