BNBEffectPlayer Class Reference

The EffectPlayer class provides ability to play AR effects on set of images or video. More...

#import <BNBEffectPlayer.h>

Inheritance diagram for BNBEffectPlayer:

Instance Methods
(void)	- addFrameDurationListener:
	Add callback to receive FPS information. More...
(void)	- removeFrameDurationListener:
	Remove callback to receive FPS information. More...
(void)	- addFaceNumberListener:
	Add callback to receive faces count in frame. More...
(void)	- removeFaceNumberListener:
	Remove callback to receive faces count in frame. More...
(void)	- addFrameDataListener:
	Add callback to receive frame data right after processing in recognizer. More...
(void)	- removeFrameDataListener:
	Remove callback to receive frame data right after processing in recognizer. More...
(void)	- addCameraPoiListener:
	Add callback to receive center of the face in frame relative to top-left corner in [0, 1] space. More...
(void)	- removeCameraPoiListener:
	Remove callback to receive center of the face in frame relative to top-left Thread-safe. More...
(void)	- addEffectInfoListener:
	Add callback to receive current effect info from Effect Player. More...
(void)	- removeEffectInfoListener:
	Remove callback to receive current effect info from Effect Player. More...
(void)	- addEffectActivationCompletionListener:
	Add callback to receive current effect activation notification from Effect Player. More...
(void)	- removeEffectActivationCompletionListener:
	Remove callback to receive current effect activation notification from Effect Player. More...
(void)	- setMaxFaces:
	Sets maximum allowed face results, if face tracking feature is present. More...
(void)	- setExternalCameraTexture:width:height:
	Set OpenGL name of input external texture. More...
(void)	- setUseExtCamTex:
	Use external texture as input source. More...
(void)	- surfaceCreated:height:
	Use to notify the EffectPlayer that the surface exists and effect can be played. More...
(void)	- surfaceChanged:height:
	Notify about rendering surface being resized. More...
(void)	- surfaceDestroyed
	This method should be called right before an active context will become invalid. More...
(int64_t)	- draw
	Draw the current effect into the current OpenGL framebuffer. More...
(int64_t)	- drawWithExternalFrameData:
	Draw the current effect into the current OpenGL framebuffer. More...
(void)	- captureBlit:captureHeight:
	Record last rendering ("draw_()") result into current OpenGL framebuffer. More...
(nonnull BNBPixelBuffer *)	- readPixels
	Receive the last rendered frame in binary form. More...
(void)	- setRenderConsistencyMode:
	Thread-safe. More...
(nonnull NSData *)	- processImage:outputPixelFormat:
	Process an image with current effect. More...
(nonnull NSData *)	- processImageFrameData:outputPixelFormat:
	Process an image with current effect. More...
(nonnull NSData *)	- processImageData:width:height:orientation:isMirrored:inputPixelFormat:outputPixelFormat:
	Process image with current effect. More...
(void)	- pushFrame:
	Provides image to process and to play effect. More...
(void)	- pushFrameWithNumber:frameNumber:
	Provides image to process and to play effect. More...
(void)	- pushFrameData:
	Provides FrameData to process and to play effect. More...
(void)	- pushFrameDataWithNumber:frameNumber:
	Provides FrameData with frame number to process and to play effect. More...
(void)	- playbackPlay
	MUST be called from the main(render) thread. More...
(void)	- playbackPause
(void)	- playbackStop
(BNBEffectPlayerPlaybackState)	- getPlaybackState
	Thread-safe. More...
(nullable BNBInputManager *)	- getInputManager
	Get interface to control user iterations. More...
(void)	- startVideoProcessing:screenHeight:orientation:resetEffect:offlineMode:
	Initialize video processing. More...
(void)	- stopVideoProcessing:
	Finish processing and return renderer to normal state. More...
(nullable BNBFrameData *)	- processVideoFrameData:recognizerIterations:
	Provide frame data to evaluate video processing. More...
(nullable BNBFrameData *)	- processVideoFrame:recognizerIterations:
	Provide frame to evaluate video processing. More...
(nonnull NSData *)	- drawVideoFrame:timeNs:outputPixelFormat:
	Draw processed frame to image buffer. More...
(nonnull NSData *)	- drawVideoFrameAllocated:timeNs:outputPixelFormat:allocatedBuffer:
	The same as draw_video_frame but accepts preallocated buffer to avoid recreation for every frame. More...
(void)	- writeRecordedAudio:lengthMs:
	Save recorded audio. More...
(void)	- onVideoRecordStart:
	Trigger record start events in EP and Effect. More...
(void)	- onVideoRecordEnd
	Triggers record stop events in EP and Effect. More...
(nullable BNBEffectManager *)	- effectManager
	Get effect manager object Thread-safe. More...
(void)	- setRecognizerOfflineMode:
	Force recognizer offline mode Thread-safe. More...
(void)	- setRecognizerUseFutureFilter:
	Set future frame filtrarion mode. More...
(void)	- setRecognizerUseFutureInterpolate:
	Set future frame interpolation mode. More...
(void)	- setFrameProcessor:
	Set frame processor as current Thread-safe. More...
(nullable BNBFrameProcessor *)	- frameProcessor
	Get current frame processor Thread-safe. More...

Class Methods
(nullable BNBEffectPlayer *)	+ create:
(void)	+ setRenderBackend:
(BNBRenderBackendType)	+ getCurrentRenderBackendType

Detailed Description

The EffectPlayer class provides ability to play AR effects on set of images or video.

Lifecycle

In order to support an application lifecycle EffectPlayer provides special methods to control it during application states like pause or losing focus.

Effect playback

Initial playback state of newly created Effect Player instance is active. With playback control methods described below, the player can be launched/resumed or paused. State of effect player becomes stopped/inactive at the moment of surface loss or by calling the EffectPlayer::playback_stop method. One can use next three methods to control effect playback:
- EffectPlayer.playback_play. Switching to active state attempt. Possible from paused or stopped state and has no effect if effect playback is already active. Playback resumes from the position saved before EffectPlayer.playbackPause call.
- EffectPlayer.playback_pause. Suspend current playback attempt. The recognizer thread is stopped and all the video textures and audio units playback is stopped as well. Effect player doesn't react on EffectPlayer::pushFrame calls in suspended state except asynchronous-inconsistent mode.
- EffectPlayer.playbackStop. Switch to inactive state. In addition to pause clears recognizer's buffer. The next switch to active state will result in total rerun of active effect which means that it will be started from the very beginning.

Android usage example

If you want an application to be active on windowed mode and if the focus is on another application you can switch EffectPlayer into active state when onStart activity callback occurs and switch to paused when onStop occurs. Otherwise you can suspend EffectPlayer on onPause and resume on onResume callback to make an instance of the EffectPlayer inactive while activity losing focus.

Context/surface control

There are two methods that have to be used to ensure correct operating of the EffectPlayer. Losing context without notifying the EffectPlayer leads to application crash. Methods for context changes notification:
- EffectPlayer.surfaceCreated
- EffectPlayer.surfaceDestroyed

Multi-Threading performance and safety. Listeners.

EffectPlayer allows to call some of its methods from several threads (simultaneously) for convenience and increased performance. This is allowed only during normal operation, all invocations from other threads MUST finish before you start destroying the EffectPlayer instance. Most methods belong to one of the two groups:
- MUST be called from the "Main" (also "Render") thread ("Not Reentrant, Not Thread-safe")
- MAY be called from "Any" thread. Most are "Thread-safe" Listener callbacks can be called back from any thread. Generally for best performance and responsiveness you'll have 3 threads:
- Main(render) thread for drawing-related operations
- "Camera" thread for handling and push()-ing frames into EffectPlayer
- UI Thread for handling user interactions and other tasks

Definition at line 78 of file BNBEffectPlayer.h.

Method Documentation

addCameraPoiListener:

- (void) addCameraPoiListener:

(nullable id< BNBCameraPoiListener >)

cameraPoiListener

Add callback to receive center of the face in frame relative to top-left corner in [0, 1] space.

Thread-safe. May be called from any thread

addEffectActivationCompletionListener:

- (void) addEffectActivationCompletionListener:

(nullable id< BNBEffectActivationCompletionListener >)

effectActivationCompletionListener

Add callback to receive current effect activation notification from Effect Player.

Thread-safe. May be called from any thread

addEffectInfoListener:

- (void) addEffectInfoListener:

(nullable id< BNBEffectInfoListener >)

effectInfoListener

Add callback to receive current effect info from Effect Player.

Thread-safe. May be called from any thread

addFaceNumberListener:

- (void) addFaceNumberListener:

(nullable id< BNBFaceNumberListener >)

faceNumberListener

Add callback to receive faces count in frame.

Thread-safe. May be called from any thread

addFrameDataListener:

- (void) addFrameDataListener:

(nullable id< BNBFrameDataListener >)

frameDataListener

Add callback to receive frame data right after processing in recognizer.

Thread-safe. May be called from any thread

addFrameDurationListener:

- (void) addFrameDurationListener:

(nullable id< BNBFrameDurationListener >)

frameDurationListener

Add callback to receive FPS information.

Thread-safe. May be called from any thread

captureBlit:captureHeight:

- (void) captureBlit:		(int32_t)	captureWidth
captureHeight:		(int32_t)	captureHeight

Record last rendering ("draw_()") result into current OpenGL framebuffer.

Content is cropped to maintain effect_size() aspect ratio MUST be called from the render thread

create:

+ (nullable BNBEffectPlayer *) create:

(nullable BNBEffectPlayerConfiguration *)

configuration

draw

- (int64_t) draw

Draw the current effect into the current OpenGL framebuffer.

Uses internal frame_data object obtained from latest push_frame recognition result.

Return current frame number if drawing was performed and caller should swap buffers otherwise DRAW_SKIPPED(-1) MUST be called from the render thread

drawVideoFrame:timeNs:outputPixelFormat:

- (nonnull NSData *) drawVideoFrame:		(nullable BNBFrameData *)	inputFrameData
timeNs:		(int64_t)	timeNs
outputPixelFormat:		(BNBPixelFormat)	outputPixelFormat

Draw processed frame to image buffer.

MUST be called from the render thread

Parameters

inputFrameData	FrameData to draw.
timeNs	Frame position on timeline.
outputPixelFormat	Output image format.

Returns

Buffer with processed image in selected format.

drawVideoFrameAllocated:timeNs:outputPixelFormat:allocatedBuffer:

- (nonnull NSData *) drawVideoFrameAllocated:		(nullable BNBFrameData *)	inputFrameData
timeNs:		(int64_t)	timeNs
outputPixelFormat:		(BNBPixelFormat)	outputPixelFormat
allocatedBuffer:		(nonnull NSData *)	allocatedBuffer

The same as draw_video_frame but accepts preallocated buffer to avoid recreation for every frame.

MUST be called from the render thread

drawWithExternalFrameData:

- (int64_t) drawWithExternalFrameData:

(nullable BNBFrameData *)

frameData

Draw the current effect into the current OpenGL framebuffer.

Uses externally provided frameData object instead of internal one obtained from latest push_frame recognition result.

Return frame number from provided frameData if drawing was performed and caller should swap buffers otherwise DRAW_SKIPPED(-1) MUST be called from the render thread

effectManager

- (nullable BNBEffectManager *) effectManager

Get effect manager object Thread-safe.

May be called from any thread

frameProcessor

- (nullable BNBFrameProcessor *) frameProcessor

Get current frame processor Thread-safe.

May be called from any thread

getCurrentRenderBackendType

+ (BNBRenderBackendType) getCurrentRenderBackendType

getInputManager

- (nullable BNBInputManager *) getInputManager

Get interface to control user iterations.

This events will be passed to effect. Thread-safe. May be called from any thread

getPlaybackState

- (BNBEffectPlayerPlaybackState) getPlaybackState

Thread-safe.

May be called from any thread

onVideoRecordEnd

- (void) onVideoRecordEnd

Triggers record stop events in EP and Effect.

Thread-safe. May be called from any thread

onVideoRecordStart:

- (void) onVideoRecordStart:

(BOOL)

playAudioWhileRecording

Trigger record start events in EP and Effect.

Records all sounds during recording to be replayed by processRecordedAudio. If

Parameters

playAudioWhileRecording

is true, the audio will continue to play. Otherwise, the audio will be muted. Thread-safe. May be called from any thread

playbackPause

- (void) playbackPause

playbackPlay

- (void) playbackPlay

MUST be called from the main(render) thread.

playbackStop

- (void) playbackStop

processImage:outputPixelFormat:

- (nonnull NSData *) processImage:		(nonnull BNBFullImageData *)	inputImage
outputPixelFormat:		(BNBPixelFormat)	outputPixelFormat

Process an image with current effect.

Must be called from the render thread.

Parameters

inputImage	to avoid conversion recommended to use YUV image
outputPixelFormat	to avoid conversion recommended to use RGBA
params	extra image processing arguments

processImageData:width:height:orientation:isMirrored:inputPixelFormat:outputPixelFormat:

- (nonnull NSData *) processImageData:		(nonnull NSData *)	inputImage
width:		(int32_t)	width
height:		(int32_t)	height
orientation:		(BNBCameraOrientation)	orientation
isMirrored:		(BOOL)	isMirrored
inputPixelFormat:		(BNBPixelFormat)	inputPixelFormat
outputPixelFormat:		(BNBPixelFormat)	outputPixelFormat

Process image with current effect.

Must be called from render thread. NOTE: inputImage have copy overhead, preferable to use process_image method MUST be called from the render thread

Parameters

inputImage	input image with width * height * 4 size
outputPixelFormat	to avoid conversion recommended to use RGBA

processImageFrameData:outputPixelFormat:

- (nonnull NSData *) processImageFrameData:		(nullable BNBFrameData *)	inputFrameData
outputPixelFormat:		(BNBPixelFormat)	outputPixelFormat

Process an image with current effect.

Prefer this method over processImage when you have extra input data besides just an input image.

Must be called from render thread.

Parameters

inputFrameData	FrameData with an image
outputPixelFormat	to avoid conversion recommended to use RGBA
params	extra image processing arguments

processVideoFrame:recognizerIterations:

- (nullable BNBFrameData *) processVideoFrame:		(nonnull BNBFullImageData *)	inputImage
recognizerIterations:		(nullable NSNumber *)	recognizerIterations

Provide frame to evaluate video processing.

MUST be called from the render thread

Parameters

inputImage	Image to process.
params	Processing params.
recognizerIterations	Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.

Returns

Frame data with frame processing results.

processVideoFrameData:recognizerIterations:

- (nullable BNBFrameData *) processVideoFrameData:		(nullable BNBFrameData *)	inputFrameData
recognizerIterations:		(nullable NSNumber *)	recognizerIterations

Provide frame data to evaluate video processing.

MUST be called from the render thread

Parameters

inputFrameData	Frame Data to process.
params	Processing params.
recognizerIterations	Number of processing iterations. Higher number means higher processing quality, but lower speed. Must be greater than 1. Pass null value for default number.

Returns

Frame data with frame processing results.

pushFrame:

- (void) pushFrame:

(nonnull BNBFullImageData *)

fullImage

Provides image to process and to play effect.

Thread-safe. May be called from any thread

pushFrameData:

- (void) pushFrameData:

(nullable BNBFrameData *)

frameData

Provides FrameData to process and to play effect.

Must contain full image. Thread-safe. May be called from any thread

pushFrameDataWithNumber:frameNumber:

- (void) pushFrameDataWithNumber:		(nullable BNBFrameData *)	frameData
frameNumber:		(int64_t)	frameNumber

Provides FrameData with frame number to process and to play effect.

Must contain full image. Thread-safe. May be called from any thread

pushFrameWithNumber:frameNumber:

- (void) pushFrameWithNumber:		(nonnull BNBFullImageData *)	fullImage
frameNumber:		(int64_t)	frameNumber

Provides image to process and to play effect.

Thread-safe. May be called from any thread

readPixels

- (nonnull BNBPixelBuffer *) readPixels

Receive the last rendered frame in binary form.

Returns

4-byte per pixel data, size is fx_width * fx_height * 4 MUST be called from the render thread

removeCameraPoiListener:

- (void) removeCameraPoiListener:

(nullable id< BNBCameraPoiListener >)

cameraPoiListener

Remove callback to receive center of the face in frame relative to top-left Thread-safe.

May be called from any thread

removeEffectActivationCompletionListener:

- (void) removeEffectActivationCompletionListener:

(nullable id< BNBEffectActivationCompletionListener >)

effectActivationCompletionListener

Remove callback to receive current effect activation notification from Effect Player.

Thread-safe. May be called from any thread

removeEffectInfoListener:

- (void) removeEffectInfoListener:

(nullable id< BNBEffectInfoListener >)

effectInfoListener

Remove callback to receive current effect info from Effect Player.

Thread-safe. May be called from any thread

removeFaceNumberListener:

- (void) removeFaceNumberListener:

(nullable id< BNBFaceNumberListener >)

faceNumberListener

Remove callback to receive faces count in frame.

Thread-safe. May be called from any thread

removeFrameDataListener:

- (void) removeFrameDataListener:

(nullable id< BNBFrameDataListener >)

frameDataListener

Remove callback to receive frame data right after processing in recognizer.

Thread-safe. May be called from any thread

removeFrameDurationListener:

- (void) removeFrameDurationListener:

(nullable id< BNBFrameDurationListener >)

frameDurationListener

Remove callback to receive FPS information.

Thread-safe. May be called from any thread

setExternalCameraTexture:width:height:

- (void) setExternalCameraTexture:		(int32_t)	name
width:		(int32_t)	width
height:		(int32_t)	height

Set OpenGL name of input external texture.

https://developer.android.com/reference/android/graphics/SurfaceTexture . MUST be called from the render thread

Parameters

name	Texture name
width	Textute width
height	Texture height

See also

setUseExtCamTex

Note: This function is Android only.
Note: We use texture size to calculate aspect ratio only.

setFrameProcessor:

- (void) setFrameProcessor:

(nullable BNBFrameProcessor *)

processor

Set frame processor as current Thread-safe.

May be called from any thread

setMaxFaces:

- (void) setMaxFaces:

(int32_t)

maxFaces

Sets maximum allowed face results, if face tracking feature is present.

Thread-safe. May be called from any thread This option is relevant only for GLFX effects.

setRecognizerOfflineMode:

- (void) setRecognizerOfflineMode:

(BOOL)

on

Force recognizer offline mode Thread-safe.

May be called from any thread

setRecognizerUseFutureFilter:

- (void) setRecognizerUseFutureFilter:

(BOOL)

on

Set future frame filtrarion mode.

Produce smoother recognition result (anti jitter), however adds inconsistency in push'ed/draw'ed frames (one frame lag) Example: push frame 1 - draw frame 1, push frame 2 - draw frame 1, push frame 3 - draw frame 2, ...

setRecognizerUseFutureInterpolate:

- (void) setRecognizerUseFutureInterpolate:

(BOOL)

on

Set future frame interpolation mode.

Produce faster recognition result (skip even frames), however adds inconsistency in push'ed/pop'ed frames (one frame lag) Example: push frame 1 - pop frame 1, push frame 2 - pop frame 1, push frame 3 - pop frame 2, ...

setRenderBackend:

+ (void) setRenderBackend:

(BNBRenderBackendType)

backendType

setRenderConsistencyMode:

- (void) setRenderConsistencyMode:

(BNBConsistencyMode)

value

Thread-safe.

May be called from any thread

setUseExtCamTex:

- (void) setUseExtCamTex:

(BOOL)

value

Use external texture as input source.

MUST be called from the render thread

See also

setExternalCameraTexture

Note

This function is Android only

startVideoProcessing:screenHeight:orientation:resetEffect:offlineMode:

- (void) startVideoProcessing:		(int64_t)	screenWidth
screenHeight:		(int64_t)	screenHeight
orientation:		(BNBCameraOrientation)	orientation
resetEffect:		(BOOL)	resetEffect
offlineMode:		(BOOL)	offlineMode

Initialize video processing.

To provide frames use process_video_frame methods. Effect audio is recorded as well and can be accessed using process_recorded_audio and write_recorded_audio. push_frame should not be called during processing. MUST be called from the render thread

Exceptions

Exception

Must be aborted with stop_video_processing()

stopVideoProcessing:

- (void) stopVideoProcessing:

(BOOL)

resetEffect

Finish processing and return renderer to normal state.

MUST be called from the render thread

surfaceChanged:height:

- (void) surfaceChanged:		(int32_t)	width
height:		(int32_t)	height

Notify about rendering surface being resized.

MUST be called from the render thread

surfaceCreated:height:

- (void) surfaceCreated:		(int32_t)	width
height:		(int32_t)	height

Use to notify the EffectPlayer that the surface exists and effect can be played.

0, 0 for width and height are valid values. MUST be called from the render thread

surfaceDestroyed

- (void) surfaceDestroyed

This method should be called right before an active context will become invalid.

Switches playback state to inactive state. If it's not done an application will be crashed on next draw iteration. After losing the surface effect playback can't be resumed from last position. MUST be called from the render thread

writeRecordedAudio:lengthMs:

- (void) writeRecordedAudio:		(nonnull NSString *)	filename
lengthMs:		(int64_t)	lengthMs

Save recorded audio.

Not thread-safe but can be called from any thread.

Parameters

filename	".wav" extension must be used.
lengthMs	Track length. Set to '0' to use full duration.

---

The documentation for this class was generated from the following file:

• BNBEffectPlayer.h