Static & Animated Background

image

This tutorial will guide you on how to create effects with background segmentation using various types of media files, such as:

  • Video (.mp4) for animated backgrounds
  • 2D textures (.png) for static backgrounds

The example of creating a 3D background, one that changes with the device rotation, using environment textures (.ktx) may be found here.

Effect examples: test_BG_Texture and test_BG_Video.

Steps to implement

Preparation stage:

  1. Create a folder for your effect and name it the way you name the effect (bg_segmentation).
  2. Open Filter Editor, Press ‘Open’ button in the upper left corner and select ‘bg_segmentation’ folder created in the step 1.
  3. Press ‘Construct’ button in the upper right corner to create the effect. We’ll add our background files later. Now, no notifications will appear.
  4. Check your folder. You'll see a new folder created inside the effect folder with the same name. Files such as “config.js”, “cfg.toml”, “config.json” and “preview.png” are added to the new folder.

Setting up background segmentation (manually):

  1. Along with generated files put your sample media file into the newly generated effect folder.

    • For 2D texture background segmentation:
      • bg.vert
      • bg_tex.frag
      • paper.png (or any texture(s) you want)
      • tri.bsm2
    • For Video background segmentation:
      • bg.vert
      • bg_video.frag
      • Video.mp4 (or any video file you want)
      • tri.bsm2

    Sample files can be found in downloaded effect’s example archive.

  2. Edit cfg.toml:

    • For 2D texture background segmentation:
      • Write the material section:
        [materials.TriMat]
        vs = "bg.vert"
        fs = "bg_tex.frag"
        blend = "off" # "alpha", "premul_alpha", "screen", "add", "multiply"
        backfaces = false
        colorwrite = true
        zwrite = false
        shadow = false
        samplers = { tex=0 } # texture is set from config.js
      • Add TriMat material from tri.bsm2 model to the draw order:
        draw_order = [“TriMat”]
    • For Video background segmentation:
      • Write the material section:
        [materials.TriMat]
        vs = "bg.vert"
        fs = "bg_video.frag"
        blend = "off" # "alpha", "premul_alpha", "screen", "add", "multiply"
        backfaces = false
        colorwrite = true
        zwrite = false
        shadow = false
        samplers = {}
      • Add TriMat material from tri.bsm2 model to the draw order:
        draw_order = [“TriMat”]

    More about cfg.toml here.

  3. For background segmentation edit config.json:

    • For 2D texture background segmentation:
      • Enable the features. Add to general section:
        "recognizer": [
        "background"
        ]
    • For Video background segmentation:
      • Add media option to the frx section:
        "frx": {
        "type": "3d_anim",
        "3d_anim": {
        "type": "vfx"
        },
        "media": {
        "type": "VIDEO",
        "file": "video.mp4"
        }
        }
      • Enable the features. Add to general section:
        "recognizer": [
        "background"
        ]

    More about config.json here.

  4. Add the following code to Effect’s init method:

    • For 2D texture background segmentation:
      • Add tri.bsm2 mesh spawn: Api.meshfxMsg("spawn", 0, 0, "tri.bsm2");

      • Set BG texture: Api.meshfxMsg("tex", 0, 0, "paper.png");

        note

        You can change BG Texture in runtime, calling Api.meshfxMsg("tex"…) with new texture name;

    Bonus. You may color your texture by sending additional color data to the shader by calling: Api.meshfxMsg("shaderVec4", 0, 0, "1.0 0.0 0.0 0.");, "1.0 0.0 0.0 0.0" - RGBA data.

    • For Video background segmentation:
      • Add tri.bsm2 mesh spawn: Api.meshfxMsg("spawn", 0, 0, "tri.bsm2");
      • Add playVideo function call: Api.playVideo("frx", true, 1);

    More about config.js here.

image

The effect construction is completed. You can preview your effect in the Banuba Viewer application. To preview the effect, drag'n'drop it to the Previewer window.

Last updated on