Virtual Background (Beta)

Introduction

Virtual Background plugin helps in customising one’s background that is, replacing the background with a static image. This guide provides an overview of usage of the Virtual Background plugin of 100ms.

VirtualBackground

Supported Devices/Versions/Resolutions

Minimum Android api level required to support Virtual Background plugin is 21, same as that required to use 100ms SDK
Minimum 100ms SDK version it can work with is 2.2.8
Virtual background plugin is built for armeabi-v7a, arm64-v8a, x86 and x86_64 architectures
Maximum supported resolution for this feature is 480p
We recommend that you use this feature on a high performance device for smooth experience

Size Increase Per Architecture

x86_64 - 9.2 MB
x86 - 9.5 MB
arm64-v8a - 7.6 MB
armeabi-v7a - 5 MB

Common Terms

plugin - We are calling virtual background feature as a plugin
image background - The image which the plugin should replace the background with in the user's video.
plugin load time - The time taken by the plugin to load the machine learning model for the first time. The machine learning model will identify that what part of the image the background is and is essential to virtual background working.

Add dependency

Adding the Virtual Background plugin and SDK dependency to your app-level build.gradle.

build.gradle


dependencies {
 // See the version in the badge above.
 // There are separate libraries for the sdk, virtual background and hls-player-stats.
 //     add just the ones you need.
 def hmsVersion = "x.x.x"
 implementation "live.100ms:android-sdk:$hmsVersion"    // Essential
 implementation "live.100ms:virtual-background:$hmsVersion" // Optional
}

build.gradle


dependencies {
 def hmsVersion = "x.x.x"
 // See the version in the jitpack badge above.
 implementation "com.github.100mslive.android-sdk:lib:$hmsVersion"
 implementation "com.github.100mslive.android-sdk:stats:$hmsVersion"
}

How to Integrate Virtual Background

Instantiate VirtualBackgroundPlugin

Instantiate the 100ms Virtual Background plugin like this:


val hmsSdk = HMSSDK
      .Builder(application)
      .build()
val virtualBackgroundPlugin = HMSVirtualBackground(hmsSdk, imageBitmap)


HMSSDK hmsSdk = new HMSSDK
                       .Builder(application)
                       .build();
HMSVirtualBackground virtualBackgroundPlugin = new HMSVirtualBackground(hmsSdk, imageBitmap);

That's it. You have instantiated Virtual Background plugin successfully. 🥳

Now let's take a look at the method signature of HMSVirtualBackground.

HMSVirtualBackground accepts 2 argument -

hmsSdk: the HMSSDK instance you have used to join the room
imageBitmap- Bitmap - The image that the background will be replaced with. We are supporting JPG/PNG/JPEG type images.

Check if plugin is supported

HMSVirtualBackground.isSupported can be used to check if the API level/Resolution of device is supported or not. If the input resolution is set more than 480p we are not supporting the feature It will return True/False based on these parameters.


if(virtualBackgroundPlugin.isSupported()){
    // Device/Resolution is supported
}else{
    // Device/Resolution is not supported, check Supported Devices/Versions/Resolutions section
}


if(virtualBackgroundPlugin.isSupported()){
    // Device/Resolution is supported
}else{
   // Device/Resolution is not supported, check Supported Devices/Versions/Resolutions section
}

Init(Optional)

Init is used to load the machine learning model of Virtual Background for the first time. It takes on an average 50-200 ms. Calling init is handled internally by SDK if not done by user, in this case addPlugin call will take 50-200 ms for the first time and then less than 10ms in the subsequent calls. Check this section for addPlugin API usage


try{
   virtualBackgroundPlugin.init()
}catch(err){
    // failed to init Plugin
}


try{
   virtualBackgroundPlugin.init();
}catch(err){
   // failed to init Plugin
}

Change Background

Call setBackground on the HMSVirtualBackground instance to update the background again later if required. It accepts imageBitmap as a parameter

imageBitmap- Bitmap - The image that the background will be replaced with.

Image will be scaled to fit to the video. Scaling will maintain image aspect ratio. If the aspect ratio of the background image is not the same as the video, the image will be cropped to fit in the background.


//Use this function to get imageBitmap from a JPG/PNG Image
fun getBitmapFromAsset(context: Context, filename: String): Bitmap? {
  val assetManager = context.assets
  val instr: InputStream
  var bitmap: Bitmap? = null
  try {
    instr = assetManager.open(filename)
    bitmap = BitmapFactory.decodeStream(istr)
  } catch (e: IOException) {
    // error reading virtual background image
  }
  return bitmap
}


//BACKGROUND_FILE_PATH is the background image file location
val imageBitmap = getBitmapFromAsset(context, BACKGROUND_FILE_PATH)
//To set background as imageBitmap, Use
virtualBackgroundPlugin.setBackground(imageBitmap)


//Use this function to get imageBitmap from a JPG/PNG Image
Bitmap getBitmapFromAsset(context: Context, filename: String) {
  AssetManager assetManager = context.assets;
  InputStream instr;
  Bitmap bitmap;
  try {
    instr = assetManager.open(filename);
    bitmap = BitmapFactory.decodeStream(istr);
  } catch (e: IOException) {
     // error reading virtual background image
  }
  return bitmap;
}


//BACKGROUND_FILE_PATH is the background image file location
Bitmap imageBitmap = getBitmapFromAsset(context, BACKGROUND_FILE_PATH);
//To set background as imageBitmap, Use
virtualBackgroundPlugin.setBackground(imageBitmap);

Tuning pluginFrameRate(Optional)

pluginFrameRate - pluginFrameRate helps in controlling the performance and experience of Virtual Background plugin. pluginFrameRate translates to the number of frames for which background is detected. Higher value will use more resources (CPU/memory/battery), while making the Virtual Background experience smooth. Lower value will be generous on resources, while lowering the Virtual Background smoothness. Recommended value is 15. Values higher than this will not significantly improve Virtual Background smoothness but will be heavy on resources. For lower end devices value can be in the range of 7-10.

Add Plugin

To add Virtual background plugin app needs to call addPlugin method of HMSSDK which takes in 3 parameters -

virtualBackgroundPlugin: An instance of the plugin.
HMSActionResultListener: A callback for success or failure of adding the virtual background.
pluginFrameRate: An optional parameter for how many frames per second the virtual background image replacement should be updated. Default=15 if not set by user


// This will apply the Virtual Background effect to the local video
// pluginFrameRate is an optional parameter
hmsSdk.addPlugin(virtualBackgroundPlugin, object : HMSActionResultListener {
       override fun onError(error: HMSException) {
         // an error occurred
       }
       override fun onSuccess() {
         // added successfully
       }
      },
      pluginFrameRate)


// This will apply the Virtual Background effect to the local video
// pluginFrameRate is an optional parameter
hmsSdk.addPlugin(virtualBackgroundPlugin, new HMSActionResultListener {
       @Override
       public void onError(error: HMSException) {
         // an error occurred
       }
       @Override
       public void onSuccess() {
         // added successfully
       }
     },
     pluginFrameRate)

Remove Plugin

To remove the virtual background plugin the app needs to call removePlugin method of HMSSDK which takes in 2 parameters.

virtualBackgroundPlugin: An instance of the plugin.
HMSActionResultListener: A callback for success or failure of adding the virtual background.


//This will remove the virtual background effect from the video
hmsSdk.removePlugin(virtualBackgroundPlugin, object : HMSActionResultListener {
       override fun onError(error: HMSException) {
         // an error occurred
       }
       override fun onSuccess() {
         // added successfully
       }
      })


//This will remove the virtual background effect from the video
hmsSdk.removePlugin(virtualBackgroundPlugin, new HMSActionResultListener {
       @Override
       public void onError(error: HMSException) {
         // an error occurred
       }
       @Override
       public void onSuccess() {
         // added successfully
       }
     }
    )

Stats for HLS Player Noise Cancellation

Have a suggestion? Recommend changes ->