Tushar Tripathi

How cool would it be if you could build your own Video Chat app with Snapchat-like filters in it!

 ![Super Cool and Amazing Meme Corey Vidal](https://storage.googleapis.com/100ms-cms-prod/cms/super_cool_and_amazing_corey_vidal_21b3f82939/super_cool_and_amazing_corey_vidal_21b3f82939.gif)  

Ikr! That's what I was thinking when I came across [Jeeliz](https://github.com/jeeliz/jeelizFaceFilter). Now I have worked with `tensorflow.js` based libraries in the past but they're usually quite CPU intensive for a live video use case. Jeeliz looked promising as it's designed for this use case. So I thought why not try it out by adding some 3d AR filters to our video calls? Well! that is what we're going to do.

**Click here to learn more on [how to add live interactive videos to your product](https://www.100ms.live/blog/adding-live-interactive-video)**.

We'll use React and 100ms' React SDK for the video call part of our application. 100ms, in short, is building developer-focused live SDKs which abstract away the low-level complexities. Support for [video plugins](https://docs.100ms.live/javascript/v2/plugins/custom-video-plugins) was released recently which makes it easier to experiment with AR filters after setting up a basic app. 

**Stream in style! Add these AR filters to your [Twitch like app built with 100ms](https://www.100ms.live/blog/twitch-clone-in-react)**

And so I set forth on the journey. I'll mostly be talking about the implementation details related to the filters themselves in this blog than setting up the video call app from scratch. You can check out the [quickstart guide](https://docs.100ms.live/javascript/v2/guides/react-quickstart) for a quick overview of the SDK and how it works, or you can just fork it (it's also the first step 😀) and follow along with my exploration.

**[Start with 100ms React SDK and get 10,000 free minutes every month](https://dashboard.100ms.live/register)**

I have split the blog into parts so it's not overwhelming. In this part, we'll try to understand the plugin interface exposed by the SDK, learn a bit about HTML Canvas elements and implement a basic filter. We'll go into more detail about AR, WebGL, and implementing the AR filter plugin in further parts.

Everything we'll do is available in [this Github repo](https://github.com/triptu/100ms-face-filters), and I have linked to the relevant commit for each step. By the end of this blog, we'll be able to build a simple grayscale filter.

![AR Filters Demo](https://storage.googleapis.com/100ms-cms-prod/cms/AR_Filters_Demo_9d07e317ab/AR_Filters_Demo_9d07e317ab.gif) 

Looks cool? You can check the [demo of the final app here](https://hms-face-filters.netlify.app/). Let's get started with the code part.

## Fork the quickstart

This step can be skipped if you're integrating filters in an existing web app already using the 100ms SDKs. If that is not the case let's start with forking the [codesandbox](https://codesandbox.io/s/happy-meddling-syndrome-q4ukf) linked in the doc to a GitHub repo. Now I have already done it so forking my [GitHub repo](https://github.com/triptu/100ms-face-filters/tree/original) will be much faster. The initial code lies in the branch named `original`.

You can also check out the branch to follow locally -

```bash
git clone -b original https://github.com/triptu/100ms-face-filters.git
```

## Run the app locally

We can clone the repo now and run it locally. Feel free to update to the latest versions [here](https://www.100ms.live/docs/javascript/v2/changelog/release-notes) of the SDKs and then run using `yarn install` followed by `yarn start`. We'll see a screen like this if everything worked fine -

![100ms Dashboard](https://storage.googleapis.com/100ms-cms-prod/cms/100ms_Login_Dashboard_62c6c2c5ba/100ms_Login_Dashboard_62c6c2c5ba.png) 

In case you're wondering what that auth token is, we can imagine them to be the meeting id that tells 100ms which room to put you in. Getting such a token is fairly straightforward(doesn't require anything technical or code) and is given in more detail [here](https://docs.100ms.live/javascript/v2/guides/token). Once you get the token, verify that everything is working fine. You can try joining from multiple tabs or sharing the link with your friends(after exposing with [ngrok](https://ngrok.com/) ofc). You can also join the same room from the link available on the dashboard(where the token was copied from).

## Grayscale Filter

![Images explaining how grayscale filter works](https://storage.googleapis.com/100ms-cms-prod/cms/Grayscale_Filter_ac4983bdcf/Grayscale_Filter_ac4983bdcf.png) 

Let's say that we have to convert a colorful image to Grayscale and we're wondering what would it take. To answer this let's try to break down the image into further parts. An image is a matrix of many pixels where a single pixel can be described using three numbers from 0-255, the intensity values of red, green, and blue. For a grayscale image, each pixel can be described as only one number ranging from 0-255 with 0 being black(lowest intensity) and 255 being white(highest intensity).
Now if we were to convert a colored pixel with RGB values into grayscale, we will need some sort of mapping between both. A fairly straightforward way to map these is to average out the three intensities -

```js
intensity = (red + blue + green)/3
``` 
But this won't result in a balanced grayscale image. The reason for it is that our eyes react differently to each color being most sensitive to green and least to blue. For our filter, we'll go with [Luma](https://en.wikipedia.org/wiki/Luma_(video)) which is a weighted sum of the RGB values and maps to the luminance much more accurately.


```js
// Luma
intensity = red * 0.299 + green * 0.587 + blue * 0.114 
```

## Going through the Plugin Docs

Now that we're all set with the algorithm to convert an RGB image to grayscale, let's move ahead with checking out how we can write a plugin to implement this. The documentation is [here](https://docs.100ms.live/javascript/v2/plugins/custom-video-plugins), and fortunately, I've read it so you don't have to.

![100ms documentation for AR filters](https://storage.googleapis.com/100ms-cms-prod/cms/100ms_Docs_For_AR_Plugin_cd5022483a/100ms_Docs_For_AR_Plugin_cd5022483a.gif) 

The gist of it is that we have to write a class that implements a method `processVideoFrame(inputCanvas, outputCanvas)`, where we're passed in an image on the input canvas and have to put a result image on the output canvas. This makes the job fairly easy for us as we don't have to worry about video but just one image at a time. So as long as we can find a way to get RGB values from the input canvas and put the grayscale values on the output canvas, we should be able to implement the algorithm discussed and we'll be good.

## Implementing the Grayscale Plugin

Check out the full commit [in my github](https://github.com/triptu/100ms-face-filters/commit/be082a87e1330d66d18eb32e702881558d26e183).

So as we figured out from the docs, it's [HTML Canvas](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API) we're going to deal with. Now canvas has something called a context which exposes direct methods for getting the RGB values from a canvas([getImageData](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/getImageData)) and applying them([putImageData](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/putImageData)). With this information, we can begin writing our GrayScale Plugin. I have added further comments in the code below. Note that some other methods are present too as they're required by the SDK.

```js
class GrayscalePlugin {
   /**
   * @param input {HTMLCanvasElement}
   * @param output {HTMLCanvasElement}
   */
  processVideoFrame(input, output) {
    // we don't want to change the dimensions so set the same width, height
    const width = input.width;
    const height = input.height;
    output.width = width;
    output.height = height;
    const inputCtx = input.getContext("2d");
    const outputCtx = output.getContext("2d");
    const imgData = inputCtx.getImageData(0, 0, width, height);
    const pixels = imgData.data; 
    // pixels is an array of all the pixels with their RGBA values, the A stands for alpha
    // we will not actually be using alpha for this plugin, but we still need to skip it(hence the i+= 4)
    for (let i = 0; i < pixels.length; i += 4) {
      const red = pixels[i];
      const green = pixels[i + 1];
      const blue = pixels[i + 2];
      // the luma algorithm as we discussed above, floor because intensity is a number
      const lightness = Math.floor(red * 0.299 + green * 0.587 + blue * 0.114);
      // all of RGB is set to the calculated intensity value for grayscale
      pixels[i] = pixels[i + 1] = pixels[i + 2] = lightness;
    }
    // and finally now that we have the updated values for grayscale we put it on output
    outputCtx.putImageData(imgData, 0, 0);
  }

  getName() {
    return "grayscale-plugin";
  }

  isSupported() {
    // we're not doing anything complicated, it's supported on all browsers
    return true;
  }

  async init() {} // placeholder, nothing to init

  getPluginType() {
    return HMSVideoPluginType.TRANSFORM; // because we transform the image
  }

  stop() {} // placeholder, nothing to stop
}
```

## Adding a button component to add the plugin

Checkout the full commit [here](https://github.com/triptu/100ms-face-filters/commit/934137bf423a9b4a2256279a2d1b9981167a9ffd).

![Screenshot of the AR filter button component](https://storage.googleapis.com/100ms-cms-prod/cms/7_Yli_Y5_Gdk_494c933d8d/7_Yli_Y5_Gdk_494c933d8d.png) 

Let's write a toggle button component now which will turn the on/off the filter. The component will take in a plugin and button name to display.

```jsx
// also intialise the grayscale plugin for use by the Button's caller
export const grayScalePlugin = new GrayscalePlugin();

export function PluginButton({ plugin, name }) {
  const isPluginAdded = false;
  const togglePluginState = async () => {};

  return (
    <button className="btn" onClick={togglePluginState}>
      {`${isPluginAdded ? "Remove" : "Add"} ${name}`}
    </button>
  );
}
```

We'll use it as below, this is added in the header component in the above commit.
```jsx
<PluginButton plugin={grayScalePlugin} name={"Grayscale"} />
```

Clicking on the button won't work yet though, because we're not adding the plugin to the video track. Let's see how to do that in the next section.


## Making the button functional

Checkout the full commit [in my github](https://github.com/triptu/100ms-face-filters/commit/9f129591907847283185ae1c29d093557da14b90).

 ![Grayscale filter in action](https://storage.googleapis.com/100ms-cms-prod/cms/AR_Filter_Demo_fd344c1eb1/AR_Filter_Demo_fd344c1eb1.gif) 

With some help from the [documentation](https://docs.100ms.live/javascript/v2/plugins/custom-video-plugins#adding-and-removing-plugins), we can make our button component functional using the hooks exposed by the SDK. There are two hooks from the SDK we need to use to implement our toggle function -
1. `useHMSStore` for knowing the current state i.e. whether the plugin is currently part of the video track.
2. `useHMSActions` to get access to the methods for adding and removing the plugin.

```jsx
import {
  selectIsLocalVideoPluginPresent,
  useHMSActions,
  useHMSStore,
} from "@100mslive/react-sdk";

export function PluginButton({ plugin, name }) {
  const isPluginAdded = useHMSStore(
    selectIsLocalVideoPluginPresent(plugin.getName())
  );
  const hmsActions = useHMSActions();

  const togglePluginState = async () => {
    if (!isPluginAdded) {
      await hmsActions.addPluginToVideoTrack(plugin);
    } else {
      await hmsActions.removePluginFromVideoTrack(plugin);
    }
  };

  return (
    <button className="btn" onClick={togglePluginState}>
      {`${isPluginAdded ? "Remove" : "Add"} ${name}`}
    </button>
  );
}
``` 

## Voilà!

And that's it, our button is functional now. Everything works and looks amazing. To recap, we were able to write a grayscale filter from scratch which transforms our video for everyone in the room.

 ![We did it meme](https://storage.googleapis.com/100ms-cms-prod/cms/Theo_Lawrence_We_did_it_Meme_7b9b086b5c/Theo_Lawrence_We_did_it_Meme_7b9b086b5c.gif) 

You can go on from here to have more filters(for e.g. [sepia](https://stackoverflow.com/questions/1061093/how-is-a-sepia-tone-created), saturation, contrast), or experiment with other image processing algorithms to explore the possibilities. Check out [Standford](https://ai.stanford.edu/~syyeung/cvweb/tutorial1.html) and [Towards Data Science](https://towardsdatascience.com/image-processing-and-pixel-manipulation-photo-filters-5d37a2f992fa) articles for some starting points. We'll talk about creating an AR filter in upcoming parts which will build upon the fundamentals learned in this part.

large_video chat with filters

medium_video chat with filters

small_video chat with filters

thumbnail_video chat with filters

video chat with filters

Adding AR Filters in a 100ms Video Call - Part 1

100ms' SDK is customizable to the point where adding custom video chat filters to a live audio-video feed is as easy as following this guide.

100ms team

In 2010, Apple Inc. changed the way we interact with each other online by launching the FaceTime app, since then the legacy product has become part of our social life interactions. Today, this blog will take you through a step-by-step guide on how to build a basic FaceTime clone app using 100ms SDK in iOS.

**[You can also build an audio conferencing app like Clubhouse with 100ms iOS SDK](https://www.100ms.live/blog/building-an-audio-app-like-clubhouse-for-ios)**.

## Prerequisites

- Basic understanding of iOS Development.
- SwiftUI basics.
- Third-party integration.

## Getting Started

- Building UI using SwiftUI framework.
- Adding 100ms package to iOS app using Swift Package Manager.
- Setup 100ms.live free account
- Setting up SDK methods.
- Join a Room.
- Mute/Unmute Audio.
- Switch Camera.
- Stop the Camera.
- Leave a room.

**[Setup your 100ms account now and get 10000 free minutes every month](https://dashboard.100ms.live/register)**.

Eventually, this is how your app will look like:

![Facetime Clone Final Version](https://storage.googleapis.com/100ms-cms-prod/cms/1_7c30ef9b95/1_7c30ef9b95.gif)


The structure of the application will have majorly two main views, one is the ContactListView on which you'll be able to see all your contact and from where you can make the call. The other one is the FacetimeView on which you can see the video of the person you have made calls to and handle all the video controls over there.


## ContactListView

The ContactListView consists of the list of people you can talk to, which means you need a person model first then you need a list to populate the view which will be treated as a list of contacts.

### UserModel

```swift

struct UserModel: Identifiable {
    var id = UUID()
    var image: String
    var name: String
}

```
Now that the UserModel is ready with a unique UUID for the user, you need to create a ContactListCell which will use this UserModel to populate data in the List, that will be created inside ContactListView.

```swift

struct ContactListCell: View {
    var user: UserModel
    var body: some View {
        HStack(spacing: 20) {
            Image(user.image)
                .resizable()
                .aspectRatio(contentMode: .fill)
                .frame(width: 70, height: 70, alignment: .center)
                .clipShape(Circle())
            Text(user.name)
                .bold()
        }
        .padding()
    }
}


```

## ContactListCell

Create ContatactListView that includes a List that reuses the ContactListCell, declares a variable of type [UserModel] called contacts, and uses it in ContactListCell.

```swift

import SwiftUI

struct ContactListView: View {
    
    @State var contacts = [UserModel(image: "girl", name: "Krystal"), UserModel(image: "girl", name: "Krystal"), UserModel(image: "girl", name: "Krystal")]
    
    var body: some View {
        NavigationView {
            List(self.$contacts, rowContent: {
                contact in
                NavigationLink(destination: FacetimeView(), label: {
                    ContactListCell(user: contact.wrappedValue)
                })
            })
                .navigationTitle("Facetime Clone")
        }
    }
}

struct ContactListView_Previews: PreviewProvider {
    static var previews: some View {
        ContactListView()
    }
}


```
![Facetime Clone - Contact List](https://storage.googleapis.com/100ms-cms-prod/cms/2_42ce8ab18a/2_42ce8ab18a.png)

The ContactListView is ready to go, next step is to create the FacetimeView which will work as the main video view where both the current user and the contacted user can have a video call.


## FacetimeView

The **FacetimeView** consists of the view where you can see the person you have called can mute/unmute & end the call by clicking on any of the options available at the bottom of the view. To maintain the mute/unmute and video pause state, declare variables of type Bool which will help to switch between states.

```swift

import SwiftUI

struct FacetimeView: View {
    
    @State var isMuted: Bool = false
    @State var videoIsShowing: Bool = true
    
    var body: some View {
        Spacer()
        videoOptions
            .padding(.bottom, 10)
        
    }
    
    var videoOptions: some View {
        HStack(spacing: 20) {
            Spacer()
            Button {
                
            } label: {
                Image(systemName: videoIsShowing ? "video.fill" : "video.slash.fill")
                    .frame(width: 60, height: 60, alignment: .center)
                    .background(Color.white)
                    .foregroundColor(Color.black)
                    .clipShape(Circle())
            }
            
            Button {
                
            } label: {
                Image(systemName: "phone.down.fill")
                    .frame(width: 60, height: 60, alignment: .center)
                    .background(Color.red)
                    .foregroundColor(Color.white)
                    .clipShape(Circle())
            }
            
            Button {
                
            } label: {
                Image(systemName: isMuted ? "mic.slash.fill" : "mic.fill")
                    .frame(width: 60, height: 60, alignment: .center)
                    .background(Color.white)
                    .foregroundColor(Color.black)
                    .clipShape(Circle())
            }
            Spacer()
        }
    }
    
}

struct FacetimeView_Previews: PreviewProvider {
    static var previews: some View {
        FacetimeView()
    }
}



```

![Facetime Clone Empty Screen](https://storage.googleapis.com/100ms-cms-prod/cms/3_948ca4d312/3_948ca4d312.png)


## Integrate 100ms SDK using SPM

![Adding packages to Xcode - 1](https://storage.googleapis.com/100ms-cms-prod/cms/4_1485e60764/4_1485e60764.png)
![Adding packages to Xcode - 2](https://storage.googleapis.com/100ms-cms-prod/cms/5_322f9f2fa2/5_322f9f2fa2.png)
![Adding packages to Xcode - 3](https://storage.googleapis.com/100ms-cms-prod/cms/6_4889a5c6b4/6_4889a5c6b4.png)


In your 100ms project, go to File -> Add Packages -> Search 100ms git url

Click on Add Package button to complete the package integration, enable Local Network, Camera and Microphone Usage Access to the application by adding Privacy - Local Network Usage Description, Privacy - Camera Usage Description & Privacy - Microphone Usage Description to your project's Info.plist

![Package Integration Screenshot](https://storage.googleapis.com/100ms-cms-prod/cms/7_3888ff5f3f/7_3888ff5f3f.png)

## Create Free Account on 100ms.live

Click on Try for Free button on 100ms.live website and create an account using your Gmail account, after selecting your Gmail account you'll find the option to choose the subdomain for your hosted sessions on the 100ms platform. 

![100ms App Deployment Screenshot - 1](https://storage.googleapis.com/100ms-cms-prod/cms/8_6c98fb4887/8_6c98fb4887.png)
![100ms App Deployment Screenshot - 2](https://storage.googleapis.com/100ms-cms-prod/cms/9_2d2dd5aaef/9_2d2dd5aaef.png)
![100ms App Deployment Screenshot - 3](https://storage.googleapis.com/100ms-cms-prod/cms/10_d2d9ac211d/10_d2d9ac211d.png)


You get the option to choose your hosted sessions subdomain, here let's name it facetime and add more details. The key step is to choose the right template, in this case since the Facetime Clone demo has video conferencing capability, select the Video Conferencing template from the listed options and create the app.

![100ms App Deployment Screenshot - 4](https://storage.googleapis.com/100ms-cms-prod/cms/11_e2eab06691/11_e2eab06691.png)
![100ms App Deployment Screenshot - 5](https://storage.googleapis.com/100ms-cms-prod/cms/12_6a1a5c295d/12_6a1a5c295d.png)

The basic onboarding setup is now complete and you'll be able to find all the options on your dashboard, so click on Go to Dashboard button. On Dashboard you'll find that all the templates, rooms, sessions, etc. you have created so far are available and now all you need to get things working for the Facetime Clone is first, to create a new room under the Rooms tab and use its roomID and second, to get the tokenEndpoint which is available under the Developer tab option. These two key pieces of information will help to connect the facetime application to the room and share the room invite with other peers, who can connect to join the conference. 

![100ms App Deployment Screenshot - 6](https://storage.googleapis.com/100ms-cms-prod/cms/13_b844437b99/13_b844437b99.png)

The Video Conferencing template has two types of roles by default, Guest and Host which have a different set of permissions that can be modified here in this panel. You can also add a new type of role by clicking on the Add a Role button.

![100ms App Deployment Screenshot - 7](https://storage.googleapis.com/100ms-cms-prod/cms/14_18a7bb8a44/14_18a7bb8a44.png)

Under the Rooms tab, you'll find the button to create a new room "Create Room", click on it, and fill in the details of the newly created room.

![100ms App Deployment Screenshot - 8](https://storage.googleapis.com/100ms-cms-prod/cms/15_a6f7d0d887/15_a6f7d0d887.png)

A new room needs details like Room name, description, and region. Here in this example we named it facetime-ios, you can call it whatever you want and the description is Facetime clone. Click on Create Room to complete the process.

![100ms App Deployment Screenshot - 9](https://storage.googleapis.com/100ms-cms-prod/cms/16_40dd0299cd/16_40dd0299cd.png)
![100ms App Deployment Screenshot - 10](https://storage.googleapis.com/100ms-cms-prod/cms/17_02eafadef0/17_02eafadef0.png)


From the newly created room, grab the roomID and from the Developer's tab grab the tokenEndpoint.


Add this TokenProvider swift file to your project that helps to set up the communication with the 100ms server and add the grabbed roomId & tokenId respectively, which will be used to authenticate the connection and get the stream going.

## Token Provider


```swift

enum TokenProviderError: Error {
	case invalidURL
	case invalidRequestPayload
	case invalidResponseFormat
	case tokenMissing
	case networkError(reason: String?)
}

class TokenProvider {
	struct Constants {
		static let tokenEndpoint = "https://prod-in.100ms.live/hmsapi/facetime.app.100ms.live/"
		static let roomID = "61eea74dd85bf05ee4e175b2"
	}
	
	func getToken(for userID: String, role: String, completion: @escaping (String?, Error?) -> Void) {
		let tokenRequest: URLRequest
		
		do {
			tokenRequest = try request(for: userID, role: role)
		} catch {
			completion(nil, error)
			return
		}

		URLSession.shared.dataTask(with: tokenRequest) { [weak self] data, response, error in
			DispatchQueue.main.async {
				self?.process(data: data, error: error, completion: completion)
			}
		}.resume()
	}

	private func request(for userID: String, role: String) throws -> URLRequest {
		guard let url = URL(string: Constants.tokenEndpoint + "api/token") else {
			throw TokenProviderError.invalidURL
		}

		var request = URLRequest(url: url)
		request.httpMethod = "POST"

		let body = ["room_id": Constants.roomID,
					"user_id": userID,
					"role": role]

		request.httpBody = try JSONSerialization.data(withJSONObject: body, options: .prettyPrinted)

		request.addValue("application/json", forHTTPHeaderField: "Content-Type")
		request.addValue("application/json", forHTTPHeaderField: "Accept")

		return request
	}
	
	private func process(data: Data?, error: Error?, completion: (String?, Error?) -> Void) {
		guard error == nil, let data = data else {
			completion(nil, TokenProviderError.networkError(reason: error?.localizedDescription))
			return
		}

		let token: String
		do {
			token = try parseToken(from: data)
		} catch {
			completion(nil, error)
			return
		}
		
		completion(token, nil)
	}

	private func parseToken(from data: Data) throws -> String {
		let json = try JSONSerialization.jsonObject(with: data)
		
		guard let responseDictionary = json as? [String: Any]  else {
			throw TokenProviderError.invalidResponseFormat
		}
		
		guard let token = responseDictionary["token"] as? String else {
			throw TokenProviderError.tokenMissing
		}

		return token
	}
}


```


In order to listen back from the 100ms server, import HMSSDK in the FacetimeView file and create an extension of Facetime View, and declare a ViewModel that confirms to HMSUpdateListener protocol. Add all the required methods of HMSUpdateListener protocol to the view model and use on(track: HMSTrack, update: HMSTrackUpdate, for peer: HMSPeer) to listen to any change in peerTrack.


```swift

extension FacetimeView {
	class ViewModel: ObservableObject, HMSUpdateListener {
		
		@Published var addVideoView: ((_ videoView: HMSVideoTrack) -> ())?
		@Published var removeVideoView: ((_ videoView: HMSVideoTrack) -> ())?
		
		func on(join room: HMSRoom) {
			
		}
		
		func on(room: HMSRoom, update: HMSRoomUpdate) {
			
		}
		
		func on(peer: HMSPeer, update: HMSPeerUpdate) {
			
		}
		
		func on(track: HMSTrack, update: HMSTrackUpdate, for peer: HMSPeer) {
			switch update {
			case .trackAdded:
				if let videoTrack = track as? HMSVideoTrack {
					addVideoView?(videoTrack)
				}
			case .trackRemoved:
				if let videoTrack = track as? HMSVideoTrack {
					removeVideoView?(videoTrack)
				}
			default:
				break
			}
		}
		
		func on(error: HMSError) {
			
		}
		
		func on(message: HMSMessage) {
			
		}
		
		func on(updated speakers: [HMSSpeaker]) {
			
		}
		
		func onReconnecting() {
			
		}
		
		func onReconnected() {
			
		}
		
	}
}


```


To set up video frames for the contacted person and current user in FacetimeView, create a VideoView using UIViewRepresentable


## Video View

```swift

import Foundation
import SwiftUI
import HMSSDK

struct VideoView: UIViewRepresentable {
	var track: HMSVideoTrack

	func makeUIView(context: Context) -> HMSVideoView {
		let videoView = HMSVideoView()
		videoView.setVideoTrack(track)
		videoView.backgroundColor = UIColor.black
		return videoView
	}
	
	func updateUIView(_ videoView: HMSVideoView, context: Context) {
		videoView.setVideoTrack(track)
	}
}



```

The Video View is ready to use in FacetimeView, but before adding it to the View first import HMSSDK and declare the hmsSdk object, token provider object, viewModel object, and other tracks objects.


```swift

import Foundation
import SwiftUI
import HMSSDK

struct FacetimeView: View {
	var hmsSDK = HMSSDK.build()
	@State var localTrack = HMSVideoTrack()
	@State var friendTrack = HMSVideoTrack()
	let tokenProvider = TokenProvider()
	@StateObject var viewModel: ViewModel
	@Environment(\.presentationMode) var presentationMode
	
	@State var isMuted: Bool = false
	@State var videoIsShowing: Bool = true
	

```

All the required variables have been declared, so let's go back to the FacetimeView file and inside the body, add a video view that requires a track in its initializer.


```swift

	var body: some View {
		ZStack(alignment: .bottomTrailing) {
			VideoView(track: friendTrack)
				.edgesIgnoringSafeArea(.all)
			if !videoIsShowing {
				ZStack {
					Color.gray
					Image(systemName: "video.slash.fill")
				}
				.frame(width: 150, height: 250, alignment: .center)
				.padding()
			} else {
				VideoView(track: localTrack)
					.frame(width: 150, height: 250, alignment: .center)
					.cornerRadius(10)
					.shadow(radius: 20)
					.padding(.bottom, 80)
					.padding()
			}
			videoOptions
				.padding(.bottom, 10)
		}
	}


```

## Join Room & Listen To Peer Track

In the joinRoom() method, use the newly created roomID in the token provider's getToken() call. The listen() method listens to the change in peer track from the 100ms server, based on which you can add or remove the video view for the peer connected in the room. 

```swift

	func listen() {
		self.viewModel.addVideoView = {
			track in
			hmsSDK.localPeer?.localAudioTrack()?.setMute(false)
			hmsSDK.localPeer?.localVideoTrack()?.startCapturing()
			self.localTrack = hmsSDK.localPeer!.videoTrack!
			self.friendTrack = track
		}
		
		self.viewModel.removeVideoView = {
			track in
			self.friendTrack = track
		}
	}
	
	
	func joinRoom() {
		tokenProvider.getToken(for: "61eea74dd85bf05ee4e175b2", role: "host", completion: {
			token, error in
			let config = HMSConfig(authToken: token!)
			hmsSDK.join(config: config, delegate: self.viewModel)
		})
	}



```


## Create & Add Video Control Event Methods

There are three actionable events that can take place in our Facetime Clone namely, switchCamera(), muteMic(), stopCamera(), and endRoom(). After declaring these methods, call them in respective button action built inside videoOptions object of type some View.


```swift

	
	func switchCamera() {
		self.hmsSDK.localPeer?.localVideoTrack()?.switchCamera()
	}
	
	func muteMic() {
		self.hmsSDK.localPeer?.localAudioTrack()?.setMute(isMuted)
		isMuted.toggle()
	}
	
	func stopCamera() {
		if self.videoIsShowing {
			self.hmsSDK.localPeer?.localVideoTrack()?.stopCapturing()
			self.videoIsShowing = false
		} else {
			self.hmsSDK.localPeer?.localVideoTrack()?.startCapturing()
			self.videoIsShowing = true
		}
	}
	
	func endRoom() {
		hmsSDK.endRoom(lock: false, reason: "Meeting has ended") { didEnd, error in
			if didEnd {
				self.presentationMode.wrappedValue.dismiss()
			} else {
				
			}
		}
	}
	
	///////////////////////////////////////////////////

	var videoOptions: some View {
		HStack(spacing: 20) {
			Spacer()
			Button {
				stopCamera()
			} label: {
				Image(systemName: videoIsShowing ? "video.fill" : "video.slash.fill")
					.frame(width: 60, height: 60, alignment: .center)
					.background(Color.white)
					.foregroundColor(Color.black)
					.clipShape(Circle())
			}
			
			Button {
				endRoom()
			} label: {
				Image(systemName: "phone.down.fill")
					.frame(width: 60, height: 60, alignment: .center)
					.background(Color.red)
					.foregroundColor(Color.white)
					.clipShape(Circle())
			}
			
			Button {
				muteMic()
			} label: {
				Image(systemName: isMuted ? "mic.slash.fill" : "mic.fill")
					.frame(width: 60, height: 60, alignment: .center)
					.background(Color.white)
					.foregroundColor(Color.black)
					.clipShape(Circle())
			}
			Spacer()
		}
	}
	
	

```


## Final Method Calls

In FacetimeView, add .onAppear modifier and call joinRoom() & listen() methods to it.


```swift


		.onAppear(perform: {
			joinRoom()
			listen()
		})		



```

Build and run the Xcode project, and share the newly created room invite with any of your peers to join the room. You'll be able to have a video call with your peer and will be able to manage the controls like muting the call, stopping the camera, switching your camera, and ending the call. If you wish to experiment more then do play with the role types and their permissions to build a next-level application that you can show the world. Please find the project link below to take reference;


Project Github link : [Facetime Clone in iOS ](https://github.com/iosmayank26/FacetimeClone).

large_Blog - 3.png

medium_Blog - 3.png

small_Blog - 3.png

thumbnail_Blog - 3.png

Facetime Clone | Cover Image

Building a FaceTime app for iOS using 100ms

In this article, we will go through the process of building a Clubhouse clone with 100ms and Svelte. Clubhouse is a popular app that enables people to speak together in audio rooms over the internet.

## What is Svelte?

Svelte is a new framework that institutes a whole new way for frameworks to interact with the DOM. It doesn't use a VDOM but surgically updates the DOM based on what you want to display. 

**We also have step-by-step guides to build a Clubhouse like app with different technologies**
- [Clubhouse clone with 100ms iOS SDK](https://www.100ms.live/blog/building-an-audio-app-like-clubhouse-for-ios)
- [Clubhouse clone with 100ms React SDK](https://www.100ms.live/blog/clubhouse-clone-in-react)
- [Clubhouse clone with 100ms Flutter SDK](https://www.100ms.live/blog/building-clubhouse-clone-using-100ms-in-flutter)
- [Clubhouse clone with 100ms Javascript SDK](https://www.100ms.live/blog/clubhouse-clone-with-javascript)

It’s also lightweight and, thus, faster because it doesn't ship svelte to the front end. Instead, it ships the code that performs the update.

## What is 100ms?

100ms provides video conferencing infrastructure designed to help businesses build powerful video applications in hours. Its SDK is tailored to suit numerous use cases like game streaming, virtual events, audio rooms, classrooms, and much more. It abstracts the complexities involved in building these apps and reduces development time drastically. 

To top it all off, 100ms has been created and is managed by the team who created live infrastructure at Disney and Facebook (Meta).

## Prerequisites

- A fair understanding of Javascript
- A 100ms account; [if you don't have one, you can create your account here](https://dashboard.100ms.live/register)

## Explanation of 100ms terms

There are a couple of terms used by 100ms that we need to get familiar with to understand this article:


- **Room:** A room is the basic object that 100ms SDKs return on a successful connection. This contains references to peers, tracks, and everything you need to render a live audio/visual app.
- **Peer:** A peer is the object returned by 100ms SDKs that contains all information about a user: name, role, video track, etc.
- **Track:** A track represents either the audio or video that a peer is publishing.
- **Role:** A role defines who a peer can see/hear, the quality at which they publish their video, whether they have permissions to publish video/screen share, mute someone, and/or change someone's role.

[You can read about the other terms here](https://www.100ms.live/docs/javascript/v2/foundation/basics)

## Steps around 100ms

- Setting up 100ms
- Creating an account
- Creating Roles & Rooms

## Steps around setting up the Frontend

- Add 100ms to the Svelte app
- Join a Room
- Leave a Room
- Mute/Unmute a Peer
- Chat in a Room

## Setting up 100ms

- Log in to your 100ms account. On the dashboard, click the **'Add a New App'** Button. 
 
![100ms Dashboard Screenshot - 1](https://storage.googleapis.com/100ms-cms-dev/cms/1_2522574aa2/1_2522574aa2.png)

- Choose a subdomain for your app.

![100ms Dashboard Screenshot - 2](https://storage.googleapis.com/100ms-cms-dev/cms/2_d58d4e2863/2_d58d4e2863.png)

- Choose the template for the app. Since we're building a Clubhouse Clone, 'Audio Rooms' will give us the right setup out-of-the-box. Select that option.

![100ms Dashboard Screenshot - 3](https://storage.googleapis.com/100ms-cms-dev/cms/3_1943c9c41b/3_1943c9c41b.png)

- You should then see this appear on your screen. 

![100ms Dashboard Screenshot - 4](https://storage.googleapis.com/100ms-cms-dev/cms/4_324e21b67f/4_324e21b67f.png)

Click on any of the **Join as** buttons to test out the platform.

To save time, head to the Developer tab of the dashboard, copy the token endpoint and store it in a safe place. Additionally, head to the Rooms tab, and store the Room ID of the room we just created. 

## Setting up the Frontend

To get started, clone this [starter pack](https://github.com/dro-1/clubhouse-clone). It contains the main setup needed for the app, like SCSS and page routing as well as its components. After cloning, run


```bash
yarn
```

to install all dependencies of the starter pack. 

Run


```bash
yarn dev
```

to start the project. You should see the following:

![Join Room Screen](https://storage.googleapis.com/100ms-cms-dev/cms/5_f78efa352a/5_f78efa352a.png)

Under `src/services/hms.js`, we have set up the basic 100ms functions. These functions enable us to connect our components to 100ms.

Head into the `App.svelte` file in `src` and replace its content with:


```js
<script>
  import router from "page";
  import Home from "./pages/home.svelte";
  import Room from "./pages/room.svelte";
//NEW LINE HERE
  import { onMount } from "svelte";
  import { hmsStore } from "./services/hms";
  import { selectIsConnectedToRoom } from "@100mslive/hms-video-store";
//NEW LINE ENDS

  let page;

  router("/", () => (page = Home));

  router("/room", () => (page = Room));

  router.start();
//NEW LINE HERE
  const onRoomStateChange = (connected) => {
    if (connected) router.redirect("/room");
    else router.redirect("/");
  };

  onMount(async () => {
    hmsStore.subscribe(onRoomStateChange, selectIsConnectedToRoom);
  });
//NEW LINE ENDS
</script>

<svelte:component this={page} />

```

Going from the top, 3 new variables are imported:

- **onMount:** This is a function by Svelte that runs after a component mounts (like `componentDidMount` in React). You mainly use it to subscribe to listeners or make requests to API endpoints.
- **hmsStore:** This contains the complete state of the room at any given time. It includes participant details, messages, and track states.
- **selectIsConnectedToRoom:** The 100ms package provides a number of selectors that we can use to extract information from the store. In this case, we're extracting a `boolean` value that tells you if you're connected to a room or not. 

[You can read about other selectors here.
](https://www.100ms.live/docs/api-reference/javascript/v2/home/content#functions)

In the **onMount** function, we set a listener that calls **onRoomStateChange** whenever the connection state changes. The **onRoomStateChange** reacts to this by redirecting you to the appropriate page based on its input.

Head to the `home.svelte` file and replace its contents with:


```js
<script>
  import { hmsActions } from "./../services/hms";
  import { getToken } from "./../utils/utils";
  let userName = "";
  let role = "";

  const submitForm = async () => {
    if (!userName || !role) return;
    try {
      const authToken = await getToken(role, userName);
      const config = {
        userName,
        authToken,
        settings: {
          isAudioMuted: true,
          isVideoMuted: false,
        },
        rememberDeviceSelection: true,
      };
      hmsActions.join(config);
    } catch (error) {
      console.log("Token API Error", error);
    }
  };
</script>

<main>
  <form>
    <header>Join Room</header>
    <label for="username">
      Username
      <input
        bind:value={userName}
        id="username"
        type="text"
        placeholder="Username"
      />
    </label>
    <label>
      Role
      <select bind:value={role} name="role">
        <option value="speaker">Speaker</option>
        <option value="listener">Listener</option>
        <option value="moderator">Moderator</option>
      </select>
    </label>
    <button on:click|preventDefault={submitForm}> Join </button>
  </form>
</main>

```

Here we import:

- **hmsActions:** This is used to perform any action such as joining, muting, and sending a message.
- **getToken:** It helps us generate a token that we need to join any 100ms room.

We also have a function, `submitForm`, that couples the `config` variable and adds us to the room using `hmsAction`.  

In the markup, you'll notice we have `bind`: in the input. This is called a directive and Svelte gives us numerous directives to make our lives easier. 

The `bind:value` directive links the value of the input to the specified variable. 

In your case, this variable is the `username` variable. You also use it in the `select` element. The `on:click` directive, on the other hand, attaches the specified function as the handler to the click event on that button. 

Svelte also gives us modifiers like `|preventDefault` that customize the directive to our taste. In our case, `|preventDefault` calls the `event.preventDefault` function before running the handler.

You'll also notice that we haven't implemented the `getToken` function, so let's get to it. Create a `utils.js` file in the directory `src/utils` and paste the following:


```
const TOKEN_ENDPOINT = process.env.TOKEN_ENDPOINT;
const ROOM_ID = process.env.ROOM_ID;

export const getToken = async (userRole, userName) => {
  const role = userRole.toLowerCase();
  const user_id = userName;
  const room_id = ROOM_ID;
  let payload = {
    user_id,
    role,
    room_id,
  };
  let url = `${TOKEN_ENDPOINT}api/token`;
  const response = await fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify(payload),
  });
  let resp = await response.json();
  return resp.token;
};

```

First, you extract the environment variables from `process.env`. Then, make a call to the endpoint provided to you by 100ms. This endpoint responds with the needed token.

But we haven't set up our environmental variables. We can do this easily by installing some packages. Run

```bash
yarn -D dotenv @rollup/plugin-replace
```

to get them installed. Then open the **rollup.config.js** in the root of the folder and paste the following:


```js
//NEW LINE STARTS
import replace from "@rollup/plugin-replace";
import { config } from "dotenv";
//NEW LINE ENDS

const production = !process.env.ROLLUP_WATCH;

//CODE OMITTED FOR BREVITY

export default {
  input: "src/main.js",
  output: {
    sourcemap: true,
    format: "iife",
    name: "app",
    file: "public/build/bundle.js",
  },
  plugins: [
      //NEW LINE STARTS
    replace({
      "process.env.NODE_ENV": JSON.stringify("production"),
      "process.env.TOKEN_ENDPOINT": JSON.stringify(
        config().parsed?.TOKEN_ENDPOINT || process.env.TOKEN_ENDPOINT
      ),
      "process.env.ROOM_ID": JSON.stringify(
        config().parsed?.ROOM_ID || process.env.ROOM_ID
      ),
    }),
      //NEW LINE ENDS
    svelte({
      preprocess: preprocess(),
      compilerOptions: {
        dev: !production,
      },
    }),

```

Our `getToken` function should be up and running now.
Next, replace the code in `room.svelte` with the following:


```js
<script>
  import page from "page";
  import Peer from "./../components/peer.svelte";
  import { hmsActions, hmsStore } from "./../services/hms";
  import { selectPeers,selectLocalPeerRole,
    selectIsLocalAudioEnabled, } from "@100mslive/hms-video-store";
  import { onMount, onDestroy } from "svelte";
  import { PeerStore } from "./../stores";

  let peers = [];
  let localPeerRole = "";
  let audioEnabled = null;

  const handlePeers = (iPeers) => {
    let res = hmsStore.getState(selectLocalPeerRole);
    localPeerRole = res ? res.name : "";
    audioEnabled = hmsStore.getState(selectIsLocalAudioEnabled);
    PeerStore.set(iPeers);
  };

  const handleMute = async () => {
    await hmsActions.setLocalAudioEnabled(!audioEnabled);
    audioEnabled = hmsStore.getState(selectIsLocalAudioEnabled);
  };

  onMount(async () => {
    hmsStore.subscribe(handlePeers, selectPeers);
  });

  const leaveRoom = () => hmsActions.leave();

  onDestroy(leaveRoom);

  $: peers = $PeerStore;
</script>

<main>
  <h1>Welcome To The Room</h1>

  <section class="peers">
    {#each peers as peer (peer.id)}
      <Peer {localPeerRole} {peer} />
    {/each}
  </section>
  <div class="buttons">
    {#if localPeerRole != "listener"}
      <button on:click={handleMute} class="mute"
        >{audioEnabled ? "Mute" : "Unmute"}</button
      >
    {/if}
    <button on:click={leaveRoom} class="leave">Leave Room</button>
  </div>
</main>

```

This page houses the most important features of our app. First, we import the required variables. Some of these are:

- **onDestroy:** This function is similar to `onMount` except it is called immediately before the component is unmounted.
- **PeerStore:** This is a store that would keep track of the current peers in the room.

The `handlePeers` function does three things:

- It stores the role of the local peer in the `localPeerRole` variable.
- It sets the audio state of the local peer in the `audioEnabled` variable.
- It stores the current peers in the room within the `PeerStore` store.

The `handleMute` function simply toggles the audio state of the local peer. A `leaveRoom` is called when the component is to be unmounted or when the `Leave Room` button is clicked.

The `$:` syntax helps us create reactive statements. These statements run immediately before the component updates, whenever the values that they depend on have changed.

We have 2 new syntaxes in our markup:

- `{#each peers as peer (peer.id)}`:This helps us map out each peer in the `peers` array using the peer's ID as a key.
- `{#if localPeerRole != "listener"}`:This renders the component between the `if` block, if the condition is true. Therefore, it renders the `Mute` button if the local peer is not a listener.

On to the last component, **peer.svelte**. For the last time, copy the code below into the file:


```js
<script>
  import {
    selectIsPeerAudioEnabled,
  } from "@100mslive/hms-video-store";
  import { onMount } from "svelte";
  import { hmsActions, hmsStore } from "../services/hms";
  export let peer = null;
  export let localPeerRole = "";

  let isContextOpen = false;
  let firstCharInName = "";
  let isPeerMuted = false;

  const togglePeerAudio = () => {
    hmsActions.setRemoteTrackEnabled(peer.audioTrack, isPeerMuted);
  };

  const changeRole = (role) => {
    hmsActions.changeRole(peer.id, role, true);
  };

  onMount(async () => {
    hmsStore.subscribe((isPeerAudioEnabled) => {
      isPeerMuted = !isPeerAudioEnabled;
    }, selectIsPeerAudioEnabled(peer?.id));
  });

  $: firstCharInName = peer ? peer.name.split(" ")[0][0].toUpperCase() : "";
</script>

<div class="peer">
  <div on:click={() => (isContextOpen = !isContextOpen)} class="content">
    <div class="image">
      <p>{firstCharInName}</p>
    </div>
    <p>{peer ? peer.name : ""}{peer && peer.isLocal ? " (You)" : ""}</p>
  </div>
  {#if localPeerRole == "moderator" && !peer.isLocal}
    <div class="context" class:open={isContextOpen}>
      <button on:click={togglePeerAudio}
        >{isPeerMuted ? "Unmute" : "Mute"}</button
      >
      <button on:click={() => changeRole("speaker")}>Make Speaker</button>
      <button on:click={() => changeRole("listener")}>Make Listener</button>
    </div>
  {/if}
</div>

```

Once again, all needed variables are imported. You are expecting 2 props: **peer** and **localPeerRole**. 

2 functions are declared: **togglePeerAudio** and **changeRole**. They do exactly what their names describe. In the **onMount** function, a handler is added to update the **isPeerMuted **state of a peer.

Each peer component has a context menu that has options for muting the peer or changing their role. But this menu is only available to moderators as only they should have such permissions.

At this point, we are done. 

You can run

```bash
yarn dev
```

in the terminal to see the application. 

![Clubhouse Clone Demo](https://storage.googleapis.com/100ms-cms-dev/cms/6_f5c3d068fe/6_f5c3d068fe.gif)

- [You can see the completed application here.](https://dro-ch-clone.netlify.app/)
- [You can see the GitHub code for the application here.](https://github.com/dro-1/clubhouse-clone-fin)


large_Blog - 1.png

medium_Blog - 1.png

small_Blog - 1.png

thumbnail_Blog - 1.png

Blog - 1.png

Building a Clubhouse Clone with Svelte and 100ms

Deepankar.jpeg

Deepankar Bhade

thumbnail_Hacktober.png

Hacktober.png

100ms Blog

Hacktoberfest is a month-long celebration of open-source software by DigitalOcean that encourages participation in giving back to the open-source community.

Hacktoberfest 2021 with 100ms

Hacktoberfest 2021 with 100ms

What is 100ms ?

What you can contribute

Issues

What's in it for you?