October 29, 2021
The mighty SurfaceViewRenderer! It's the very end of a long WebRTC chain that gets video from other people to your screen.
Learn how to build Twitter Spaces with 100ms iOS SDK
It can also be a bit difficult to use if it's your first time around. Especially in a dynamic android app with a lots of views flicking in and out.
Are you a Flutter developer? checkout our comprehensive guide on Flutter WebRTC
Let's a take a tour through the many ways it can go wrong and what you should know to keep your video conference calls running smoothly.
Here's the lifecycle of a SurfaceViewRenderer:
Once released the SurfaceViewRenderer can be inited again and the cycle renews. Now let's look at what happens when any one of these is missed or not called with the right parameters.
This one is actually mentioned in the source code of SurfaceViewRenderer, when you call SurfaceViewRenderer.init(context) it must only be with a shared EGL context.
What does that mean? That you call org.webrtc.EglBase.create() once in something like a singleton and then always initialise all the SurfaceViewRenderers with that saved context's eglBaseContext.
If you call create() for each SurfaceView, you'd end up with no errors and no video playing at all. It's easy to get stuck here and wonder why SurfaceViewRenderer is not showing any video when seemingly nothing went wrong.
If you don't release after you've removed a video sink, you're going to get the same result as if you initialised twice.
java.lang.IllegalStateException: videoSurfaceViewAlready initialized at org.webrtc.EglRenderer.init(EglRenderer.java:212)
So you've just got to release at the right time.
The view just never shows up ¯\_(ツ)_/¯
Behind the scenes a render thread was never created and the loop to draw wasn't kicked off so, of course, nothing would show up.
Calling addSink on the same SurfaceViewRenderer is a pretty fun bug. You won't realise anything's wrong if you just added all the the videos for people twice, until the order of the videos is changed.
Then you'll see flicker as two different video sources try to play onto the same surface. Both video sources will take race condition turns rendering.
Oh, and you'll keep downloading the video from the source peer if you only called removeSink once when disposing of the view.
Here's an image of what the two video streams look like separately and what happens when addSink for both was called on the same SurfaceViewRenderer.
Without initialising them, you can create a heck of a lot. But there's a limit to how many different initialised SurfaceViewRenderers you can prepare at a time.
The limit varies from device to device, when I'd tried on one modestly powered Nokia 3.4 phone it got up to 30 contexts initialised until it failed with:
E/AndroidRuntime: FATAL EXCEPTION: mainProcess: com.example.myvideocallapp, PID: 532java.lang.RuntimeException: java.lang.RuntimeException: Failed to create EGL context: 0x3003at org.webrtc.EglBase14Impl.createEglContext(EglBase14Impl.java:282)at org.webrtc.EglBase14Impl.(EglBase14Impl.java:78)at org.webrtc.EglBase.createEgl14(EglBase.java:215)at org.webrtc.EglBase.create(EglBase.java:158)
On others, like an Oppo F19 it gets beyond 150 and keep going. Considering that that's the number of videos you're displaying simultaneously, you may not need to get anywhere near that much.
Now let's look at something even tricker than managing a single or a handful of a static number of SurfaceViewRenderers.
As a refresher, the RecyclerView is an essential View on Android that lets you avoid creating expensive view objects when scrolling through a large list.
What's even more expensive than a typical view object? The SurfaceViewRenderer of course.
Let's think about what a RecyclerView does in android.
How does that compare to the lifecycle of a SurfaceView?
This poses a natural problem because there are multiple things that could cause the view to be changed.
To further complicate matters, someone's video may not have been on at first, so you may just show the person's initials, but later they do turn it on, so now the SurfaceViewRender has to be initialized.
If you're to Android, rebinding views is straightforward, you'd just do that in the AdapterView's bind method but a slightly unconventional approach is required here because anyone's video can go on or off at any time.
So the big question becomes, when do you initialize the video?
You'd quickly see a 'bind' method is insufficient.
What we need here are some lesser known adapter methods:
Called when a new view is added to the window or a cached one is restored.
Called when a view is removed, we don't know if it's going into cache or out permanently.
Take a look at the PeerViewHolder and the PeerAdapter to see how all the disparate cases are handled. This is the simplest example of how you could manage the tricky SurfaceViewRenderer in an Android RecyclerView.
This is part of the "hello world" demonstration of how the 100ms SDK can be used.
To look into more advanced use cases, take a look at our fully featured sample app.
Like what you’re reading?
Get Audio/video engineering tips straight into your inbox