Add support for simulcast streams. #5535

fancycode · 2021-04-29T08:18:59Z

Initial version for some testing.

Stream selection:

in grid mode, the medium quality will be subscribed for all videos
in normal mode, the high quality will be subscribed for the selected video
in normal mode, the low quality will be subscribed for all other videos

This has some nice bandwidth savings (receiving side) and with that also CPU usage is reduced:

Should be backwards compatible for mobile clients which will receive the highest layer (the same as before).

Requires strukturag/nextcloud-spreed-signaling#104

Open issues:

Test with more browsers (I tested with Chrome and Firefox).
Test with screensharing streams.
Test with mobile clients.
Adjust selection of substream / temporal layer.
Check if SentVideoQualityThrottler can be disabled if simulcast is used.
Select lower quality for slow clients (e.g. also use "low" for grid mode / selected video).

Fixes #5509

nickvergessen · 2021-06-24T09:02:27Z

src/utils/webrtc/simplewebrtc/peer.js

+function mungeSdpForSimulcasting(sdp) {
+	// Let's munge the SDP to add the attributes for enabling simulcasting
+	// (based on https://gist.github.com/ggarber/a19b4c33510028b9c657)
+	var lines = sdp.split("\r\n");


We have this lib included:
https://www.npmjs.com/package/sdp-transform

Should make your life easier?

Maybe also https://www.npmjs.com/package/@jitsi/sdp-simulcast could be used. For the initial proof-of-concept I was just copying the existing code from Janus.

danxuliu

Sorry for the delay.

As far as I can tell in general simulcast works with video streams (see below for screen shares), also when the streams are disabled or when changing between devices. I have also tested with a signaling server version that does not support simulcast and everything works as expected (that is, the high quality is used).

There was an issue with how the simulcast quality was set; selected participants are not the same as promoted participants (selected are explicitly set by the user, promoted are automatically set based on who is speaking), so in speaker mode the current speaker was shown with low quality.

Independently of that, the video quality is directly related to the call view, so I moved the watchers from the WebRTC code to the CallView component and added methods in the model to select a specific quality. I also fixed the issue with the promoted participants by taking that into account in adjustSimulcastQualityForParticipant. It may be possible to improve it a bit further by setting high quality only to the selected participant rather than to the selected and promoted participants when one is selected, but I have not checked that.

Besides that, sometimes the following message appears in Janus logs: No packet received on substream 2 for a while, falling back to 1. This could be related to how the bandwidth is allocated between the different streams (it seems that there are issues in that regard), but I have no idea.

Regarding the open issues:

Test with screensharing streams.

Simulcast with screensharing streams does not seem to work. But I have no idea why, because they should work just like regular videos 🤷

The received video resolution does not change when the simulcast stream is changed, neither in Firefox nor in Chromium.

Moreover, in the case of Chromium the following warning is shown in Janus logs (for the sender): [WARN] Incoming RTCP SR SSRC (1192582037) does not match the expected one (4197326786) video=1. I have checked the SDPs for a normal video and a screen share and, besides the IDs, the only difference is in the data channel related fields (because the screen sharing peer has no data channels). But if data channels are disabled in the normal video peer the only differences in the SDP of a normal video and a screen share are the IDs, yet the error is shown only for screen shares.

Adjust selection of substream / temporal layer.

What is missing here? Could you elaborate? :-)

Check if SentVideoQualityThrottler can be disabled if simulcast is used.

I would keep the quality throttler even if simulcast is used, as it should help to keep the CPU usage low due to not sending a stream with a high bitrate when not needed, and in calls with a high number of participants I think that it is fine that a participant who is not speaking is not sent with high quality even if selected. Having said that I have not actually checked how the low quality version of a throttled quality stream looks...

Select lower quality for slow clients (e.g. also use "low" for grid mode / selected video).

If this is meant to be done by analyzing the stats reported by the received peer connections I would leave this for a later stage. The stats reported by the received peer connections are updated at irregular intervals, so it is difficult to calculate things like number of packets per second or the ratio of lost packets. The approach used in #5885 could be extended to handle received peer connections too, but I have not checked if that would actually work or not.

Of course if this is going to be done in a different way (or if I am missing something about the stats) it would be a nice addition :-)

Some additional issues:

Simulcast Firefox bitrates should be based on the maximum bandwidth allowed by the HPB?
Use rid instead of SDP munging for recent Chromium versions, as that will make possible to configure too the bitrates (see https://www.meetecho.com/blog/simulcast-janus-ssrc/)

fancycode · 2021-07-01T13:31:35Z

Thanks for the first review! The API for substream selection in the signaling server was released with 0.3.0 today.

Simulcast with screensharing streams does not seem to work. But I have no idea why, because they should work just like regular videos 🤷

There might be issues in the browser implementations with screensharing streams. A quick search found https://bugs.chromium.org/p/webrtc/issues/detail?id=11310

Adjust selection of substream / temporal layer.

What is missing here? Could you elaborate? :-)

The selection which substream is subscribed for which video could probably be improved. I wasn't sure if the simple selection with grid, promoted, other was sufficient.

Imho we should focus on getting basic simulcast working in a first step and then improve based on feedback from real-world usage / feedback in follow-up PRs.

Simulcast Firefox bitrates should be based on the maximum bandwidth allowed by the HPB?

I would move this also to a follow-up PR if necessary. Bandwidth configurations on the HPB side might not be that easy if proxies are used which could have their own bw limits.

danxuliu · 2021-07-01T15:31:20Z

There might be issues in the browser implementations with screensharing streams.

Given that it does not seem to work neither with Firefox nor Chromium and that with Chromium it also spams Janus logs I would remove the calls to setSimulcastScreenQuality() for now and use it only for normal videos.

The selection which substream is subscribed for which video could probably be improved. I wasn't sure if the simple selection with grid, promoted, other was sufficient.

Maybe we could use something like this:

If a participant is selected:
- Selected participant -> High quality
- Any other participant fallback to the grid quality
If no participant is selected but a participant is promoted and the call view is in speaker mode:
- Promoted participant -> High quality
- Any other participant fallback to the grid quality
If the call view is in grid mode, or for any other participant besides the promoted and selected ones in speaker mode (they will be shown also in a grid below the main speaker):
- Set quality based on the size of the cells of the grid

But this can be done in a follow up.

Imho we should focus on getting basic simulcast working in a first step and then improve based on feedback from real-world usage / feedback in follow-up PRs.

Fine by me 👍

I would move this also to a follow-up PR if necessary. Bandwidth configurations on the HPB side might not be that easy if proxies are used which could have their own bw limits.

👍

Signed-off-by: Joachim Bauch <[email protected]>

Simulcast does not currently work for screenshares neither in Firefox nor in Chromium. Moreover, if a screen is shared in Chromium and simulcast is enabled for the peer connection of the screen Janus logs will be spammed with "[WARN] Incoming RTCP SR SSRC (XXXXXXXXX) does not match the expected one (YYYYYYYYYY) video=1" messages. Therefore, for now, simulcast is disabled for screen peers. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

danxuliu

As mentioned, I have disabled simulcast for screen shares. Other than that in general it works, so fine to address the pending issues in follow up pull requests 👍

nickvergessen · 2021-07-01T17:18:35Z

/backport to stable22

fancycode requested a review from danxuliu April 29, 2021 08:18

nickvergessen added this to the 💖 Next Major (22) milestone Apr 29, 2021

nickvergessen added 3. to review feature: WebRTC 🚡 WebRTC connection between browsers and/or mobile clients labels Apr 29, 2021

fancycode mentioned this pull request Apr 29, 2021

Mixing all participant’s streams into one | MCU for larger conferences even with low bandwith participants #4467

Open

fancycode mentioned this pull request May 10, 2021

Hardware Requirements for a complete high performance backend strukturag/nextcloud-spreed-signaling#18

Open

nickvergessen reviewed Jun 24, 2021

View reviewed changes

nickvergessen modified the milestones: 💙 Next Major (22), 💖 Next Major (23) Jun 25, 2021

danxuliu force-pushed the simulcast-support branch from fabc8b1 to c31bd29 Compare July 1, 2021 05:18

danxuliu reviewed Jul 1, 2021

View reviewed changes

Add support for simulcast streams.

262023c

Signed-off-by: Joachim Bauch <[email protected]>

danxuliu force-pushed the simulcast-support branch from c31bd29 to 262023c Compare July 1, 2021 16:21

danxuliu approved these changes Jul 1, 2021

View reviewed changes

nickvergessen merged commit 44ae334 into master Jul 1, 2021

nickvergessen deleted the simulcast-support branch July 1, 2021 17:18

backportbot-nextcloud bot added the backport-request label Jul 1, 2021

backportbot-nextcloud bot mentioned this pull request Jul 1, 2021

[stable22] Add support for simulcast streams. #5914

Merged

backportbot-nextcloud bot removed the backport-request label Jul 1, 2021

nickvergessen mentioned this pull request Jul 1, 2021

Add changelog for 12.0.0-rc.1 #5913

Merged

This was referenced Jul 2, 2021

Improve selection of simulcast streams #5919

Open

Make simulcast work with screen shares too #5920

Open

Use a lower simulcast quality for slow clients #5921

Open

Adjust simulcast bitrates based on the bandwidth limits set in the HPB #5922

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for simulcast streams. #5535

Add support for simulcast streams. #5535

Uh oh!

fancycode commented Apr 29, 2021 •

edited

Loading

Uh oh!

nickvergessen Jun 24, 2021

Uh oh!

fancycode Jun 24, 2021

Uh oh!

danxuliu left a comment

Uh oh!

fancycode commented Jul 1, 2021

Uh oh!

danxuliu commented Jul 1, 2021

Uh oh!

danxuliu left a comment

Uh oh!

nickvergessen commented Jul 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add support for simulcast streams. #5535

Add support for simulcast streams. #5535

Uh oh!

Conversation

fancycode commented Apr 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickvergessen Jun 24, 2021

Choose a reason for hiding this comment

Uh oh!

fancycode Jun 24, 2021

Choose a reason for hiding this comment

Uh oh!

danxuliu left a comment

Choose a reason for hiding this comment

Uh oh!

fancycode commented Jul 1, 2021

Uh oh!

danxuliu commented Jul 1, 2021

Uh oh!

danxuliu left a comment

Choose a reason for hiding this comment

Uh oh!

nickvergessen commented Jul 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fancycode commented Apr 29, 2021 •

edited

Loading