Make possible to select input media devices before calls #3913

danxuliu · 2020-07-16T16:11:36Z

Requires #3912
Fixes #358

This pull request adds a new Preview tab to the right sidebar to select the audio and video devices to be used when a call is started, and also to be able to check whether the devices are properly working (or whether your lack of pants is visible or not before starting the call).

The MediaDevices Web API provides device labels only if a stream is active or if permanent media permissions have been granted. Due to this a fallback label is generated to be used when the actual device label is unknown. This fallback label is persistent for the whole session, so if a device is connected, disconnected and connected again later after other devices have also been connected the same fallback label as before will be used.

Nevertheless, the fallback label should be rarely visible; the device labels are updated whenever a stream is started, which is anytime that a different device preview starts, so in most cases the actual device label will be shown to the user.

Currently the audio and video input devices can be selected only when not in a call. The Preview tab is not usable during calls, so it would be better to remove it. However I tried using v-if="!showChatInSidebar" and the sidebar tabs behaved weirdly (they were out of order when the preview tab was shown again). As (eventually) it will be possible to change the devices during calls too for now the preview tab is shown but with its contents disabled during calls.

Caveats:

Detecting device changes only works if media permissions have been granted (either permanently or because there is an active stream). In most cases it should not be a problem, but if the user does not grant permanent media permissions, disables microphone and camera and then connects a new device the device will not be shown in the device list, which could be confusing for the user (although fortunately it should not be a common scenario).
Currently this could be solved in Firefox by polling the devices when permissions are not granted, but that would be a short term solution. Due to a change in the MediaDevices spec (already partially implemented in Chromium) eventually it will not be possible to enumerate devices either when permissions are not granted, only to know if there is a microphone or camera connected, but nothing else. Due to this I do not think that adding the polling is worth it.
The sources can be overriden by the system and it does not seem possible to detect that from the web app. For example, if the user selects an audio device but then changes the source used by the browser with PulseAudio (through pavucontrol) Talk will still show the original device as being the one used. If the user does the change explicitly it should not be a big problem, although it could be if some other element in the system does it and then the user does not understand why the device that is selected is not the one being used. Alas there does not seem to be anything that can be done here.
Chromium does not properly set PulseAudio virtual audio sources (and it does not offer PulseAudio monitors as sources either). For example, if the module-echo-cancel is loaded (pactl load-module module-echo-cancel) a virtual device on top of each normal input device will be created. If the virtual device is selected in Chromium, however, the normal device will be actually used instead of the virtual one (as can be seen by checking the recording device being used in pavucontrol). Unfortunately it does not seem possible to detect this; if the user selects a virtual device... it would not work as expected, and virtual devices can not be filtered out either :-(
When switching several times between audio devices in Firefox a NotReadableError with message Concurrent mic process limit is sometimes thrown. When that happens Firefox will get stuck to the last selected microphone and it will not be possible to change it. Moreover, it will not be possible to change it even from pavucontrol. Reloading the page seems to be the only way to solve this. In this pull request streams are explicitly stopped before a new one is started, so this looks like a lower level issue; maybe under the hood stopping the stream does not immediately stops it and in some cases it clashes with the new requested stream, but I have no idea.
If permissions have been rejected the user must explicitly remove the denied permissions before previews work again. It does not seem to be possible to trigger the permissions dialog once they have been rejected (the Permissions API has a request method, but it is not implemented in any browser yet).

Pending:

If the preview tab is never opened before a call and then it is opened during a call "No microphone/camera" will be shown, even if those devices are actually being used in the call - If an error happened when getting the media for the call this will not be reflected in the preview tab, but this can be addressed when making possible to change the devices during a call
Selecting the same option again removes it, ~~not sure if a bug or expected behaviour when using the Multiselect component~~ - It is a configurable property in the https://vue-multiselect.js.org component used by Multiselect
If an error happened when getting the preview provide also a text with the error instead of just an icon
Improve toasts when media could not be got (as both audio and video could have been explicitly disabled but an error would be shown nevertheless)
Update devices when getting user media if some device label is not known yet
~~Get a single stream for both audio and video instead of separate ones for previews?~~ - Separate media permission requests for each type feels better (to me, at least :-P )
Move MediaDevicesPreview to the Settings tab? In that case the preview should not be immediately visible and a button should be added to enable and disable the preview. Otherwise guests will always be asked for media permissions just by opening the conversation. Besides that I prefer to have a Preview tab with a video icon as it seems more intuitive about what it does. On the other hand, once changing the devices also during calls is implemented the name Preview is less appropriate... So I do not know 🤷 Other alternatives are also welcome, of course :-)
Selecting the media devices is currently possible in the main Talk UI. When Talk is shown in the sidebar a Preview tab would be too much (specially in the Files app). What could be done in those cases? For the Files app and the public share page maybe adding a settings button next to the Start/Join call button that shows a dialog could be an option. In the video verification, on the other hand, when Request password is clicked the user immediately starts the call. Maybe in that case it would make sense to first show a screen to select the audio and video devices (and allow the user to check how the video looks even if there is only one camera) and when that dialog is accepted then start the call? All this could be done in follow up pull requests, though.

Follow up pull requests:

Make possible to select the audio output device
Make possible to change the devices also during calls
Persist the selected device ID between sessions?

MediaDevicesManager stores the id of the audio and video input devices to be used when requesting media. If no id is set then it behaves like before, that is, letting the browser decide which device to use. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

Signed-off-by: Daniel Calviño Sánchez <[email protected]>

The device label returned by "MediaDevices.enumerateDevices()" is empty if permanent media permissions have not been granted and there is no MediaStream currently active. Therefore a fallback label is generated to help the user differentiate between devices. The labels are persistent during the session, so if a device is connected, disconnected and then connected again some time later after other devices have been connected too it will retain its original fallback label. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

MediaDevicesSelector is a dropdown to select a media device of a specific kind. MediaDevicesPreview has selectors for audio and video inputs that modify the devices to be used in MediaDevicesManager. Additionally it also shows a preview of the selected devices with a volume bar in the case of audio or a video element in the case of video. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

When there is no selected device (either because the selected one was removed or because there were no devices before) but there are other devices now a fallback device is automatically selected. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

Now it is possible to disable audio and video devices by setting the selected id to null. Fallback devices will still be used when the selected device is undefined, but not when it is explicitly set to null. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

The preview tab makes possible to select the audio and video devices to be used in a call, as well as showing a preview of them (so the user now can check what is visible in the camera before joining a call). In order to show the preview of the devices a media stream has to be requested, which will ask the user for permissions (unless they have been granted permanently). To prevent permissions requests as soon as the conversation is opened the preview tab is not shown as the first tab, and to limit the use of the devices only to the scrictly necessary the previews are enabled only when the preview tab is the active one. Currently changing the devices during a call has no effect, so the device selectors (as well as the previews themselves, as they will have the same content already shown in the call view) are disabled during calls. The tab itself is not removed due to ordering issues when removing and adding it back. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

Signed-off-by: Daniel Calviño Sánchez <[email protected]>

When an audio or video device is selected a stream from that device is requested to show its preview. The stream is resolved asynchronously, so while that happens the user could select a different device, which in turn will request another stream. As only a single stream for each type is expected to be active in the previews this could lead to streams being opened but never closed. Now this is enforced by preventing further stream requests while a previous one has not been completed yet. If several stream updates are triggered while waiting for a previous one once that previous one is finished a new stream will be requested for the last selected device. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

When a preview stream is updated there is no need to request the stream again if the current stream comes already from the selected device. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

In Firefox the dialog to grant media permissions allows the user to change the device to use. In that case the device selected in the dialog will be the one used, no matter which one was originally requested. Now this is taken into account and the selected device is updated to reflect the one selected by the user in the dialog. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

In latest MediaDevices spec if permanent media permissions have not been granted and there is no active stream "enumerateDevices" returns at most one device of each kind, and all of them with empty attributes (including the deviceId) except for the kind. This is already partially implemented by Chromium, so devices with an empty "deviceId" need to be taken into account. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

nickvergessen

🦈 lets move forward bite by bite

nickvergessen · 2020-08-05T09:24:51Z

src/components/MediaDevicesSelector.vue

+				}
+			})
+
+			options.push({


I would put it in the top, but well

nickvergessen · 2020-08-05T09:26:33Z

src/components/RightSidebar/RightSidebar.vue

 			icon="icon-settings">
 			<SetGuestUsername />
 		</AppSidebarTab>
+		<AppSidebarTab


The policy says to avoid using more than 3 tabs.
I think with some other features coming up we discussed to rename projects to advanced, and this could go there then from my pov.

nickvergessen · 2020-08-05T09:36:18Z

Move MediaDevicesPreview to the Settings tab? In that case the preview should not be immediately visible and a button should be added to enable and disable the preview. Otherwise guests will always be asked for media permissions just by opening the conversation.

👍 lets do that and I was about to bring up the same concern.

Selecting the media devices is currently possible in the main Talk UI. When Talk is shown in the sidebar a Preview tab would be too much (specially in the Files app). What could be done in those cases?

Ignore non-talk UI for now, but while in a call, I think a ... or option via the icon-video and icon-audio buttons are necessary anyway, so you don't have to move away from chat/sidebar-calls. But a preview/pre-selecting is just not done outside default UI.

danxuliu added 2. developing enhancement feature: frontend 🖌️ "Web UI" client feature: call 📹 Voice and video calls labels Jul 16, 2020

danxuliu added this to the 💚 Next Major (20) milestone Jul 16, 2020

danxuliu force-pushed the make-possible-to-select-input-media-devices-before-calls branch 2 times, most recently from 542b94f to 243a57c Compare August 5, 2020 03:37

danxuliu added 12 commits August 5, 2020 05:39

Keep track of devices in MediaDevicesManager

e59629b

Signed-off-by: Daniel Calviño Sánchez <[email protected]>

Show also an error message instead of just an error icon

50d772d

Signed-off-by: Daniel Calviño Sánchez <[email protected]>

Do not request a stream again for the active device

51a9aed

When a preview stream is updated there is no need to request the stream again if the current stream comes already from the selected device. Signed-off-by: Daniel Calviño Sánchez <[email protected]>

danxuliu force-pushed the make-possible-to-select-input-media-devices-before-calls branch from 243a57c to ec7fa54 Compare August 5, 2020 03:41

danxuliu added 3. to review and removed 2. developing labels Aug 5, 2020

danxuliu marked this pull request as ready for review August 5, 2020 03:57

danxuliu requested review from marcoambrosini and nickvergessen August 5, 2020 03:57

nickvergessen approved these changes Aug 5, 2020

View reviewed changes

nickvergessen merged commit d598653 into master Aug 5, 2020

nickvergessen deleted the make-possible-to-select-input-media-devices-before-calls branch August 5, 2020 09:36

danxuliu mentioned this pull request Aug 19, 2020

Make possible to select input media devices during calls #4023

Merged

3 tasks

danxuliu mentioned this pull request Aug 24, 2020

Do not use "finally" on promises returned by "getUserMedia" #4055

Merged

danxuliu mentioned this pull request Sep 24, 2020

Remember selected input devices #4218

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make possible to select input media devices before calls #3913

Make possible to select input media devices before calls #3913

Uh oh!

danxuliu commented Jul 16, 2020 •

edited

Loading

Uh oh!

nickvergessen left a comment

Uh oh!

nickvergessen Aug 5, 2020

Uh oh!

nickvergessen Aug 5, 2020

Uh oh!

nickvergessen commented Aug 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Make possible to select input media devices before calls #3913

Make possible to select input media devices before calls #3913

Uh oh!

Conversation

danxuliu commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickvergessen left a comment

Choose a reason for hiding this comment

Uh oh!

nickvergessen Aug 5, 2020

Choose a reason for hiding this comment

Uh oh!

nickvergessen Aug 5, 2020

Choose a reason for hiding this comment

Uh oh!

nickvergessen commented Aug 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

danxuliu commented Jul 16, 2020 •

edited

Loading