AVPlayer audio does not mix with Room audio.

[podkovyrin]

If you have AVPlayer that plays some video with sound it will become muted once video call established. And after that nothing helps: I've tried almost all combinations of AVAudioSession's AVAudioSessionCategoryOptions, AVAudioSessionSetActiveOptions, tried to setActive:NO and then YES back again before and after call connection.
It also doesn't matter in which position iPhone's sound switch is.
There is only thing that restores sound in player and keeps it going in the call is leaving application (with home button) and open it back.

[piyushtank]

@podkovyrin Playing audio+video with AVPlayer when a video call is established is not currently supported by Video SDK. We are discussing internally on supporting this feature, I will update the ticket when we have more information on this.

[ceaglest]

Hi @podkovyrin,

As @ptankTwilio mentioned we have been discussing this feature request internally.

The problem with using AVPlayer or AVAudioPlayer while connected to a Room is that the media engine is configured for echo cancellation and doesn't know about your player's audio content. Because of this, the echo canceller is fighting with the audio from your player. This is definitely a solvable problem, but I'd like to understand more about your use case first.

If you have AVPlayer that plays some video with sound it will become muted once video call established.

At the moment, the audio pipeline is engaged before the first frame of video / audio flows from the remote Participant. Is the goal to seamlessly playback content from AVPlayer until the Room audio / video content starts and then cross fade between the two? If so, would you be okay with not being able to share or record audio until AVPlayer is stopped?

The reason why I ask is because we are considering two solutions to the problem.

1. Disable audio from the media engine entirely. Re-enable when the pre-roll (AVPlayer) content has completed.

2. Operate the media engine in a playback only audio mode. Switch to playback and recording once the pre-roll (AVPlayer) content has completed. Playback only support has also been requested in this ticket: Support for playback only audio #72.

I'm looking forward to hearing your feedback @podkovyrin. If anyone else has interesting use cases, and would benefit from these APIs please share them on this ticket.

Best,
Chris Eagleston

[DabKick]

Hi @ceaglest,

We are trying a similar project in our company. In addition to the feature requested by @podkovyrin, we would like to duck the volume of the AVPlayer when someone talks on the Twilio Room audio/video channel and then resume the volume of AVPlayer when no one is talking. Is that possible?

Thanks,
Balaji

[podkovyrin]

Hi @ceaglest,

Thanks for your detailed response!
Our use case is to play a video and having a call simultaneously and sound should be mixed from a video call and AVPlayer. That's why any switching between audio stream from a call and sound from a video is not we're trying to accomplish.
I don't know the nature why that workaround (with leaving-entering app) works, but it does pretty well - there is no echo or noise in the audio from call.

[ceaglest]

@podkovyrin okay, so sounds like the AVPlayer content will play the entire time you are connected to a Room. If thats the case does your Client need to share audio or just receive it from other Participants?

Hi @DabKick. Once we add playback only support this should work, assuming you don't need to record audio while the AVPlayer is active. With 1.0.0-beta13 you could use the getStats API to determine the current volume of a TVIAudioTrack.

Check out the docs for: getStatsWithBlock: and TVIAudioTrackStats.

We are also considering adding audio levels to TVIAudioTrack, so you don't have to fetch full stats just to know this information.

Best,
Chris

[podkovyrin]

@ceaglest

sounds like the AVPlayer content will play the entire time you are connected to a Room.

Yes, you're right.

does your Client need to share audio or just receive it from other Participants?

No, the call itself should transfer only audio from microphone, in other words it should be the same simple video call like in example.

[Amrit1901]

@ceaglest

Once we add playback only support this should work, assuming you don't need to record audio while the AVPlayer is active

So does this mean that the new SDK version won't support recording a livestream and play media with AVPlayer at the same time? Is it something possible in the current version. We did some testing and it seems like the AVPlayer object always blocks an ongoing livestream's audio. Is this because the audio track gets "damaged" and we have to create one again?

[ceaglest]

Hi @Amrit1901,

Thank you for the feedback. It sounds like it may be possible to support both AVPlayer playback, and Room Media playback / recording at the same time, but we will need to investigate this issue further. From my experience with WebRTC it is common to disable audio from the media engine while using AVPlayer content, but I'm open to the possibility of having both coexist.

My apologies for the lack of progress on this issue. We know that this is an important use case, and will be focusing more effort on it once we ship our first General Availability release at the end of the month.

Thanks,
Chris Eagleston

[ceaglest]

Hello,

Now that we've shipped our first GA release I've started looking into AVPlayer interoperability. Hopefully I will have more updates to share soon.

Best,
Chris Eagleston

[ceaglest]

Hi,

I've been doing some experimentation with AVPlayer and it is indeed possible to use it alongside a TVIRoom, all while sending & receiving audio. The solution involves using the TVIAudioController (CallKit) category methods to ensure that the application has manual control over the AVAudioSession.

I've completed an internal PoC that works reliably, and requires little or no changes to the SDK itself. I'll converse with my team members, and we will follow up with new sample code demonstrating the use of AVPlayer alongside our Video SDK.

Best,
Chris

[piyushtank]

Hi,

We have written a sample app to demonstrate how to use AVPlayer in your app with TwilioVideo. As @ceaglest mentioned earlier, the app is using TVIAudioController (CallKit) category methods to ensure that the application has manual control over AVAudioSession.

Please try it out and let us know if you have any questions.

Sample App: https://github.com/twilio/video-quickstart-objc#examples

Thanks,
Piyush

[MrBendel]

Running into the same issue. I get really inconsistent behavior, sometimes the audio between users drops out and sometimes the audio from video plays through the earpiece not the speaker.

I followed the code in the example app. In best case, I get audio between participants and audio from the video, but as soon as the video changes, everything gets screwed up again.

I've tried the following to no avail. It doesn't always respect the output to speaker.

[_audioSession setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker error:nil];

[piyushtank]

@MrBendel During the call setup, Video SDK changes the AVAudioSesssion's category and mode. I am suspecting, the AVAudioSession properties you are setting in your app gets overwritten by the SDK. So, rather setting properties on AVAudioSessin, I would recommend using TVIAudioController APIs instead. If you want to enforce speaker audio route you can set audio output using TVIAudioController API, here is the code snippet to do so -

 [[TVIAudioController sharedController] setAudioOutput:TVIAudioOutputVideoChatSpeaker];

Please let me know if you have any questions.