Local Audio collection of WEBRTC

Transferred from: http://www.cnblogs.com/fangkm/p/4374668.html

The previous blog introduces the local video collection, this article introduces the audio capture process, but also introduces WEBRTC native audio collection, and then introduces chromium source to its customization.

1. WEBRTC native Audio capture

Let's first introduce the interface concepts in WEBRTC that seem to be related to audio acquisition:

Does the structure look similar to the video track structure? However, before mentioned, if you use symmetrical thinking, in this structure to find the same as video track capture source and output source, it is certainly reactive and return, Localaudiosource to audiosourceinterface implementation is an empty implementation, there is no audio source, The audio processing interface audioprocessorinterface and the output interface Audiorenderer have become bricks without straw. These interfaces first placed in this, may be similar to the audiocapturer of the framework is being implemented in the way, it is possible that these interfaces of different uses, such as the remote audio stream abstraction, and so on, put aside, first note that there is this thing. This is only about WEBRTC local audio collection and processing. As mentioned in the previous introduction of the audio and video interface, the acquisition of local sound is uniformly encapsulated by the Audiodevicemodule interface:

Audiodevicemodule is a chatty interface that would like to encapsulate all the audio-related interfaces (practically), including: Enumerate the audio capture device (Record) and playback device (playout), set the current capture device/playback device, start/ Stop audio capture/playback, set audio gain control switch (AGC), etc. Audiotransport is a critical external interface that is responsible for incoming audio data (calling the Needmoreplaydata method for playout use) and output (invoking the Recordeddataisavailable method, Data is generated by record capture operations).

The Audiodevicemoduleimpl implements the Audiodevicemodule interface, When created, call the Createplatformspecificobjects method to create a platform-dependent Audiodevicegeneric interface implementation. The interface abstracts the acquisition and playback logic of audio, and there are two implementations under the Windows platform:

• Audiodevicewindowswave implements the traditional Windows Wave APIs scenario.
• Audiodevicewindowscore implements the Windows Core Audio APIs scenario that is supported after Vista.

In addition, Audiodevicemoduleimpl maintains a Audiodevicebuffer object to manage the buffer of audio data, which interacts directly with the external interface audiotransport. Like what:

• When Audiodevicewindowswave or audiodevicewindowscore need to play audio data, the Audiodevicebuffer Requestplayoutdata method is called to request the data to be played. The Getplayoutdata method is then used to obtain the data that was just requested. The requestplayoutdata of Audiodevicebuffer is to invoke the Needmoreplaydata method of the Audiotransport interface to request the audio stream data to be played.
• When Audiodevicewindowswave or Audiodevicewindowscore acquires audio data, it calls Audiodevicebuffer's Setrecordedbuffer method to pass the captured audio data in, Then call the Deliverrecordeddata method to distribute it, which is done by invoking the recordeddataisavailable of the Audiotransport interface.

In short, the Audio collection module has revealed the chatty structure design everywhere. If you can, really should refine the concept of design, such as audio capture and audio playback logic separation, audio input and output interface split, and so on, so as to talk about the structure of the design.

2. Chromium to WEBRTC Audio acquisition adapter

Based on the WEBRTC local audio interface design, Chromium provides a Webrtcaudiodeviceimpl class to implement the Audiodevicemodule interface. This class of objects is created and maintained by Peerconnectiondependencyfactory and is structured as follows:

, Webrtcaudiodeviceimpl rejects the design of the native Audiodevicemoduleimpl implementation chatty, but separates the audio collection from the audio rendering logic, Corresponds to Webrtcaudiocapturer and Webrtcaudiorenderer respectively. Webrtcaudiorenderer is rendered by the Webrtcaudiodeviceimpl request audio stream data through the Renderdata method of the Webrtcaudiorenderersource interface. Webrtcaudiodeviceimpl forwards the request to the previously mentioned external interface audiotransport. The Webrtcaudiocapturer encapsulates the audio acquisition logic, which distributes the collected data through the Peerconnectionaudiosink interfaces held by the Webrtclocalaudiotrack object, Webrtcaudiodeviceimpl is implementing this interface to receive audio capture data, which is then passed out through the Audiotransport interface. As for the holders of the Webrtcaudiocapturer object Mediastreamaudiosource and Webmediastreamtrack, here is a concept for the time being, they are chromium the implementation of the HTML5 media stream interface. The next step is to analyze webrtcaudiocapturer and webrtcaudiorenderer two key classes, without a doubt, they all involve a specific platform implementation, and in chromium also across the render and browser processes. And the introduction of chromium video capture mode, as is not the focus of this article, here are only listed structure, not to explain, if you have the development of the need, you can follow the structure of the map to see the source.

This is the structure of Webrtcaudiocapturer acquisition of audio data, involving cross-process communication, and the structure is very complex. Webrtcaudiorenderer's structure is not ready to introduce, because this design of chromium is very symmetrical, the basic class name of the input into output is almost the webrtcaudiorenderer architecture.

Local Audio collection of WEBRTC