WEBRTC Source Analysis: Audio module structure analysis

First, an overview of the WEBRTC audio processing flow, see:

WEBRTC The audio session is abstracted into a channel channels, such as A and b for audio calls, a needs to establish a channel and b for audio data transmission. There are three channel, each channel contains codec and rtp/rtcp send function.

In the case of a channel, the application will contain three active threads, a recording thread, an audio receive thread, and a playback thread.

1) Recording thread: Responsible for the microphone audio collection, see the Red path in the picture, after the acquisition of audio, cache to a certain length, audio processing, mainly including EC,AGC and NS and so on. and send it to the channel, through the audio

Codec module encoding, encapsulated into RTP packet, sent through the socket;

2) receive thread: See Blue Path, responsible for receiving the remote sent over the audio packet, unpack the RTP packet, decode the audio data, fed into the Neteq module cache.

3) Play thread: Responsible for headphone sound playback, see green Path. The playback thread goes to the outmixer to get the audio data to play, first obtains the audio frame Neteq stored in the channel that participates in the session, can do the AGC and NS processing, then mixes the audio signal of multiple channel, obtains the mixed audio, Passed to the Audioprocessing module for remote analysis. Finally play it out.

The following is a local loopback recording and playback code:

voiceengine* ve = voiceengine::create ();
voebase* base = Voebase::getinterface (VE);
Base->init ();
int chId = Base->createchannel ();
Base->setsenddestination (chid,3000, "127.0.0.1", 4000);
Base->setlocalreceiver (chid,3000,3001, "127.0.0.1");
Base->startplayout (CHID);
Base->startreceive (CHID);
Base->startsend (CHID);

....... sleep...wait.

Base->stopsend (CHID);

Base->stopreveive (CHID);

Base->stopplayout (CHID);

Base->terminate ();

This paper introduces the composition and structure of WEBRTC Audio module, describes the configuration and start-up of audio engine in detail, believing that after reading this article, many people can use WEBRTC to complete an audio calling program development.

First, external interface

The external main interface of the audio section is as follows, as shown in the relationship between each interface 1.

1) Voiceengine: Responsible for all interface queries of the engine, storing shared data information sharedata.

2) Voebase: Responsible for the basic operation of audio processing.

3) Voeaudioprocessing: Audio signal Processing interface, set the parameters of each audio processing item.

4) Voecodec: Audio Codec interface, provides support for codec query, audio codec settings.

5) Voehardware: Audio Hardware Device Interface, responsible for the audio hardware device settings.

The other interfaces are VOENETEQSTATS,VOENETWORK,VOERTP_RTCP,VOEVIDEOSYNC,VOEVOLUMECONTROL,VOEFILE,VOECALLREPORT,VOEDTMF, Voemeidaprocess and Voeencryption.

WebRTC use inheritance to implement Interface transformation and query, the data sharing between interfaces is done through Sharedata, first voiceengineimpl inherit the implementation of various external interfaces, so it is easy to get other external interfaces from Voiceengineimpl. And Voiceengineimpl itself also inherits Sharedata, when obtains the other external interface from the Voiceengineimpl simultaneously, implicitly passes the Sharedata pointer, therefore each interface can conveniently obtain to the Sharedata the data information. So although the relationship between classes and classes looks confusing, it's easier to use.

Using Voiceengine to obtain external interface: voeinterfacexx* Pinterf = Voeinterfacexx:getinterface (pvoiceengine);

Second, the module composition

Mainly consists of five major modules: Audiodevicemodule audio device module, audioprocess audio processing module, audiocodingmodule audio encoding module, Audioconferencemixer mixing module and RTPRTCP transmission module.

The sharedata is used to glue the relationships between the various modules, and is responsible for managing global objects, including Audiodevicemodule,transmitmixer,outputmixer,channelmanager and audioprocess.

Recording process: Audiodevicewincore is responsible for collecting audio data, passing it to audiodevicebuffer cache, audiodevicebuffer the data into Transmixmixer, First to audioprocess for the near-end audio processing, the completion of distribution to the various channel, the channel is encoded by audiocodingmodule, encoded and then delivered to the RTPRTCP through Rtpsender sent out.

Receiving process: Rtpreceiver is responsible for receiving the audio RTP packet, receiving the RTP packet and handing it over to the Acmneteq module in Audiocodingmodule to decode the cache.

Playback process: The channel extracts the cached decoded audio data from the Acmneteq module and passes it to the audioprocess processing if the remote data processing is required. Finally, all the channel is transferred to the outputmixer for mixing, and then transferred to the audioprocess for remote audio analysis. The final feed is fed into the audiodevicemodule audiodevcewincore playback.

Third, the configuration

1. Audio engine creation and deletion

Voiceengine*pvoeengine = Voiceengine::create ();

Voiceengine::D elete (Pvoeengine);

2. Audio Transceiver

1) Audio Call link creation

The channel in the WEBRTC, for all the way audio. As a network voice communication, create at least one audio channel.

The channel does not provide an external interface, it is managed by the Voebase, and the corresponding channel is selected by the index number.

Voebase*base = Voebase::getinterface (pvoeengine);

int Ch0 =base->createchannel ();

2) Network Port settings

The audio is sent out via RTP and RTCP, RTP and RTCP are implemented using UDP, and network ports and addresses need to be configured.

Set up 3000 ports sent to. 2 Machines

Base->setsenddestination (ch0,3000, "192.168.8.2");

Receiving RTP packets on the 3000 port of this machine

Base->setlocalreceiver (ch0,3000);

3) Audio Encoding selection

Voecodec is responsible for the codec configuration.

Voecodec*codec = Voebase::getinterface (pvoeengine);

Before setting the channel's encoding type, query the supported Encodings list.

Codecinstinst;

Intnum = Codec->numofcodecs ();

for (int i=0; i<num; ++i)

{

Codec->getcodec (I,inst);

Printing encoding Information

}

Setting the encoding 0

Codec->getcodec (0,inst);

Codec->setsendcodec (Ch0,inst);

WEBRTC automatically recognizes the encoding type, so the decoding does not need to be set.

4) Start

Start playback: base->startplayout (ch0); This action means that the call Ch0 is mixed out.

Start receive: Base->startreceive (ch0); Once received, the engine mixes the audio and then outputs each additional call.

Start send: Base->startsend (ch0), start sending, will check whether the recording is in progress, if the recording has been turned on, it will not be turned on, otherwise the audio device recording operation.

3, the Audio processing configuration

Voeaudioprocessing is responsible for the configuration of audio processing.

Voeaudioprocessing*paudioproc = Voeaudioprocessing::getinterface (pvoeengine);

Start AGC function

Paudioproc->setagcstatus (TRUE);

4, the audio equipment configuration

The Voehardware interface allows you to view recording and playback devices, and you can select the specified device for audio calls.

Voehardware*phardware=voeaudioprocessing::getinterface (Pvoeengine);

Int numin =phardware->getnumofrecordingdevices ();

for (int i=0;i<numin; ++i)

{

Phardware->getrecordingdevicenames (...)

Print recording Equipment

}

Select Device 0 as the recording device

Phardware->setrecordingdevice (0);

Playback device configuration is similar.

WEBRTC Source Analysis: Audio module structure analysis