WEBRTC Audio and Video engine research (1)--Overall architecture analysis

WEBRTC Technology Group: 234795279 Original Address: http://blog.csdn.net/temotemo/article/details/7530504 1, WebRTC purpose WebRTC (Web real-time communication) The ultimate purpose of the project The main is to allow web developers to be based on the browser (chrome\firefox\ ... Fast and easy to develop rich real-time multimedia applications, without the need to download and install any plug-in, Web developers do not need to pay attention to multimedia digital signal processing, just write a simple JavaScript program can be achieved, such organizations are developing JavaScript standard API, Currently WEBRTC 1.0 version, draft status, URL; In addition, WEBRTC also hopes to build a robust real-time communication platform between multi-internet browsers, creating a good ecological environment for developers and browser vendors.  At the same time, Google also hopes and is committed to make WEBRTC technology become one of the HTML5 standards, visible Google layout far-reaching. 2, WEBRTC frame composition structure diagram color identification description: (1) The purple part is the Web Developer API layer, (2) The blue solid line part is facing the browser vendor's API layer (that is, the red box inside the module, also I focus on the research part) (3) Blue dashed some browser vendors can customize the implementation of 3, WEBRTC Architecture Component Introduction (1) Your Web App
Web developers develop programs that enable Web developers to develop real-time communication applications based on video and audio based on Web APIs provided by the integrated WebRTC browser. (2) Web API
The WEBRTC standard API (Javascript) for third-party developers makes it easy for developers to develop Web applications similar to network video chats, and the latest standardization process can be viewed here。

(3) WebRTC Native C + + API
The local C + + API layer makes it easy for browser vendors to implement the WEBRTC standard Web API to process digital signal processes in an abstract way. (4) transport/sessionThe transport/Session Layer Session layer component is implemented using a partial component of the Libjingle library without using the Xmpp/jingle protocol

A. RTP Stack protocol stack
Real Time Protocol

B. Stun/ice
Call connections between different types of networks can be established through stun and ice components.

c. Session Management
An abstract session layer that provides session building and management capabilities. This layer protocol is left to the application developer to customize the implementation. (5) Voiceengine
The audio engine is a framework that includes a range of audio multimedia processing, ranging from video capture cards to network-based transmission solutions.
Ps:voiceengine is one of WEBRTC's most valuable technologies and is open source for Google's acquisition of Gips company. On VoIP, the technology industry is leading, and later articles will learn more about A. ISACInternet Speech Audio codec wideband and ultra-wideband codec for VoIP and audio streaming, is the default codec for the WEBRTC audio engine
Sampling frequency: 16khz,24khz,32khz; (default is 16khz)
The adaptive rate is 10kbit/s ~ 52kbit/;
Adaptive packet size: 30~60ms;
Algorithm delay: Frame + 3ms B. ILBC
Internet Low bitrate Codec
Narrowband voice codecs for VoIP audio streams
Sampling frequency: 8khz;
20ms frame bit rate is 15.2kbps
30MS frame bit rate is 13.33kbps
Standard defined by IETF RFC3951 and RFC3952
C. Neteq for VoiceThe NETEQ algorithm of speech signal processing element for audio software: Adaptive jitter Control algorithm and speech packet loss concealment algorithm. Enables fast and high resolution adaptation to changing network environments, ensuring a graceful sound quality with minimal buffering latency. Gips Company is a cavatina technology that can effectively handle the impact of voice quality due to network jitter and voice packet loss. Ps:neteq is also a very valuable technology in WEBRTC, which has obvious effect on improving the quality of VoIP, and it is better to integrate the AEC\NR\AGC modules. D. acoustic Echo Canceler (AEC)
The Echo Canceller is a software-based signal processing element that removes the echo collected from the mic in real time. E. Noise Reduction (NR)
Noise suppression is also a software-based signal processing element that is used to eliminate certain types of background noise (hiss, fan noise, etc.) associated with VoIP. ） (6) Videoengine
WEBRTC Video Processing engine
Videoengine is a comprehensive framework that includes a range of video processing, from camera capture video to video information network transmission to video display throughout the entire process of the solution. A. VP8
Video image codec, the default codec for the WEBRTC video engine
The VP8 is suitable for real-time communication applications because it is primarily a codec designed for low latency.
The PS:VPX codec is open source after Google acquired ON2, VPX is now part of the WEBM project, and the WEBM project is one of the HTML5 standards that Google is committed to driving. B. Video jitter Buffer
Video jitter buffers can reduce the negative effects of video jitter and video packet loss. c. Image Enhancements
Image quality Enhancement module
The images captured by the webcam are processed, including shading detection, color enhancement, noise reduction processing and other functions to improve video quality. 4, WEBRTC core Module API (1), Network Transmission module: LIBJINGLEWEBRTC re-libjingle some of the components, mainly network and transport components, information about Libjingle can be viewed here。 (2), audio, video image processing of the main data structure constant \videoengine\voiceengine   Note: All of the following methods, classes, structs, enumeration constants, etc. are in the WebRTC namespace

Class, struct, enumeration constant	Header file	Description
Structures	Common_types.h	Lists the structures common to the Voiceengine & Videoengine
Enumerators	Common_types.h	List the enumerators common to the Voiceengine & Videoengine
Classes	Common_types.h	List the classes common to Voiceengine & Videoengine
Class Voiceengine	Voe_base.h	How to allocate and release resources for the Voiceengine using factory methods in the Voiceengine class. It also lists the APIs which is required to enable file tracing and/or traces as callback messages
Class Videoengine	Vie_base.h	How to allocate and release resources for the Videoengine using factory methods in the Videoengine class. It also lists the APIs which is required to enable file tracing and/or traces as callback messages

(3), Audio Engine (voiceengine) module APIs   The following table lists the sub APIs that are currently available in Voiceengine

Sub-api	Header file	Description
Voeaudioprocessing	Voe_audio_processing.h	Adds Support for Noise Suppression (NS), Automatic Gain Control (AGC) and Echo control (EC). Receiving side VAD is also included.
Voebase	Voe_base.h	Enables full duplex VoIP using g.711. Note: This API must always be created.
Voecallreport	Voe_call_report.h	Adds support for Call reports which contains number of dead-or-alive detections, RTT measurements, and Echo metrics.
Voecodec	Voe_codec.h	Adds Non-default codecs (e.g. ILBC, ISAC, g.722 etc), Voice Activity Detection (VAD) support.
Voedtmf	Voe_dtmf.h	Adds Telephone event transmission, DTMF tone generation and telephone event detection. (telephone events include DTMF.)
Voeencryption	Voe_encryption.h	Adds external encryption/decryption support.
Voeerrors	Voe_errors.h	Error Codes for the Voiceengine
Voeexternalmedia	Voe_external_media.h	Adds support for external media processing and enables utilization of an external audio resource.
Voefile	Voe_file.h	Adds file playback, file recording and file conversion functions.
Voehardware	Voe_hardware.h	Adds sound device handling, CPU load monitoring and device information functions.
Voeneteqstats	Voe_neteq_stats.h	Adds buffer statistics functions.
Voenetwork	Voe_network.h	Adds external transport, port and address filtering, Windows QoS support and packet timeout notifications.
Voertp_rtcp	Voe_rtp_rtcp.h	Adds support for RTCP Sender reports, SSRC handling, RTP/RTCP statistics, Forward Error Correction (FEC), RTCP APP, RTP CA Pturing and RTP keepalive.
Voevideosync	Voe_video_sync.h	Adds RTP Header modification Support, playout-delay tuning and monitoring.
Voevolumecontrol	Voe_volume_control.h	Adds speaker volume controls, microphone volume controls, mute support, and additional stereo scaling methods.

(4), Video engine (videoengine) module APIs The following table lists the sub APIs that are currently available in Videoengine

Sub-api	Header file	Description
Viebase	Vie_base.h	Basic functionality for creating a Videoengine instance, channels and voiceengine interaction. Note: This API must always be created.
Viecapture	Vie_capture.h	Adds support for capture device allocation as well as capture device capabilities.
Viecodec	Vie_codec.h	Adds Non-default codecs, codec settings and packet loss functionality.
Vieencryption	Vie_encryption.h	Adds external encryption/decryption support.
Vieerrors	Vie_errors.h	Error codes for the Videoengine
Vieexternalcodec	Vie_external_codec.h	Adds support for using external codecs.
Viefile	Vie_file.h	Adds support for file recording, file playout, background images and snapshot.
Vieimageprocess	Vie_image_process.h	Adds effect Filters, deflickering, denoising and color enhancement.
Vienetwork	Vie_network.h	Adds send and receive functionality, external transport, port and address filtering, Windows QoS support, packet timeout n Otification and changes to network settings.
Vierender	Vie_render.h	Adds rendering functionality.
Viertp_rtcp	Vie_rtp_rtcp.h	Adds support for RTCP reports, SSRS handling rtp/rtcp statistics, NACK/FEC, keep-alive functionality and key frame request Methods.

Welcome to point out the wrong place: [email protected]/==================================================================================== ==============================/Author:zengxijin [email protected] Welcome reprint, please indicate the source of the original address: Http://blog.csdn.net/temo temo/========================================================================================

WEBRTC Audio and Video engine research (1)--Overall architecture analysis