The frame and interface of WEBRTC

Transferred from: http://www.cnblogs.com/fangkm/p/4370492.html

Reprint Please specify source: http://www.cnblogs.com/fangkm/p/4370492.html

The previous article simply introduced the next WEBRTC protocol process, which begins with the introduction of frameworks and interfaces.

When it comes to frames, instinctively don't know where to start. Once directly from the chromium project on the integration of the source of WEBRTC, later found that this step is too big, see more, the concept of the more chaotic, see half a month feel also not what precipitate. It is also easy to start from the example engineering peerconnection_client provided by WEBRTC. First, the construction and rendering flow of the audio and video stream, the code structure of the sample project core is as follows:

From an object-oriented point of view, WEBRTC design is very good, really do the concept of interface programming, the use of WEBRTC function through the interface, so as to maximize the WEBRTC module to ensure the customization of This allows WEBRTC to return more to the nature of the description protocol. If WEBRTC's implementation of these interfaces does not meet your business needs, you can theoretically provide your own implementation logic. The Peerconnectionfactoryinterface and peerconnectioninterface in this figure are not representative of this customization, because the requirement scenario for re-providing their implementation logic does not exist (even without rewriting, but it also supports the customization of parameters , see overloaded methods for Createpeerconnectionfactory). But audio and video related interface custom scenes are very common, such as chromium browser integration WEBRTC, but audio and video capture need to go chromium their own audio and video modules, so chromium to WEBRTC audio and video acquisition interface re-made to achieve adaptation, There will be a chance to share the next chromium source to WEBRTC integration, but that is also in the WEBRTC familiar with the work after.

The conductor is the core business class provided by the example project, and the use of the entire WEBRTC is concentrated in this class. Conductor creates an implementation object of a Peerconnectionfactoryinterface interface through the Createpeerconnectionfactory method, which, through this interface, A critical Peerconnectioninterface interface can be created, and the Peerconnectioninterface interface is the WEBRTC protocol core. In addition, the Peerconnectionfactoryinterface interface provides a functional interface for creating local audio and video streams, as described later in this section. According to the member method of the Peerconnectioninterface interface in the diagram, we can see that the interface of WEBRTC communication flow is basically in here, and the callback notification to conductor is done by Peerconnectionobserver interface. See the previous blog post for a specific interactive process.

Next analysis of local audio and video related interfaces, due to more audio and video content, here first introduce the concept of the interface, do not talk about the implementation of the specific (the next section specifically on the WEBRTC of the original audio and video collection), or the Peerconnection_client project as an example:

Here involves a lot of audio and video related interfaces, basically are conceptual, the most afraid of encountering new design concepts, mainly afraid of their own understanding of the error, the following talk about my understanding of these interface concepts:

MediaStream Concept : Represents the media stream, which is abstracted by the Mediastreaminterface interface, each with a unique identity (returned by the label member method). It consists of a series of audio track (abstracted by the Audiotrackinterface interface) and video track (abstracted by the Videotrackinterface interface).

Track concept : specifically refers to the structure of the audiotrackinterface and Videotrackinterface interfaces, tracks represents the concept of the media flow in the orbit, Audiotrackinterface identifies the audio track , Videotrackinterface identifies a video track, which is allowed to carry multiple media track data in a mediastreaminterface-identified media stream, and is independent of each other in the process of encoding rendering. If the concept is still vague, the concept of orbit is equivalent to the concept of channel in audio data (left channel, right channel), YUV field in video data. Since the track encapsulates the media data, there must be a media source as the provider of the data, such as audio tracks by the Audiosourceinterface interface as the data source provider, The video track is provided by the Videosourceinterface interface as the data provider. There is an input interface must have an output interface, this part of the figure only shows the video data output interface Videorendererinterface, where the meaning of the render name is not limited to the video and audio data rendering, should be understood as an output interface.

videosourceinterface: Abstract video source interface for videotracks use, the same source can be shared by multiple videotracks. The video source accepts a videocapturer interface, abstract video acquisition logic, we can provide their own videocapturer implementation as a source of video data acquisition. Videocapturer is a key custom interface, such as chromium source code is their own implementation of the Videocapturer interface and no use of the original WEBRTC acquisition implementation, but chromium audio and video collection are in the browser process, Therefore, its implementation of the Videocapturer interface is more complex than imagined, it needs to receive the video frame data from the main process and then trigger the Videocapturer signalframecaptured signal.

audiosourceinterface: Conceptually similar to videosourceinterface, abstract audio source interface for audiotracks use, but from the source of understanding, this is a pseudo concept, Because there is no videocapturer-like Audiocapturer interface, there is no audio acquisition logic, actually WEBRTC audio acquisition interface using Audiodevicemodule, When creating the peerconnectionfactory can be customized by the outside, if not, then the internal creation of AUDIODEVICEMODULEIMPL to achieve this interface to complete the acquisition of audio equipment. Maybe it's not enough, anyway, I don't understand. Audio capture and video capture this design asymmetry. If the concept of a Audiocapturer interface is also encapsulated, can customization be higher.

The process of building a media stream is basically building video track and audio track, and adding it to the media stream. In the Peerconnection_client project, Conductor relies on the Createvideocapturer method of the Devicemanagerinterface interface to create a currently available video device acquisition Object Videocapturer, Use it as a source of data in the video capture source (by hooking up the Videocapturer signalvideoframe signal to receive video data), In addition, Mainwnd created an inner class videorenderer derived from the Videorendererinterface interface and added it to video track. The implementation of Videorenderer is to render the received video frame data to the window.

The next article begins the analysis of WEBRTC native audio-video local acquisition module.

The frame and interface of WEBRTC