WebRtcVideoEngine2 module of WEBRTC

Transferred from: http://www.cnblogs.com/fangkm/p/4401143.html

Finally talked about the video data encoding send module, not easy. Overall also looked at a lot of time WEBRTC source, the biggest feeling is that each module in the development of the time is very independent, each module has defined its own set of interfaces, the last string up when adding a variety of adaptation objects to transfer. This gives us those who have just started to read the source of the great distress, but the WEBRTC module structure design is very good, or I do not see the power.

Note naming, WebRtcVideoEngine2 with a 2 words, do not think, this is definitely an upgrade version of the Videoengine, there is a Webrtcvideoengine class. From what I have now understood, WebRtcVideoEngine2 's improvement over Webrtcvideoengine is to split the video stream into split: The Send Stream (Webrtcvideosendstream) and the receive stream (Webrtcvideoreceivestream), Thus the structure is more reasonable, the source code clearer. This section will be in detail later. Before introducing WebRtcVideoEngine2, the first simple analysis of the WEBRTC media engine structure, to tell the truth, I really do not express how the engine is a concept, but since so named, the core module must be wrong. The structure is simple:

• mediaengineinterface: The logical interface of abstract media engine, responsible for creating videomediachannel for video transmission, Voicemediachannel for audio transmission, Register the Audio data processing interface and so on.
• Compositemediaengine: Implements the Mediaengineinterface interface, itself is a template class, two template parameters are video engine and audio engine respectively. The template parameters whose derived class webrtcmediaengine depend on are webrtcvoiceengine and Webrtcvideoengine, The WebRtcMediaEngine2 used for chromium are dependent on webrtcvoiceengine and WebRtcVideoEngine2.

The main function of WebRtcVideoEngine2 is to create a video channel object WebRtcVideoChannel2. The structure is as follows:

When the Addsendstream method of WebRtcVideoChannel2 is called, a Webrtcvideosendstream object is created, similarly, the Addrecvstream member method is called, A Webrtcvideoreceivestream object is created.

When the WebRtcVideoChannel2 Setcapturer method is called externally, it is forwarded to Webrtcvideosendstream to respond, The Webrtcvideosendstream internal Inputframe member method hooks the Videocapturer signalvideoframe signal to receive video frame data transmitted by the video collector.

WebRtcVideoChannel2 's Addsendstream and Setcapturer call timing is not considered here, these involve the network connection, and so on each node content analysis, then explore the whole process.

, Webrtcvideosendstream and Webrtcvideoreceivestream are just a wrapper class, The internal dependent call interface creates the corresponding Videosendstream interface implementation class and the Videoreceivestream interface implementation class. Within the internal namespace, there is a call class, Videosendstream class, Videoreceivestream class to implement these three interfaces, The call class creates key Videoengine objects to manage a series of processing logic during the sending of video data. From the code structure, Videoengine is a relatively independent module, it encapsulates the video data acquisition processing, encoding and other logic, the following carefully analyze the structure of Videoengine:

Videoengine module with Viebase, Viecodec, Viecapture, vieimageprocess, Vienetwork, Vierender, viertp_rtcp, VIEEXTERNALCODEC interface , note that these are functional interfaces, their corresponding implementations correspond to the Xxximpl class in, the Videoengineimpl class is derived from all XXXIMPL interfaces, so the external Videoengine interface, can be a strong way to obtain viebase, Viecapture such as the interface (according to Videoengine Strong to the corresponding interface logic encapsulated in the target interface of the GetInterface static method), the outside world can use these interfaces to complete the video data to do the corresponding settings, These settings are eventually reflected in a class object called Vieshareddata. The object is created by Viebaseimpl and shared between implementations of the interfaces, Xxximpl can be accessed through Viebaseimpl's Shared_data method, and there are three types of data for sharing: Vieinputmanager, Viechannelmanager and Vierendermanager. The key three objects are described below.

• Vieinputmanager: Encapsulated video Capture/input logic (haha, another set of video input logic), structure:

Vieinputmanager assigns a Viecapturer object to each channel as a video source, Viecapturer can pass in or create a Videocapturemodule video capture module, and receives data from it via the Videocapturedatacallback interface, You can also receive direct incoming video frame data directly from the Vieexternalcapture interface (calling the Incomingframe method of the Vieexternalcapture interface). Videosendstream is the creation of a Vieexternalcapture object through Vieinputmanager to pass in the video frame data from the outside (through the inputframe of Webrtcvideosendstream). It is important to note that Vieinputmanager assigns a capture_id to the created Viecapturer object, which can be manipulated by the external capture_id to manipulate its viecapturer. The video source incoming logic is clear, then analyze how the video came out. Regardless of the video data receiving method, Viecapturer will not immediately pass the data out, because it needs to do related processing of these video data internally. Data processing must be time-consuming, if used synchronously, will block the flow of video. Therefore, when the viecapturer is created, a collection thread is started and the thread invokes the Viecaptureprocess handler, in which the videoprocessingmodule is called to process the video data (the light is highlighted, the flashing is If the Vieeffectfilter processing object is registered in the Vieimageprocessimpl, the object is also called to process the video frame data, and the Deliverframe method is finally distributed to all the Vieframecallback interfaces registered in it.

• Viechannelmanager: Encapsulates the video encoding and transmission logic, which is more complex, as a whole:

Viechannelmanager maintains the Vieencoder and Viechannel objects, Vieencoder implements the Vieframecallback interface to receive video frame data from Viecapturer objects, Vieencoder to encode the video frame data received, and then pass the encoded data to Viechannel (through the Payloadrouter object shared between them), Viechannel sends the encoded video data through the RTP/RTCP protocol. The following analyses respectively Vieencoder and Viechannel.

1) Vieencoder class : Encapsulates the video encoding process.

Video coding is managed uniformly by Videocodingmodule module, and the video frame incoming interface is through the Videocodingmodule Addvideoframe method, The encoded video outgoing interface is tuned back and forth with the Vcmpacketizationcallback interface. The logic for choosing which video encoding to choose is located in the Vcmcodecdatabase class, which currently supports VP8 encoding, VP9 encoding, and video format conversion to I420 format.

2) Viechannel class : Encapsulates the encoded video data sending logic and the video data reception decoding logic.

The transmission logic of video data is the encapsulation of RTP protocol by Payloadrouter object to Rtprtcp module, and the specific network sending operation is to send to Viesender to do the data. The logic of Viesender is relatively simple, limited to space, the figure can not be detailed annotation. The Viesender send operation relies on the transport interface that is set externally to it (through the Vienetwork interface of the Videoengine module).

When the WEBRTCVIDEOCHANNEL2 receives a network packet (in response via the onpacketreceived or Onrtcpreceived method), The Vienetwork interface exposed by the Videoengine module in the Videoreceivestream object responds to packet processing, The Receivedrtppacket or Receivedrtcppacket method that eventually triggers to Viechannel. Viechannel assigns the task that receives and decodes the network video data to the Viereceiver object. Viereceiver First call the Rtp/rtcp module to do the protocol resolution (the picture is limited to the space is not marked out), After parsing is done, call the Videocodingmodule module for decoding the data (see the Onreceivedpayloaddata method of Viereceiver). The Videocodingmodule module internally maintains a videoreceiver corresponding to the Videosender to complete the decoding logic, which is completely symmetric with the Videosender encoding logic and is no longer expressed here.

• Vierendermanager: This class encapsulates the video rendering logic with the following structure:

When the Viechannel receives the network data and decodes it, it will turn on the triggering rendering process (see Frametorender method), and Viechannel will call the Vieframecallback interface registered to it to distribute the video frame data. Vierendermanager maintains a Vierenderer object to implement the Vieframecallback interface, which distributes the data further, Finally, the Videoreceivestream object is sent to WebRtcVideoChannel2 through the Externalrenderer interface. Videoreceivestream sends the data to Videotrack via the Videorenderer interface set up by VideoSource, and the user can hook up the Videorendererinterface interface to receive the video frame data. Really around, and so many naming similarities (such as Videorender/videorenderer), feel the development of each module, the implementation of their own set of interface specifications, and finally forced to string together.

WebRtcVideoEngine2 module of WEBRTC