Android IOS WebRTC Audio Video Development Summary (86)--analysis of implementation of RTP/RTCP protocol in WebRTC

This paper mainly introduces the RTP/RTCP protocol in WEBRTC, Weizhenwei, the earliest published articles in the Wind network, ID:BEFOIO

Support original, reprint must indicate the source, welcome attention to my public number blacker (Id:blackerteam or WEBRTCORGCN).

a preface

The RTP/RTCP protocol is the cornerstone of streaming media communications. The RTP protocol defines the packet format for streaming media data over the Internet, while the RTCP protocol is responsible for quality assurance of service such as reliable transmission, flow control, and congestion control. In the WEBRTC project, the Rtp/rtcp module, as part of the transmission module, is responsible for marshaling the media data collected by the transmitting side, then sending it to the upper network module, and after receiving the RTP/RTCP module receives the packet from the upper module, the unpacking operation Finally, the load is sent to the decoding module. Therefore, the Rtp/rtcp module plays a very important role in WEBRTC communication.

In this paper, based on in-depth study of WEBRTC source code, with the transmission and reception of video data as an example, we strive to use concise language to describe the implementation details of RTP/RTCP module and lay a good foundation for further mastery of WEBRTC.

two Rtp/rtcp protocol overview

The RTP protocol is the underlying protocol for streaming media on the internet, which details the standard packet format for transmitting audio and video over the Internet. The RTP protocol itself only guarantees the transmission of real-time data, and the RTCP protocol is responsible for the transmission quality assurance of streaming media, providing services such as flow control and congestion control. During the RTP session, each participant periodically sends RTCP messages to each other. The message contains statistical information such as the sending and receiving of each participant's data, and the participants can dynamically control the quality of streaming media transmission.

RFC3550 [1] defines the basic content of the RTP/RTCP protocol, including the message format, transmission rules, and so on. In addition, the IETF defines a range of extension protocols, including the RTP protocol-based scaling, and the RTCP protocol based on the message type extension, and so on. [2] The details can be found in the reference literature.

three WEBRTC thread relationships and data flow

WEBRTC provides two threads: signal and worker, the former responsible for signaling data processing and transmission, the latter responsible for the processing and transmission of media data. Within WEBRTC, a series of threads work together to complete the data flow pipeline. The following is an example of the video data processing process, which illustrates the thread partnership within WEBRTC.

Figure 1 WEBRTC thread relationship and data pipeline

As shown in 1, the capture thread captures raw data from the camera, getting the videoframe;capture thread to be system-dependent, and possibly the thread that invokes the V4L2 interface on the Linux system, which may be the interface that calls the Avfoundation framework on the MAC system. Then the raw data videoframe from the capture thread to the worker thread, the worker thread acts as a porter, does not do special processing of the data, but forwards to the encoder thread. The encoder thread calls the specific encoder (such as VP8, H264) to encode the original data videoframe, and the encoded output further makes RTP packets to form the RTP packet. The RTP packet is then sent to the PACER thread for smooth sending, and the PACER thread pushes the RTP packet to the network thread. The final network thread calls the Transport Layer system function to send the data to the net.

At the receiving end, the network thread receives the byte stream from the net, then the worker thread deserializes the RTP packet and takes a group frame operation in the VCM module. The decoder thread decodes the data frame that the frame completes, and the decoded raw data videoframe is pushed to the incomingvideostream thread, which puts videostream into render for rendering. At this point, a frame of video data completes the complete process from acquisition to display.

In the above process, the RTP packet is generated after the sender-side encoding is complete, its encoded output is encapsulated as RTP packets, and then sent to the network by serialization. After receiving the network packets received by the driven by network thread, the deserialization is reverted to the RTP message, and then the media data load is unpacked and decoded by the decoder. RTP packets perform a series of statistical operations during the sending and receiving process, and the statistical results are used as data sources for constructing RTCP messages. RTP message construction, send/Receive statistics and RTCP message construction, analytic feedback, is the focus of the next analysis.

four RTP message sending and receiving

The construction and transmission of RTP packets occurs after the encoder code, before the network layer sends the packet, and the receiving and the packet occurs after the network layer receives the data, before the decoder code. This section analyzes the contents of these two sections in detail.

4.1 RTP Message Construction and Dispatch

Figure 2 describes the construction and transmission of RTP packets after the sender-side encoding, involving three threads: Encoder, pacer, and network, respectively, responsible for encoding and constructing RTP packets, smoothing the sending and transmitting layer. The following is a detailed description of the three threads working together .

Figure 2 RTP Message construction and Dispatch

The encode thread calls the encoder (such as VP8) to encode the captured raw videoframe, and after the encoding is complete, its output encodedimage through the callback to the videosendstream::encoded () function, This is then routed through the Payloadrouter to Modulertprtcpimpl::sendoutgoingdata (). Next, the function calls Rtpsender::sendoutgoingdata () down and calls Rtpsendervideo::sendvideo (). The function wraps the encodedimage and then fills the RTP header with the RTP packet, and if FEC is configured, it is further encapsulated as a FEC message. Finally, return Rtpsender::sendtonetwork () for the next send.

The Rtpsender::sendtonetwork () function stores the messages in the rtppackethistory structure for caching. Next, if Pacedsending is turned on, the construction Packe is sent to the pacedsender for queuing, otherwise it is sent directly to the network layer.

The PACER thread periodically fetches the packet from the queue, and then calls Pacedsender::sendpacket () to send, followed by Modulertprtcpimpl to Rtpsender::timetosendpacket (). The function first gets the payload of the packet from the Rtppackethistory cache, and then calls the Prepareandsendpacket () function: Updates the relevant fields of the Rtpheader, counts the delays and packets, Call Sendpackettonetwork () to send the message to the transmission module.

The network thread invokes the transport layer socket to perform a data send operation. At this point, the RTP construction and send process of the sending end is complete. It is important to note that after RTP is sent in Rtpsender, the information about RTP messages is counted. This information is important as a source of data for constructing SR/RR messages in RTCP.

4.2 RTP Packet Reception and parsing

At the receiving end, the receiving and unpacking operation of the RTP message is performed mainly in the worker thread, and the RTP message is taken from the network thread, enters the worker thread, passes the unpacking operation, enters the VCM module, is decoded by the decode thread, and is finally rendered by the render thread. 3 describes the process of RTP messages in the worker thread.

Figure 3 RTP Message reception and parsing

The RTP packet arrives at the call object through the network layer and finds the corresponding Videoreceivestream according to its SSRC by calling its DELIVERRTP () function to Rtpstreamreceiver::D ELIVERRTP (). The function first parses the packet to get the RTP header information, and then performs three operations: 1. Bitrate estimation; 2. continue sending the packet; 3. Receive statistics. The rate estimation module uses the GCC algorithm to estimate the code rate, constructs the REMB message, and sends the RTPRTCP module back to the sending end. The receiving statistic then counts the RTP receiving information as the data source of the RTCP RR message. The following analysis focuses on the next packet delivery process.

Rtpstreamreceiver::receivepacket () First to determine whether the packet is a FEC message, if it is called fecreceiver to unpack, otherwise directly call Rtpreceiver::incomingrtppacket (). The function analyzes the RTP packet to obtain a general RTP header description structure, and then calls Rtpreceivervideo::P Arsertppacket () to further obtain video-related information and load, and then go through the callback to return the Rtpstreamreceiver object. The object sends RTP description information and payload to the VCM module, continuing the next Jitterbuffer cache and decoding rendering operations.

RTP packet unpacking process is the inverse process of packets, the important output information is RTP header description and media load, which is the basis of the next Jitterbuffer cache and decoding. In addition, the information obtained from the RTP messages is the data source of the RTCP RR message.

five RTCP message sending and receiving

RTCP protocol is under the control of RTP protocol, which is responsible for the quality assurance of streaming media service. The more commonly used RTCP messages are reported by the Send side report SR and the receiving end of the RR, respectively containing data sending statistics and data receiving information. This information is important for streaming media quality assurance, such as bitrate control, load feedback, and so on. Other RTCP messages, such as SDEs, BYE, sdes, etc., are defined in detail by RFC3550.

This section focuses on the construction, sending, receiving, parsing, and feedback processes of WEBRTC internal RTCP messages. Again, it is necessary to emphasize that the data source of the RTCP message comes from the statistical information of the RTP message when it is sent and received. Within WEBRTC, the RTCP message is sent in a combination of periodic sending and timely transmission: moduleprocess thread periodically sends RTCP messages, while Rtpsender determines whether to send RTCP messages before each time the RTP message is sent In addition, after the receiving terminal code rate Estimation module constructs the REMB message, the Moduleprocess module sends the RTCP message immediately by setting the timeout.

5.1 RTCP message Construction and Dispatch

On the sending side, the RTCP is periodically sent as a benchmark, supplemented by the timely sending of RTP messages and the immediate sending of REMB messages. The sending process mainly includes feedback information acquisition, RTCP message construction, serialization and transmission. Figure 4 depicts the construction and transmission of the RTCP message.

Figure 4 RTCP Message Construction and Dispatch

The moduleprocess thread periodically calls Modulertprtcpimpl::P the Rocess () function, which uses the Rtcpsender::timetosendrtcpreport () function to determine whether current RTCP messages need to be sent immediately. If so, the RTP send statistic is obtained first from rtpsender::getdatacounters (), then Rtcpsender::sendrtcp () is called, followed by Sendcompoundrtcp () to send the RTCP combination message. For the definition of RTCP combination messages, please refer to [1].

In the Sendcompoundrtcp () function, first determine what type of RTCP message will be sent through Preparereport (). Then for each kind of message, call its constructor (such as the construction SR message is BUILDSR () function), the constructed message is stored in the Packetcontainer container. Finally, call Sendpackets () to send.

Each RTCP message then calls its own serialization function, which serializes the message into a network byte stream. Finally, the Packetcontainer::onpacketready () is reached via a callback, and the byte stream is eventually sent to the Transport Layer module: Transportadapter to Basechannel, The network thread calls the transport-layer socket API to send data to the Web.

The construction and sending process of RTCP message is not very complex, the core operation is to get the data source, construct the message, serialize and send. It is relatively cumbersome to construct messages and serialization, based on the details of the RFC definition.

5.2 RTCP message Receiving and parsing

The receiving end of the RTCP message receiving and parsing process is shown in 5.

Figure 5 RTCP Message reception and parsing

At the receiving end, the RTCP message receives the same process as RTP, and then arrives at the call object after receiving the network, then finds Videoreceivestream through SSRC and then arrives at Rtpstreamreceiver. Next, the parsing and feedback operations of RTCP messages are done in the Modulertprtcpimpl::incomingrtcppacket () function. The function first calls Rtcpreceiver::incomingrtcppacket () to parse the RTCP message, gets the Rtcppacketinformation object, and then calls Triggercallbacksfromrtcppacket (), triggering a callback operation by each observer who registers here.

Rtcpreceiver::incomingrtcppacket () uses Rtcpparser to parse the combined message, and for each message type, invokes the corresponding processing function (such as the Handlesdes function that handles SDEs), and gets the description structure of the message after deserialization. Finally all messages are combined to form rtcppacketinformation objects. The object then calls the Triggercallbacksfromrtcppacket () function as a parameter to trigger a callback operation, such as handling the NACK callback, handling the SLI callback, handling the REMB callback, and so on. These callbacks in their respective modules control the streaming media data encoding, transmission, bitrate and other service quality assurance, which is where the RTCP message finally works.

At this point, we analyze the whole process of sending and receiving RTCP messages.

Vi. Summary

Based on the deep analysis of WEBRTC source code, this paper describes the realization process of the RTP/RTCP module with the flowchart, and makes a thorough and detailed study on the key issues (such as the data source of the RTCP message). It lays a good foundation for further mastering the principle and details of WEBRTC's realization.

Reference Documents

[1] rfc3550-rtp:a Transport Protocol for real-time applications
Https://www.ietf.org/rfc/rfc3550.txt

[2] Beyond RFC3550-RTP/RTCP protocol family analysis: http://www.jianshu.com/p/e5e21aeb219f

Android IOS WebRTC Audio Video Development Summary (86)--analysis of implementation of RTP/RTCP protocol in WebRTC