Introduction to RTP/RTCP protocol

Real-time transmission protocol RTP (Realtime Transport Protocol) is a transmission protocol for multimedia data streams over the Internet. It is published by IETF (Internet Engineering Task Group) as rfc1889. RTP is defined to work during one-to-one or one-to-many transmission. It aims to provide time information and implement stream synchronization. Typical RTP applications are built on UDP, but can also work on TCP, ATM, and other protocols. RTP itself only guarantees the transmission of real-time data. It does not provide a reliable transmission mechanism for transmitting data packets in sequence, nor provides traffic control or congestion control. It relies on RTCP to provide these services. Real-time transmission control protocol RTCP (Realtime Transport Control Protocol): manages the exchange of control information between transmission quality in the current application process. During the RTP session, each participant periodically transmits the RTCP packet, which contains statistics such as the number of sent packets and the number of lost packets. Therefore, the server can use this information to dynamically change the transmission rate or even the payload type. RTP and RTCP can be used together to optimize transmission efficiency with effective feedback and minimal overhead. Therefore, RTP is especially suitable for transmitting real-time online data. RTCP has four main functions: (1) feedback is used to provide the transmission quality of distributed data. This feedback can be used to control traffic congestion, it can also be used to monitor the network and diagnose problems in the network. (2) it provides a permanent canonical name (canonical name) transfer layer flag for the RTP source, because SSRC (synchronization source ID) changes when a conflict is detected or the program is updated and restarted, an operation trace is required, in a set of related sessions, the receiver also uses cname to obtain the associated data stream (such as audio and video) from a specified participant. (3) adjust the RTCP packet sending rate based on the number of participants. (4) Send Session control information. This is an optional function if the user interface displays the identity of the participants. When RTP/RTCP is working, the RTP protocol receives streaming media information streams (such as H.263) from the upper layer and assembles them into RTP data packets and sends them to the lower layer. The lower layer protocol provides RTP and RTCP shunting. For example, in UDP, RTP uses an even port number, then the corresponding RTCP uses its odd port number. RTP data packets have no length limit. The maximum packet length is limited only by the lower-layer protocol. Two levels of RTP sessions and streams use one RTP session (Session) to include all the traffic sent to a specific destination pair (destination pair). The sender may include multiple. The RTP group sequence sent from the same synchronization source is called stream. a rtp session may contain multiple RTP streams. When sending an RTP group to the server, you always need to specify the session and stream to which the Group belongs. When receiving the group, you also need to use two levels, that is, session and stream. Only when RTP uses the synchronous source identification (SSRC) and the group type (ptype) to combine the groups in the same stream can we use the serial number and timestamp) sort the groups and play back the groups correctly. Due to the uniqueness of real-time data, different real-time customers can share one RTP real-time service thread and one RTCP Real-time service thread, which can greatly reduce the burden on the server, each file client may have different speed and start time requirements due to different requested files. Therefore, they must have their own RTP file service thread and RTCP file service thread. The RTP service thread is responsible for sending real-time data streams to the customer. The RTCP service thread generates the sender report to the customer based on the statistics of the RTP thread. When the RTP thread and the RTCP thread share memory interaction statistics, the shared memory must be protected by mutex to prevent incorrect read/write. In this way, the server can provide different services conveniently based on different requests and specific conditions of each user. The processing Timestamp Field of the RTP timestamp is the synchronization information of the packet time described in the RTP Header, which is the key to data recovery in the correct chronological order. The value of the timestamp gives the sampling instant of the first byte of the data in the group. It requires that the clock of the sender's timestamp grow continuously and monotonically, this is true even when no data is input or sent. When the message is silent, the sender does not have to send data and keep the timestamp increasing. at the receiving end, because the serial number of the received data group is not lost, no data loss occurs, in addition, the output interval can be determined by comparing the difference in the timestamp between the group and the group. RTP specifies that the initial timestamp of a session must be randomly selected, but the Protocol does not specify the unit of the timestamp, and does not specify the exact interpretation of the value, but determines the clock particles by the load type, in this way, you can select an appropriate output timing accuracy for various application types as needed. When RTP is used to transmit audio data, the logical timestamp rate is usually the same as the sampling rate. However, when transmitting video data, the timestamp rate must be greater than a tick of each frame. If the data is sampled at the same time, the protocol standard also allows multiple groups to have the same timestamp value. The RTP protocol does not specify the length of the RTP group and the speed at which data is sent. Therefore, you need to adjust the speed at which the server sends media data according to the specific situation. The real-time data from the device can be accessed to the device buffer at an equal time interval. When new data is input, the data is sent. The time stamp setting is relatively easy. For a media file in H.263 format on a recorded local hard disk, as an example, because the file itself does not contain the frame rate information, you need to know the frame rate during recording or set an initial value, when sending data, find the number of frames in the sent data, calculate the latency Based on the frame rate and preset value, send data at an appropriate speed, and set the timestamp information. A key role of RTCP is to allow the receiver to synchronize multiple RTP streams. For example, when audio and video are transmitted together, due to different encoding, RTP uses two streams for transmission separately, in this way, the timestamps of the two streams run at different rates, and the receiver must synchronize the two streams to ensure that the sound is consistent with the image. For stream synchronization, RTCP requires the sender to send each canonical name that uniquely identifies the data source ), although different streams sent from a data source have different synchronization source identifiers (SSRC), but they have the same canonical names, the receiver knows which streams are associated. The sender reports that the information contained in the message can be used by the receiver to coordinate the timestamp values of the two streams. The sender report contains an absolute time value in the Network Time Protocol NTP (Network Time Protocol) format. Then, An RTP timestamp value is provided in the RTCP report, the clock that generates this value is the one that generates the timestamp field in the RTP group. Since both the stream and the report sent by the sender use the same absolute clock, the receiver can compare the absolute time of the two streams from the same data source, to determine how to map the timestamp value in a stream to the timestamp value in another stream. There are two types of server algorithm server software models: Concurrent Server and cyclic server. An iterative Server is a server that processes only one request at a time. A Concurrent Server is a server that can process multiple requests at a time. In fact, most servers do not have redundant devices to process multiple requests at the same time, but provide a superficial concurrency by executing multiple threads and each thread processes one request, from the customer's perspective, the server is like communicating with multiple customers concurrently. Due to the uncertainty of streaming media service time and real-time data interaction requests, streaming media servers generally adopt Concurrent Server algorithms. This article builds a basic streaming media server that can respond to requests from multiple users at the same time and send the streaming media files or real-time data streams on the local hard disk (H.263 format) to users. In applications, customers are divided into real-time customers requesting real-time data and file customers requesting file data. The main algorithm is: (1) Open the device and allocate resources. When the device is ready, create an RTP real-time service thread and an RTCP Real-time service thread. (2) create a UDP socket and bind it to the address of the provided service. (3) Call the receiving module repeatedly to receive RTCP reports from customers and respond to the reports based on their types. For new real-time customer requests, add the customer address to the real-time service customer list, for new file customer requests, create a new RTP file service thread and a new RTCP file service thread, and adjust the service according to the content of the RTCP report for customers who are already in the service. RTP real-time service thread 1: Initialize the customer list and RTP Header. RTP real-time service thread 2: reads media data from devices and sends the data to customers in the real-time service list. RTP real-time service thread 3: update the RTP Header and statistical data. RTP real-time service thread 4: computing latency, repeat Step 2. RTCP Real-time service thread 1: Initialize the RTCP header. RTCP Real-time service thread 2: The sender reports to customers in the real-time service list. RTCP Real-time service thread 3: computing latency, repeat Step 2. RTP file service thread 1: Initialize the RTP Header. RTP file service thread 2. Read media data from the file and send the data to the customer. RTP file service thread 3: update the statistics of sent data to prepare for generating the sender report. RTP file service thread 4: Calculate the latency, adjust the sending speed, and start to repeat Step 2 normally. RTCP file service thread 1: Initialize the RTCP header and send a source description (sdes) packet to the customer. RTCP file service thread 2: generate a sender report based on the statistics of sent data and send it to the customer. RTCP file service thread 3: computing latency. Normally, repeat the first step.