Research on Stream Media Server Technology Based on RTP/RTCP

With the rapid development of the Internet, streaming media technology has become more and more widely used. streaming media technology has been used for broadcasting on the internet, playing movies, remote teaching, and online news websites. However, most of the information reported in the existing public documents is the use of the existing Streaming Media Server to build a streaming media service system, or the research on the encoding method of streaming media data. This article focuses on how to establish a server and build a basic streaming media server based on the two basic protocols RTP/RTCP for streaming media transmission.

2 Introduction to Streaming Media Technology
2.1 Definition of "stream"
Currently, video and audio are transmitted online in two ways: Download and streaming. Streaming Transmission continuously transmits video/audio signals. When streaming media is played on the client, the remaining parts continue to be downloaded in the background. Streaming can be either a progressive streaming or a realtime streaming. Real-Time Streaming Transmission is a real-time transfer, which is especially suitable for on-site events. Real-Time Streaming Transmission must match the connection bandwidth, which means that the image quality will deteriorate due to reduced network speed to reduce the need for transmission bandwidth. The concept of "real-time" means that the delivery of data in an application must maintain a precise temporal relationship with the generation of data.

Streaming Media, which uses Stream Transmission Technology on the Internet, is called streaming media. Its video and audio are also called video streams and audio streams. Dedicated servers and players are generally required for stream transmission.

2.2 streaming media system components
Streaming media is composed of various software. These software communicate with each other at different layers. The basic streaming media system includes the following three components:
Player, used to play streaming media.

Server is used to send streaming media software to users.

Encode is used to convert the original audio and video into a streaming media format.

These components communicate with each other through a specific protocol and exchange file data in a specific format. Some files contain data decoded by a specific decoder, which compresses the data volume of a file through a specific algorithm.

3. basic functions and service methods of streaming media servers
3.1 main functions of streaming media servers
(1) Send media data to the customer in response to the customer's request. The Streaming Media Server must maintain two-way communication with the customer's player during streaming media transmission (this communication is required because the customer may pause or release a file at any time ).
(2) The newly received real-time broadcast data can be processed in a timely manner in response to the broadcast and encoded.
(3) other additional functions, such as digital permission management (DRM), Ad insertion, splitting or mirroring other server streams, and multicast.

3.2 Streaming Media Server Service Mode
(1) unicast. A separate data channel is established between the client and the Media Server. Each packet sent from one server can only be transmitted to one client.
(2) multicast. In a network built using Multicast technology, the router is allowed to copy data packets to multiple channels at a time.
(3) On-demand video and broadcast. The VOD connection is an active connection between the client and the server. In the VOD connection, you can select a content item to initialize the client connection. You can start, stop, back, fast forward, or pause a stream. Broadcast means that the user passively receives a stream. During the broadcast process, a separate copy of the data packet will be sent to all users on the network. The client receives the stream, but cannot control the flow.

4. Construct a Streaming Media Server
4.1rtp/RTCP
Real-time transmission protocol RTP (Realtime Transport Protocol) is a transmission protocol for multimedia data streams over the Internet. It is published by IETF (Internet Engineering Task Group) as rfc1889. RTP is defined to work during one-to-one or one-to-many transmission. It aims to provide time information and implement stream synchronization. Typical RTP applications are built on UDP, but can also work on TCP, ATM, and other protocols. RTP itself only guarantees the transmission of real-time data. It does not provide a reliable transmission mechanism for transmitting data packets in sequence, nor provides traffic control or congestion control. It relies on RTCP to provide these services.

Real-time transmission control protocol RTCP (Realtime Transport Control Protocol): manages the exchange of control information between transmission quality in the current application process. During the RTP session, each participant periodically transmits the RTCP packet, which contains statistics such as the number of sent packets and the number of lost packets. Therefore, the server can use this information to dynamically change the transmission rate or even the payload type. RTP and RTCP can be used together to optimize transmission efficiency with effective feedback and minimal overhead. Therefore, RTP is especially suitable for transmitting real-time online data.

RTCP has four functions:
(1) feedback is used to provide the transmission quality of distributed data. This feedback can be used to control traffic congestion, monitor the network, and diagnose problems in the network;
(2) provide a permanent cname (canonical name) transfer layer flag for the RTP source, because SSRC (synchronous source ID) changes when a conflict is found or the program is updated and restarted, an operation trace is required. In a group of related sessions, the receiver also uses the cname to obtain the associated data stream (such as audio and video) from a specified participant );
(3) Adjust the RTCP packet sending rate based on the number of participants;
(4) Send Session control information. This is an optional function if the user interface displays the ID of the participant.

4.2rtp/RTCP Workflow
During work, the RTP protocol receives streaming media information streams (such as H.263) from the upper layer and assembles them into RTP data packets and sends them to the lower layer. The lower layer protocol provides RTP and RTCP shunting. For example, in UDP, RTP uses an even port number, then the corresponding RTCP uses its odd port number. RTP data packets have no length limit. The maximum packet length is limited only by the lower-layer protocol.

4.3 server Algorithms
There are two types of server software models: Concurrent Server and cyclic server. An iterative Server is a server that processes only one request at a time. A Concurrent Server is a server that can process multiple requests at a time. In fact, most servers do not have redundant devices to process multiple requests at the same time, but provide a superficial concurrency by executing multiple threads and each thread processes one request, from the customer's perspective, the server is like communicating with multiple customers concurrently.

Due to the uncertainty of streaming media service time and real-time data interaction requests, streaming media servers generally adopt Concurrent Server algorithms. This article builds a basic streaming media server that can respond to requests from multiple users at the same time and send the streaming media files or real-time data streams on the local hard disk (H.263 format) to users. In applications, customers are divided into real-time customers requesting real-time data and file customers requesting file data. The main algorithms are:

(1) Open the device and allocate resources. When the device is ready, create an RTP real-time service thread and an RTCP Real-time service thread.
(2) create a UDP socket and bind it to the address of the provided service.
(3) Call the receiving module repeatedly to receive RTCP reports from customers and respond to the reports based on their types. For new real-time customer requests, add the customer address to the real-time service customer list, for new file customer requests, create a new RTP file service thread and a new RTCP file service thread, and adjust the service according to the content of the RTCP report for customers who are already in the service.

RTP real-time service thread 1: Initialize the customer list and RTP Header.
RTP real-time service thread 2: reads media data from devices and sends the data to customers in the real-time service list.
RTP real-time service thread 3: update the RTP Header and statistical data.
RTP real-time service thread 4: computing latency, repeat Step 2.

RTCP Real-time service thread 1: Initialize the RTCP header.
RTCP Real-time service thread 2: The sender reports to customers in the real-time service list.
RTCP Real-time service thread 3: computing latency, repeat Step 2.

RTP file service thread 1: Initialize the RTP Header.
RTP file service thread 2. Read media data from the file and send the data to the customer.
RTP file service thread 3: update the statistics of sent data to prepare for generating the sender report.
RTP file service thread 4: Calculate the latency, adjust the sending speed, and start to repeat Step 2 normally.

RTCP file service thread 1: Initialize the RTCP header and send a source description (sdes) packet to the customer.
RTCP file service thread 2: generate a sender report based on the statistics of sent data and send it to the customer.
RTCP file service thread 3: computing latency. Normally, repeat the first step.

5. considerations for implementing streaming media servers
5.1 split session and stream
A rtp session includes all the traffic transmitted to a specific destination pair (destination pair), and the sender may include multiple. The RTP group sequence sent from the same synchronization source is called stream. a rtp session may contain multiple RTP streams. When sending an RTP group to the server, you always need to specify the session and stream to which the Group belongs. When receiving the group, you also need to use two levels, that is, session and stream. Only when RTP uses the synchronous source identification (SSRC) and the group type (ptype) to combine the groups in the same stream can we use the serial number and timestamp) sort the groups and play back the groups correctly.

5.2 multi-thread management
The concurrent server mode requires multithreading to provide services, so multithreading management is very important. In the server built in this article, requests and feedback from different customers are processed by the main thread of the server. Due to the uniqueness of real-time data, different real-time customers can share one RTP real-time service thread and one RTCP Real-time service thread, which can greatly reduce the burden on the server, and each file client varies with the requested file, the corresponding requirements for speed and start time may be different, so you need to have your own unique RTP file service thread and RTCP file service thread.

The RTP service thread is responsible for sending real-time data streams to the customer. The RTCP service thread generates the sender report to the customer based on the statistics of the RTP thread. When the RTP thread and the RTCP thread share memory interaction statistics, the shared memory must be protected by mutex to prevent incorrect read/write. In this way, the server can provide different services conveniently based on different requests and specific conditions of each user.

5.3 timestamp Processing
The timestamp field is the synchronization information of the packet time described in the RTP Header, and is the key to data recovery in correct time sequence. The value of the timestamp gives the sampling instant of the first byte of the data in the group. It requires that the clock of the sender's timestamp grow continuously and monotonically, this is true even when no data is input or sent. When the message is silent, the sender does not have to send data and keep the timestamp increasing. at the receiving end, because the serial number of the received data group is not lost, no data loss occurs, in addition, the output interval can be determined by comparing the difference in the timestamp between the group and the group.

RTP specifies that the initial timestamp of a session must be randomly selected, but the Protocol does not specify the unit of the timestamp, and does not specify the exact interpretation of the value, but determines the clock particles by the load type, in this way, you can select an appropriate output timing accuracy for various application types as needed.

When RTP is used to transmit audio data, the logical timestamp rate is usually the same as the sampling rate. However, when transmitting video data, the timestamp rate must be greater than a tick of each frame. If the data is sampled at the same time, the protocol standard also allows multiple groups to have the same timestamp value.

5.4 control the transmission speed of media data
Since the RTP protocol does not specify the length of the RTP group and the speed at which data is sent, you need to adjust the speed at which the server sends media data according to the specific situation. The real-time data from the device can be accessed to the device buffer at an equal time interval. When new data is input, the data is sent. The time stamp setting is relatively easy. For a media file in H.263 format on a recorded local hard disk, as an example, because the file itself does not contain the frame rate information, you need to know the frame rate during recording or set an initial value, when sending data, find the number of frames in the sent data, calculate the latency Based on the frame rate and preset value, send data at an appropriate speed, and set the timestamp information.

More than 5.5 stream synchronization
A key role of RTCP is to allow the receiver to synchronize multiple RTP streams. For example, when audio and video are transmitted together, due to different encoding, RTP uses two streams for transmission separately, in this way, the timestamps of the two streams run at different rates, and the receiver must synchronize the two streams to ensure that the sound is consistent with the image. For stream synchronization, RTCP requires the sender to send each canonical name that uniquely identifies the data source ), although different streams sent from a data source have different synchronization source identifiers (SSRC), but they have the same canonical names, the receiver knows which streams are associated. The sender reports that the information contained in the message can be used by the receiver to coordinate the timestamp values of the two streams. The sender report contains an absolute time value in the Network Time Protocol NTP (Network Time Protocol) format. Then, An RTP timestamp value is provided in the RTCP report, the clock that generates this value is the one that generates the timestamp field in the RTP group. Since both the stream and the report sent by the sender use the same absolute clock, the receiver can compare the absolute time of the two streams from the same data source, to determine how to map the timestamp value in a stream to the timestamp value in another stream.

6 conclusion
Streaming Media Technology is widely used and has great practical significance for the study of streaming media technology. This paper analyzes the general functions and structure of the Streaming Media Server by studying the RTP/RTCP protocol, the Implementation Scheme for building a basic Streaming Media Server is provided. The experiment proves that it can meet the requirements of multiple real-time and file customers at the same time and has been applied to a remote monitoring system.