RTP network protocol Implementation of RTP---H264 of network protocol

The complete C/s architecture is based on rtp/rtcp H. 264 video transmission scheme. In this scenario, the functional module design is performed on the server side and the client respectively. Server-side: RTP Encapsulation module is mainly for H. 264-stream package; The RTCP Analysis module is responsible for producing cattle and sending RTCP packets and analyzing the received RTCP packets; The QoS Feedback control module adjusts the transmit rate according to the RR message feedback information; The transmit buffer module sets the port to send RTP, RTCP packets. Client: The RTP module is connected to the received RTP packet to parse and judge; the RTCP module produces cattle and sends RR packets according to the SR message statistics key information. Then, in Vc++6. 0 under the socket programming, complete the RTP/UDP/IP-based H. 264 video transmission, and in the LAN operating better.
Based on the RTP/UDP/LP H. 264 video transmission structure design
For H. 264 video real-time transmission applications, TCP retransmission mechanism introduced by the delay and jitter is not tolerated, so we adopt the UDP transport protocol. But the UDP protocol itself is non-connected and does not provide quality assurance. The high-level protocol rtp/rtcp based on UDP can provide traffic control and congestion control services together. The graph gives a RTP/UDP/IP-based H. 264 frame for video transmission.

H. 264 RTP encapsulation strategy for video streams
As can be seen from Figure 4-1, H. 264 video data is first encapsulated by RTP and packaged into a network-appropriate packet for transmission. So, how to design the appropriate RTP encapsulation strategy for H. 264 video data encapsulation is very important. In general, in H. 264, the RTP package should follow several design principles: 1, lower overhead, so the size of the MTU should be limited to the 100-64k byte range. 2, it is easy to distinguish the importance of grouping, without having to decode the data within the grouping. 3, should be able to detect the type of data, without the need to decode the entire data stream, and can be based on the correlation between the encoding stream discard useless data, such as the gateway should be able to detect the loss of a-type segmentation, and can discard the corresponding B-type and C-type segmentation.
4, should support a Nalu split into a number of RTP packets: different size of the input image determines that the length of the Nalu may be greater than the MTU, only after splitting to avoid the IP layer in the transmission of the Shard. 5, support the aggregation of multiple Nalu in a RTP packet, that is, in a RTP package to transmit more than one nalu, when the encoding output of multiple images is less than M1iu to consider this mode, in order to improve network transmission efficiency.
RTP Load Package Design
The network transmission of this article is based on IP protocol, so the maximum transmission Unit (MTU) is 1500 bytes, when using the IP/UDP/RTP protocol hierarchy, this includes at least 20 bytes of IP header, 8 bytes of UDP header, and 12 bytes of RTP header. This way, the header information takes at least 40 bytes, and the maximum size of the RTP payload is 1460 bytes.

On the one hand, if each IP packet is filled with 1500 bytes, the overhead of the protocol header is 2. 7%, if the RTP payload is 730 bytes long, the protocol header's overhead is still up to 5. 3%, assuming that the RTP payload is less than 40 bytes in length, then 50% of the overhead will be spent on the head, which will cause serious resource waste to the network. On the other hand, if the data to be encapsulated into the RTP payload is greater than 1460 bytes, and we do not perform a load split before the application layer data is loaded with the burst RTP packet, a packet larger than the MTU will be produced. At the IP layer it will be split into a few packets smaller than the MTU size, which will not detect the loss of data. Because neither the IP nor the UDP protocol provides packet arrival detection, if the first packet is successfully received and subsequent packets are lost because only the first package contains the complete RTP header information, and the RTP header does not identify the payload length, it is not possible to determine whether the RTP packet has a split loss, Can only be considered as complete received. and the segmentation of IP layer can not be protected in the application layer, thus reducing the effect of non-equal inclusion scheme. Because the UDP data packet is less than 64K bytes, and the length of a piece is a bit too small for some applications, the packaging of the application layer is also a necessary part of the RTP packaging mechanism. The latest RFC3984 standards are available for H. 246 the RTP payload format of media stream is mainly three kinds: single nal unit grouping, aggregation grouping, slice grouping.
NAL Unit Single Pack
Encapsulates a NAL unit into a package, which means that the RTP payload contains only one NAL unit, and the NAL head doubles as the RTP head. The RTP header type is NAL unit type 1-23, as shown in:

Reorganization of the NAL unit this grouping type is used to aggregate multiple nal cells in one RTP group. Some h. The size of the NAL unit of 264, such as SEI nal Unit, parameter set, etc. are very small, some only a few bytes, so
They should be combined into one RTP packet, which will help reduce the overhead of the header (RTP/UDP/IP). There are currently two types of aggregation groupings:
Division of NAL Units
Splits a nal cell and transmits it using multiple RTP groupings. There are two types Fu-a and fu-b, and the cell types are 28 and 29, respectively. According to the IP layer MTU size, the large size of the Nalu must be split, can be divided in two levels: 1) Video coding layer on the VCL partition
To accommodate the size of the network MTU, the encoder can be used to select the size of the encoded slice Nalu, which provides better performance. In general, the size of the encoded slice is adjusted so that it is less than 1460 bytes, so as not to divide the IP layer.
2) The segmentation of the network extraction layer NAL on the network extraction layer on the Nalu partition is mainly using the Shard unit scheme, H. The partition mechanism is proposed in the 264 standard to make the size of the NAL unit smaller than 1460 bytes. Note: This method is split for the same NAL unit and does not apply to aggregation groupings. After a nal unit is segmented, each RTP packet sequence number is incremented L,RTP the timestamp is the same and unique. The segmentation of NAL unit is an important part of RTP packaging mechanism, and the main features of the segmentation mechanism are as follows: ① division Nalu, it is to be transferred in ascending order of RTP ordinal. Under the premise that the serial number is not cyclic, all the image packets belonging to the previous frame image and the sequence number of the A/B/C data splitting packet are smaller than the sequence number of the image slice and the data splitting packet in the back frame image. ② a symbolic mechanism to mark whether a segmented Nalu is the first or last NAL unit. 3. There is another symbolic mechanism used to detect the loss of a block. ④ Auxiliary Enhanced packets and header packets can be sent at any time.
⑤ images in the same frame can be sent in any order, but for low-latency network systems, it is best to send them in their original coding order.
1) Single Time aggregation grouping (STAP): Includes single time aggregation group A (STAP-A) and single time aggregation group B (stap-b), combined by timestamps, their nal units have the same timestamp and are generally used for low latency environments. The unit types for Stap-astap-b are 24 and 25, respectively. 2) Multi-time aggregation grouping (MTAP): includes 16-bit offset multi-time aggregation groupings (MTAPL6) and 24-bit offset multi-time aggregation groupings (MTAP24) can be combined with different timestamps, typically for high-latency network environments, such as streaming media applications. Its packaging scheme is relatively complex, but it greatly enhances the H-based streaming media. 264 performance. The unit types for MTAPl6 MTAP24 are 26 and 27, respectively.
Packaging process design of RTP Package
According to H. 264NAL unit of the Nature of the Division and Reorganization and RTP packaging rules, the implementation of the RTP packaging design is as follows: 1, if the received NAL unit is less than max-size (at this time the max-size is the largest transmission unit set), it is a single package, That is, the NAL unit is placed directly into the payload portion of the RTP packet, generating a RTP packet. 2, if the received NAL unit is greater than max-size bytes, then it is segmented, and then the segmented NAL unit is packaged in step 1 ways. The partitioning scheme is as follows:

Where nsize is the NAL element size before splitting, n is the size of the NAL unit after splitting. The number of cells after the K split. The size of the last cell after the split may be less than N, when the RTP payload must be filled with the same size as the previous tile, at which point the padding identity bit value in the RTP header is 1.
3, the SEI, the parameter set and other small NAL unit reorganization, merging them into a RTP packet. Although the reorganization scheme in step 3 can reduce the IP/UDP/RTP head overhead, but for the higher packet loss rate of the network environment, which means that a loss of a RTP packet may lead to the loss of multiple pieces, often a piece of a p image, decoded video quality will inevitably seriously decline. Therefore, the NAL unit reorganization scheme can be used in the loss rate network, and the effective error control should be made when the NAL unit is reconstructed in the network environment with high loss rate. The reorganization scheme is not used in this article.
Package implementation of the RTP/RTCP package
RTP Package Package Design

Package Design of RTCP package
The RTCP message is encapsulated in a UDP datagram and is sent with a protocol number that is 1 larger than the port number of the RTP stream to which it belongs (RTP uses an even number, and RTCP uses an odd number). The following is the RTCP header data structure:

Http://www.rosoo.net/a/201108/14896.html

RTP network protocol Implementation of RTP---H264 of network protocol