A video transmission system based on RTP protocol: Realization

Implementation principle: RTP protocol-based video transmission system: Principle

Related articles:

"1" RTP protocol analysis

Introduction to "2" jrtplib

"3" QT call Jrtplib for unicast, multicast, and broadcast

"4" RTP Payload (payload) type, RTP Payload type

Production of "5" (H264) video files

"6" Format analysis

"7" H-Video compression standard

About RTP Payload Format for H + + video Be sure to refer to document RFC6184 because rfc3984 has been deprecated and rfc6184 can be opened from the following two links.

https://tools.ietf.org/html/rfc6184

Https://datatracker.ietf.org/doc/rfc6184/?include_text=1

The RTP payload defines three different basic load structures for H. A, and the receiving end may identify them by the first byte of the RTP payload. This byte is similar to the format of the Nalu header, and its Type field indicates which structure is represented, and the structure of this byte is as follows:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| F|  nri| Type |
+---------------+

The type is defined as follows:
0 Not defined
1-23 nal Unit single NAL unit package.
Stap-a Single-time combo package
Stap-b Single-time combo pack
MTAP16 Multi-time combo pack
MTAP24 Multi-time combo pack
Fu-a Shards of cells
Elements of Fu-b shards
30-31 Not defined

The difference between the first byte type field and the Type field in H. Nalu header is that when the value of type 24~31 indicates that this is a special format of the NAL unit, and that only the 1~23 is a valid value in H. These three types of load structures are described separately below.

I. Single Nalu Packet (singleton nal unit mode)
That is, an RTP payload consists only of the first byte and one Nalu load, and in this article the package is used for Nalu less than 1400 bytes. In this case, the ere byte Type field is the same as the original H. Nalu Header Type field. In other words, in this case the RTP payload is a complete nalu.

Two. Aggregation Packet (Combo packet mode)
Encapsulating multiple Nalu in one RTP, this packaging scheme can be used for smaller nalu, thus improving transmission efficiency. This could be a RTP package consisting of multiple Nalu. There are 4 combinations of stap-a, Stap-b, MTAP16 and MTAP24 respectively. Then the RTP payload first byte type values are 24, 25, 26, and 27, respectively. This model is not covered in this article.

Three. Fragmentation Units (Shard packet mode fus)
A NALU is encapsulated in multiple RTP, with each RTP payload consisting of the first byte (which is actually the FU indicator, but it is the same as the structure of the original byte, which is still called the first byte), the FU header, and a portion of the NALU payload. In this article, for Nalu larger than 1400 bytes, this scenario is used for unpacking. There are two types of fu-a and fu-b, and the type values are 28 and 29, respectively.

The fu-a type is shown in the following figure:

This article uses the fu-a type.

The Fu-b type is shown in the following figure:

Compared to Fu-a, Fu-b has one more don (decoding order number), and Don uses the network byte order. The fu-b can only be used in interlaced packet mode and cannot be used in other ways.

The FU indicator byte structure is as follows:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| F|  nri| Type |
+---------------+

type=28 or 29

The FU header byte structure is as follows:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| s| e|  r| Type |
+---------------+

S (start): 1 bit, when set to 1, this bit indicates the start of the Shard nal unit. When the subsequent FU load is not the beginning of the Shard nal unit, the bit is set to 0.
E (End): 1 bit, when set to 1, this bit indicates the end of the Shard nal unit, at which point the last byte of the load is the last byte of the Shard nal cell. When the subsequent FU load is not the end of the Shard nal unit, the bit is set to 0.
R (Reserved): 1 bit, the reserved bit must be set to 0, and the receiver must ignore the bit.

Type: Same as type value in Nalu header

four. Send RTP packaged H264 using socket sockets

The main code is as follows:

int main () {Openbitstreamfile ("480320.264");
    nalu_t *n;

    Char sendbuf[1500];
    unsigned short seq_num = 0;
    int bytes=0; Initwinsock ();
    Initializes the socket font int SOCKFD;
    struct sockaddr_in addr_in;
    float framerate=25;
    unsigned int timestamp_increse=0,ts_current=0;

    timestamp_increse= (unsigned int) (90000.0/framerate);
    Addr_in.sin_family=af_inet;
    Addr_in.sin_port=htons (Dest_port);
    ADDR_IN.SIN_ADDR.S_ADDR=INET_ADDR (DEST_IP);
    Sockfd=socket (af_inet,sock_dgram,0); Connect (SOCKFD, (const SOCKADDR *) &addr_in, sizeof (SOCKADDR_IN)),//request UDP Socket n = Allocnalu (8000000);//For Structure nalu_t and its Member Buf allocate space. The return value is a pointer to nalu_t storage space while (!feof (bits)) {Getannexbnalu (n);//each time the file pointer points to the end of the Nalu found, the next position is the starting code for the next Nalu 0x

        000001 dump (n);//output Nalu length and type memset (sendbuf,0,1500);//empty sendbuf; the last timestamp is emptied, so ts_current is required to save the last timestamp value
        The RTP fixed header, which is 12 bytes, assigns the address of sendbuf[0] to RTP_HDR, and subsequent writes to RTP_HDR are written directly to SendBuf. RTP_HDR = (rtp_fixed_header*) &sendbuf[0];   Set RTP HEADER rtp_hdr->version = 2;   Version number, this version is fixed to 2 rtp_hdr->marker = 0;
        The value of the symbol, specified by the specific agreement. Rtp_hdr->payload = h264;//Load type number, RTP_HDR-&GT;SSRC = htonl (10),///randomly specified as 10, and globally unique in this RTP session//When a NALU is less than 1 400 bytes, use a single RTP packet to send if (n->len<=max_rtp_pkt_length) {//set RTP M bit; rtp_hdr-&
            Gt;marker=1; Rtp_hdr->seq_no = htons (seq_num + +);
            Serial number, each sending a RTP packet increased by 1 Ts_current=ts_current+timestamp_increse;
            Rtp_hdr->timestamp=htonl (ts_current); Set the Nalu header and fill in the header with sendbuf[12] Nalu_hdr = (nalu_header*) &sendbuf[12];
            The address of the sendbuf[12] is assigned to NALU_HDR, and the write to Nalu_hdr is then written to SendBuf; nalu_hdr->f=n->forbidden_bit; nalu_hdr->nri=n->nal_reference_idc>>5;//valid data in the 6th, 7 bits of N-&GT;NAL_REFERENCE_IDC, need to move 5 bits to the right to assign its value to nalu_hdr-
            >nri.

            nalu_hdr->type=n->nal_unit_type; MemcpY (&sendbuf[13],n->buf+1,n->len-1);//Remove the Nalu header Nalu the remainder of the content written to sendbuf[13] start string.  Bytes=n->len + 12; Get the length of the sendbuf, for the length of the Nalu (including the Nalu header but remove the starting prefix) plus the fixed length of Rtp_header 12 bytes if (n->nal_unit_type==1 | | n->nal_unit_type=
                =5) {send (sockfd,sendbuf,bytes,0);//Send RTP Package} else { Send (sockfd,sendbuf,bytes,0);//sends the RTP packet//If it is a 6,7 type of packet, should not delay; there was a pause before, cause this Contin
            Ue
            }} else {int packetnum = n->len/max_rtp_pkt_length;

            if (n->len%max_rtp_pkt_length! = 0) Packetnum + +;
            int lastpacksize = N->len-(packetNum-1) *max_rtp_pkt_length;

            int packetindex = 1;
            Ts_current=ts_current+timestamp_increse;

            Rtp_hdr->timestamp=htonl (ts_current); Send the first fu,s=1,e=0,r=0 rtp_hdr->seq_no = htons (seq_num + +);
Serial number, each sending a RTP packet 1 increase            Set RTP M bit; rtp_hdr->marker=0; Set the FU INDICATOR and fill this header in sendbuf[12] Fu_ind = (fu_indicator*) &sendbuf[12];
            The address of the sendbuf[12] is assigned to Fu_ind, and the write to Fu_ind is then written to SendBuf; fu_ind->f=n->forbidden_bit;
            fu_ind->nri=n->nal_reference_idc>>5; fu_ind->type=28;//use FU-A//To set up the FU header and fill in the header with sendbuf[13] Fu_hdr = (fu_header*) &sendbu
            F[13];
            fu_hdr->s=1;
            fu_hdr->e=0;
            fu_hdr->r=0;

            fu_hdr->type=n->nal_unit_type; memcpy (&sendbuf[14],n->buf+1,max_rtp_pkt_length);//Remove Nalu head bytes=max_rtp_pkt_length+14;//get sendbuf length, for

            The length of the Nalu (except the starting prefix and the NALU header) plus the fixed length of Rtp_header,fu_ind,fu_hdr 14 bytes Send (sockfd, sendbuf, Bytes, 0);//Send RTP Packets
                Send intermediate fu,s=0,e=0,r=0 for (packetindex=2;packetindex<packetnum;packetindex++) { Rtp_hdr->seq_no = Htons (seq_num + +);
                Serial number, each sending a RTP packet increase 1//Set RTP M bit; rtp_hdr->marker=0; Set the FU INDICATOR and fill this header in sendbuf[12] Fu_ind = (fu_indicator*) &sendbuf[12];
                The address of the sendbuf[12] is assigned to Fu_ind, and the write to Fu_ind is then written to SendBuf; fu_ind->f=n->forbidden_bit;
                fu_ind->nri=n->nal_reference_idc>>5;

                fu_ind->type=28;
                Set the FU header and fill in the header with sendbuf[13] Fu_hdr = (fu_header*) &sendbuf[13];
                fu_hdr->s=0;
                fu_hdr->e=0;
                fu_hdr->r=0;

                fu_hdr->type=n->nal_unit_type; memcpy (&sendbuf[14],n->buf+ (packetIndex-1) *max_rtp_pkt_length+1,max_rtp_pkt_length);//
                The string that starts with the Nalu remaining content of the starting prefix written to sendbuf[14] is removed. bytes=max_rtp_pkt_length+14;//gets the length of the sendbuf, for the length of the Nalu (except the original Nalu head) plus rtp_header,fu_ind,fu_hdr fixed length of 14 bytes send (Soc
 kfd, SendBuf, Bytes, 0);//Send RTP Packets           }//Send the last fu,s=0,e=1,r=0 rtp_hdr->seq_no = htons (seq_num + +);
            Sets the RTP M bit, which is the last shard that is currently being transmitted when the position is 1 rtp_hdr->marker=1; Set the FU INDICATOR and fill this header in sendbuf[12] Fu_ind = (fu_indicator*) &sendbuf[12];
            The address of the sendbuf[12] is assigned to Fu_ind, and the write to Fu_ind is then written to SendBuf; fu_ind->f=n->forbidden_bit;
            fu_ind->nri=n->nal_reference_idc>>5;

            fu_ind->type=28;
            Set the FU header and fill in the header with sendbuf[13] Fu_hdr = (fu_header*) &sendbuf[13];
            fu_hdr->s=0;
            fu_hdr->e=1;
            fu_hdr->r=0;

            fu_hdr->type=n->nal_unit_type; memcpy (&sendbuf[14],n->buf+ (packetIndex-1) *max_rtp_pkt_length+1,lastpacksize-1);//Will nalu the last remaining 1 (
            Removed a byte of the Nalu header) byte content written to Sendbuf[14] begins the string. bytes=lastpacksize-1+14;//gets the length of the SendBuf, minus 1 for the length of the remaining Nalu plus rtp_header,fu_indicator,fu_header three headers a total of 14 bytes send (SOCKFD, SenDbuf, Bytes, 0);//Send RTP Packet} Sleep (40);
} freenalu (n);
#ifdef WIN32 WSACleanup ();
#endif//WIN32 return 0; }

Five. Use Jrtplib to send RTP packaged H264
The advantage of using Jrtplib is that you do not need to add the RTP header yourself, and the RTCP protocol is implemented inside the jrtplib. Jrtplib's introduction can refer to "2", using the method can refer to "3".

The main code is shown below.

int main () {//Initialize jrtplib uint8_t destip[]={127,0,0,1};
    INITIALRTP (Destip);
    Open 264 file Openbitstreamfile ("480320.264");
    nalu_t *n;
    Char sendbuf[1500];

    int bytes=0; n = Allocnalu (8000000);//Allocates space for structure nalu_t and its member buf. The return value is a pointer to nalu_t storage space while (!feof (bits)) {Getannexbnalu (n);//each time the file pointer points to the end of the Nalu found, the next position is the starting code for the next Nalu 0x 000001 dump (n);//output Nalu length and type memset (sendbuf,0,1500);//Empty SENDBUF//When a NALU is less than 1400 bytes, use a single RTP packet
            Send if (n->len<=max_rtp_pkt_length) {//Set Nalu header and fill this header in sendbuf[0] NALU_HDR = (nalu_header*) &sendbuf[0];
            The address of the sendbuf[0] is assigned to NALU_HDR, and the write to Nalu_hdr is then written to SendBuf; nalu_hdr->f=n->forbidden_bit; nalu_hdr->nri=n->nal_reference_idc>>5;//valid data in the 6th, 7 bits of N-&GT;NAL_REFERENCE_IDC, need to move 5 bits to the right to assign its value to nalu_hdr-
            >nri.

            nalu_hdr->type=n->nal_unit_type; memcpy (&sendbuf[1],n->buf+1,n->len-1);/Remove the string that begins with the Nalu remainder of the Nalu header written to sendbuf[1].
            bytes=n->len;  if (n->nal_unit_type==1 | | n->nal_unit_type==5) {//third parameter mark is the M-bit int in the RTP header Status = M_session.
                Sendpacket ((void *) sendbuf,bytes,96,true,3600);
                if (Status < 0) {std::cout<<rtpgeterrorstring (status) <<std::endl;
                }} else {//note this is different from sending using Socekt, at which time the timestamp does not increase int status = M_session.
                Sendpacket ((void *) sendbuf,bytes,96,true,0);
                if (Status < 0) {std::cout<<rtpgeterrorstring (status) <<std::endl;
            }//If it is a 6,7 type of package, it should not be delayed; before there is a pause, the reason for this continue;
            }} else {int packetnum = n->len/max_rtp_pkt_length; if (n->len%max_rtp_pkt_length! = 0) PackeTnum + +;
            int lastpacksize = N->len-(packetNum-1) *max_rtp_pkt_length;


            int packetindex = 1; Send the first fu,s=1,e=0,r=0 Fu_ind = (fu_indicator*) &sendbuf[0];
            The address of the sendbuf[0] is assigned to Fu_ind, and the write to Fu_ind is then written to SendBuf; fu_ind->f=n->forbidden_bit;
            fu_ind->nri=n->nal_reference_idc>>5; fu_ind->type=28;//use FU-A//To set up the FU header and fill in the header with sendbuf[1] Fu_hdr = (fu_header*) &sendbuf
            [1];
            fu_hdr->s=1;
            fu_hdr->e=0;
            fu_hdr->r=0;

            fu_hdr->type=n->nal_unit_type; memcpy (&sendbuf[2],n->buf+1,max_rtp_pkt_length);//Remove Nalu head bytes=max_rtp_pkt_length+2;//get sendbuf length, for NA The length of the LU (minus the starting prefix and the NALU header) plus the fixed length of fu_ind,fu_hdr 2 bytes int status = M_session.
            Sendpacket ((void *) sendbuf,bytes,96,false,0); if (Status < 0) {std::cout<<rtpgeterrorstring (status) <<std::endl;
                }//Send intermediate fu,s=0,e=0,r=0 for (packetindex=2;packetindex<packetnum;packetindex++) { Set the FU INDICATOR and fill this header in sendbuf[0] Fu_ind = (fu_indicator*) &sendbuf[0];
                The address of the sendbuf[0] is assigned to Fu_ind, and the write to Fu_ind is then written to SendBuf; fu_ind->f=n->forbidden_bit;
                fu_ind->nri=n->nal_reference_idc>>5;

                fu_ind->type=28;
                Set the FU header and fill in the header with sendbuf[1] Fu_hdr = (fu_header*) &sendbuf[1];
                fu_hdr->s=0;
                fu_hdr->e=0;
                fu_hdr->r=0;

                fu_hdr->type=n->nal_unit_type; memcpy (&sendbuf[2],n->buf+ (packetIndex-1) *max_rtp_pkt_length+1,max_rtp_pkt_length);//
                The string that starts with the Nalu remaining content of the starting prefix written to sendbuf[24] is removed. bytes=max_rtp_pkt_length+2;//get sendbuf length for Nalu length (except Nalu head) plus fu_ind,fu_hdr fixed length 2 byte int status = m_sEssion.
                Sendpacket ((void *) sendbuf,bytes,96,false,0);
                if (Status < 0) {std::cout<<rtpgeterrorstring (status) <<std::endl; }}//Set FU INDICATOR and fill this header in sendbuf[0] Fu_ind = (fu_indicator*) & SENDBUF[0];
            The address of the sendbuf[0] is assigned to Fu_ind, and the write to Fu_ind is then written to SendBuf; fu_ind->f=n->forbidden_bit;
            fu_ind->nri=n->nal_reference_idc>>5;

            fu_ind->type=28;
            Set the FU header and fill in the header with sendbuf[1] Fu_hdr = (fu_header*) &sendbuf[1];
            fu_hdr->s=0;
            fu_hdr->e=1;
            fu_hdr->r=0;

            fu_hdr->type=n->nal_unit_type; memcpy (&sendbuf[2],n->buf+ (packetIndex-1) *max_rtp_pkt_length+1,lastpacksize-1);//Will nalu the last remaining 1 (
            Removed a byte of the Nalu header) byte content written to Sendbuf[2] begins the string.
       bytes=lastpacksize-1+2;//get sendbuf length, minus 1 for the length of the remaining Nalu plus fu_indicator,fu_header two headers total 2 bytes     Status = M_session.
            Sendpacket ((void *) sendbuf,bytes,96,true,3600);
            if (Status < 0) {std::cout<<rtpgeterrorstring (status) <<std::endl;
    }} Sleep (40);

    } freenalu (n); M_session.
Byedestroy (Rtptime (10,0), 0,0);
#ifdef WIN32 WSACleanup ();
#endif//WIN32 return 0; }

Six. Testing

The test results are the same for both of these methods. Open VLC or mplayer player (strongly recommended MPlayer, better than VLC), drag W.SDP into the player, the player enters the wait state, then the program sends the RTP stream and the player starts playing.

The test uses 176144.264 and 480320.264 two H. 1400 files, the first resolution is 176*144,nalu smaller, no fragmentation is required for packaging, and the second resolution is 480*320,nalu greater than the number of a bit, and there is a case of fragmented packaging.

W.sdp

M=video 25000 RTP/AVP   
a=rtpmap:96 H264 a=framerate:25             
c=in IP4 127.0.0.1         

Note The port number 25000 is the same as set in the program.

The test results are as follows:

Seven. Summary

With regard to timestamps, it is important to note that the sampling rate for the h264 is 90000HZ, so the timestamp is in 1 (seconds)/90000, so if the current video frame rate is 25fps, then the timestamp interval or increment should be 3600, if the frame rate is 30fps, then the increment is 3000, and so on.
About H264 unpacking, according to the Fu-a Way description:
1. The first fu-a package of Fu Indicator:f should be the current Nalu head of F, and NRI should be the current Nalu header nri,type is equal to 28, indicating that it is fu-a package. The FU header generation method: S = 1,e = 0,r = 0,type equals the type in the Nalu header.
2. The FU indicator of the subsequent n fu-a package is exactly the same as the first one, and if it is not the last package, then the FU header should be: S = 0,e = 0,r = 0,type equals the type in the Nalu header.
3. The last Fu-a packet fu header should be: S = 0,e = 1,r = 0,type equals the type in the Nalu header.
So the summary is: the same nalu-thick FU indicator head is exactly the same, Fu header only S and e-bit difference, mark the beginning and end, respectively, their RTP sub-package sequence number should be incremented, and their timestamps must be consistent, and the load data for the Nalu packet minus 1 bytes of Nalu head after the split of the remaining data, this is very important, you can think Nalu head was split into Fu indicator and FU header, so no longer need 1 bytes of Nalu head.
About SPS and PPS, configuring frame transfer I use the first-out SPS, then send PPS, and use the same time stamp, or in accordance with the normal timestamp increment or the form of packet send processing seems to be able to see how the player decoded, and also mention, if we use VLC to play, You can set up SPS and PPS in the SDP file so that you don't have to send them.
When using VLC playback, the subcontracting mode option in the SDP file: packetization-mode=1, that is, add =fmtp:packetization-mode=1 at the end of the file, otherwise there is a problem.

A detailed description of the Packetization-mode is as follows:

When the value of Packetization-mode is 0 o'clock or does not exist, a single Nalu cell mode must be used.
Non-interleaved (non-interleaved) encapsulation mode must be used when the value of Packetization-mode is 1.
When the value of Packetization-mode is 2, the Interleaved (interleaved) packet mode must be used.

Summary of NAL unit types allowed per packaging method (yes = allowed, no = not allowed, IG = ignored)

Type Packet single NAL non-interleaved interleaved
Unit mode mode
-------------------------------------------------------------

0 undefined IG IG IG
1-23 NAL Unit Yes Yes No
Stap-a No Yes No
Stap-b No No Yes
MTAP16 No No Yes
MTAP24 No No Yes
Fu-a No Yes Yes
Fu-b No No Yes
30-31 undefined IG IG IG

SOURCE Link: See http://blog.csdn.net/caoshangpa/article/details/53009604 's comments