Rtp rtcp for audio and video transmission and Synchronization

Http://blog.sina.com.cn/s/blog_488365030100ccp8.html

1 Real-time audio and video transmission

1.1JrtplibLibrary Introduction

The system uses the open-source database jrtplib to develop the RTP transmission module. Jrtplib is a fully open-source RTP Library developed by EDM (expertise centre for digital media) of Hasselt University in Belgium and implemented in C ++, currently, it can run on Windows, Linux, FreeBSD, Solaris, UNIX, and VxWorks. The latest version is jrtplib3.7.1.

Jrtplib fully complies with the rfc3550 standard and provides a wide range of interfaces for network communication using Win32 socket. When using this library for development and design, developers only need to care about the most basic problems such as RTP load, RTCP load, timestamp increment, sending destination address and port, RTCP bandwidth, and sending interval, without having to consider some trivial details, you can download the updated source code package from the jrtplib website (http://lumumba.luc.ac.be/jori/jrtplib/jrtplib.html.

1.2 Jrtplib Initialization

Before using jrtplib for Real-Time Streaming Media data transmission, you should first generate an rtpsession instance to represent the RTP session, and then call the CREATE () method to initialize it. The create () method of the rtpsession class has only one parameter to specify the port number used for this RTP session. In example 1, a simple initialization framework is provided, which completes the initialization of the RTP session and does not have any practical functions.

1. Initial. cpp

# Include "rtpsession. H"

Int main (void)

{

Rtpsession sess;

Sess. Create (1, 5000 );

Return 0;

}

If the RTP session fails to be created, the CREATE () method returns a negative number. Although it can easily determine whether the function call is successful or failed, however, it is difficult to understand the cause of the error. Jrtplib adopts a unified error handling mechanism. If all functions provided by jrtplib return negative numbers, it indicates that some form of errors have occurred. For specific error information, you can call rtpgterrorstring () function. The rtpgterrorstring () function passes in the error code as a parameter and returns the error message corresponding to the error code. Example 2 provides a more complete initialization framework, which can better handle errors generated during RTP Session Initialization:

2. Framework. cpp

# Include

# Include "rtpsession. H"

Int main (void)

{

Rtpsession sess;

Int status;

Char * MSG;

Sess. Create (1, 6000 );

MSG = rtpgterrorstring (Status );

Printf ("error string: % s \ n", MSG );

Return 0;

}

Setting an appropriate timestamp unit is another important task in the RTP Session Initialization Process. This is achieved by calling the settimestampunit () method of the rtpsession class, this method also has only one parameter, indicating the timestamp unit in seconds. For example, when the RTP session is used to transmit 8 000Hz sampled audio data, the timestamp increases by 8000 per second, so the timestamp unit should be set to 1/8000 accordingly:

Sess. settimestampunit (1.0/8000.0 );

1.3JrtplibData Transmission

After the RTP session is successfully established, the streaming media data can be transmitted in real time. First, you need to set the destination address for data transmission. The RTP protocol allows multiple destination addresses for the same session. You can call the adddestination (), deletedestination (), and cleardestinations () of the rtpsession class () method. For example, the following statement indicates that the RTP session sends data to Port 6000 of the local host:

Unsigned long ADDR = ntohl (inet_addr ("127.0.0.1 "));

Sess. adddestination (ADDR, 6000 );

After all the target addresses are specified, you can call the sendpacket () method of the rtpsession class to send streaming media data to all target addresses. Sendpacket () is an overload function provided by the rtpsession class. It has the following forms:

Int sendpacket (void * data, int Len)

Int sendpacket (void * data, int Len, unsigned char PT, bool mark, unsigned long timestampinc)

Int sendpacket (void * data, int Len, unsigned short hdrextid, void * hdrextdata, int numhdrextwords)

Int sendpacket (void * data, int Len, unsigned char PT, bool mark, unsigned long timestampinc, unsigned short hdrextid, void * hdrextdata, int numhdrextwords)

The most typical use of sendpacket () is similar to the following statement. The first parameter indicates the data to be sent, and the second parameter indicates the length of the data to be sent, the following are the RTP load types, identifiers, and timestamp increments.

Sess. sendpacket (buffer, 5, 0, false, 10 );

For the same RTP Session, the load type, identifier, and timestamp increment are usually the same. jrtplib allows you to set them as the default parameters of the session. This is done by calling setdefaultpayloadtype () of the rtpsession class () setdefamark mark () and setdefatimetimestampincresMent () method. Setting these default parameters for RTP sessions can simplify data transmission. For example, if you set the default parameters for RTP sessions:

Sess. setdefapaypayloadtype (0 );

Sess. setdefamark mark (false );

Sess. setdefatimetimestampincresMent (10 );

Then, when sending data, you only need to specify the data to be sent and its length:

Sess. sendpacket (buffer, 5 );

1.4JrtplibReceive data

For the receiver of streaming media data, you must first call the polldata () method of the rtpsession class to receive the RTP or RTCP datagram sent. Since multiple participants (sources) are allowed in the same RTP session, you can call the gotofirstsource () and gotonextsource () Methods of the rtpsession class to traverse all sources, you can also use the gotofirstsourcewithdata () and gotonextsourcewithdata () Methods of the rtpsession class to traverse those sources with data. After a valid data source is detected in the RTP session, you can call
The getnextpacket () method of the rtpsession class extracts RTP datagram from it. When the received RTP datagram is processed, remember to release it in time. The following code demonstrates how to process the received RTP datagram:

If (sess. gotofirstsourcewithdata ()){

Do {

Rtppacket * pack;

Pack = sess. getnextpacket (); // process received data

Delete pack;

} While (sess. gotonextsourcewithdata ());

}

Jrtplib defines three receiving modes for RTP datagram, each of which specifies which RTP datagram will be accepted and which RTP datagram will be rejected. You can set the following receiving modes by calling the setreceivemode () method of the rtpsession class:

A) receivemode_all: the default receiving mode. All received RTP data packets will be accepted;

B) receivemode_ignoresome except some specific senders, all incoming RTP datagram will be accepted, and the list of rejected senders can be called by addtoignorelist (), deletefromignorelist () and clearignorelist () method;

C) receivemode_acceptsome except for some specific senders, all incoming RTP datagram will be rejected, and the list of accepted senders can be called by addtoacceptlist (), deletefromacceptlist () and clearacceptlist () method.

2 Audio/Video Synchronization

2.1RTCPControl Parameters

Because audio and video streams are transmitted as different RTP sessions, they are not directly associated at the RTP layer. Although different streams sent from a data source have different synchronization source identifiers (SSRC, RTCP requires the sender to send a canonical name that uniquely identifies the data source to the receiver. the application layer associates the audio and video streams for synchronization.

RTP/RTCP contains timestamp (relative and absolute), serial number, and other information. You can use it to synchronize timestamp-based Multimedia Streams. Use the relative Timestamp and serial number to synchronize data in the stream. Use the correspondence between the relative and absolute timestamp to synchronize data between streams. The algorithm for obtaining relative and absolute timestamps is as follows:

While (pack = getnextpacket ())! = NULL)

        {

If (srcdat-> sr_hasinfo () & srcdat-> sr_getrtptimestamp ()! = App-> mvideortcprtp) 

                  { 

                           App-> mvideortcprtp = srcdat-> sr_getrtptimestamp ();

                           App-> mvideortcpntp = srcdat-> sr_getntptimestamp (). getmsw ();

                           Srcdat-> flushpackets (); }

                           Deletepacket (pack );

}

 

 

2.2 Synchronous implementation between audio and video streams

 

When sending audio and video data, the sender also sends an SR package so that the receiver can correctly play the audio and video simultaneously. The specific implementation method is to traverse the data source again after the receiver receives the data packet each time, obtain the ss_rtptime and ss_ntptime data of all sources, and obtain the data of the audio end and the video end, the following formula can be used for calculation. First, describe the variables listed in Table 2.1:

Table 2.1  Variable description table

Type	RTP data	NTP data	RTP timestamp frequency
Audio	Audio_srrtptime	Audio_srntptime	Audio_fre
Video	Video_srrtptime	Video_srntptime	Video_fre

The RTP and NTP data of the audio and video can be read from the SR package, but the timestamp frequency needs to be calculated. The following formula is used:

Audio_fre = (audiosrrtptime2-audiosrrtptime1)/(audiosrntptime2-audiosrntptime1 );                                   (4.3)

Video_fre = (videosrrtptime2-videosrrtptime1)/(videosrntptime2-video srntptime1 );                                        (4.4)

Then calculate the video RTP time, namely:

Video_rtptime = video_srrtptime + (Audio_SRNTP-Video_SRRTP) × video _ fre;  

(4.5)

Last click(4.6) You can calculate the video_truertp time and compare it with the time read from the RTP package to perform synchronous playback.

(Video_TRUERTP-Video_RTPTime)/video_fre = (Audio_TRUERTP-Audio_SRRTPTime)/audio_fre;                                       (4.6)

The following uses a temporary buffer to synchronize audio and video. To ensure normal audio transmission, the system uses the audio as the main axis and the video as the auxiliary axis. Uses the maximum latency as the buffer size to store m audio frame data. When the receiving end obtains M audio frames, it starts to play the audio. Each time a video frame is obtained, it is calculated that the RTP Time of the current video should be compared with the current RTP time. If it is within ms, play quickly. If the latency is greater than ms, discard it and compare it to the next frame. If the latency is less than ms, store it in the video buffer zone, if the video buffer size exceeds the value of the maximum temporary buffer, playback continues.