Ortp Usage Details 2

5. receive and send data

1. Sending process:

The rtp_session_send_with_ts interface is called when an application sends data. The parameter is a session handle, data buffer address, data length, and the current timestamp of the application. In this interface, the rtp_session_create_packet interface is called to construct a new message block based on the buffer address and Data Length, and initialize the RTP Header information according to the session information. Copy the data in the buffer to the Message Block. Call the rtp_session_sendm_with_ts interface to send data with the message block as the parameter. Rtp_session_sendm_with_ts calls the lower-layer function _ rtp_session_sendm_with_ts to complete specific sending. The following describes the implementation of the function:

If the sending is not started yet, that is, the snd_ts_offset variable is set to the current start value of the application. If scheduling is started, snd_time_offset is set to the time when the scheduler runs until now. This is the initialization of the timestamp.

If scheduling is enabled, adjust some timestamps as follows:

First, calculate the packet sending time, that is, packet time. The calculation method is to add the packet sending interval to the scheduler time when the first packet is sent, this interval is calculated based on the difference between the current time of the application and the time of the first sending by dividing by the clock rate of payload. For example, if the first sending time is 100, the current time is 300, that is to say, the sending goes through 200 units. If the clock rate of payload is 10, 20 timestamp units are passed. That is to say, the timestamp of the current package is the time of the scheduler plus 20. (Packet time should actually convert the sending time of the next package to the time of the scheduler and hand it over to the Scheduler for scheduling) if the difference between the calculated packet time and the current running time of the scheduler is less than 31 of 2, and the two are not equal, set the wait point to wake up at packet time. (For more information about the comparison, see other instructions)

Before sending data, set the RTP timestamp to the current timestamp passed in by the application. The snd_last_ts timestamp is also set to the current timestamp of the application.

Then call the actual sending interface rtp_session_rtp_send for sending. This interface will call the send system call to send data packets to the other end of the network.

Call rtp_session_rtcp_process_send to check whether the RTCP package needs to be sent. The following principles apply:

If the time difference between the last timestamp of the application query and the last RTCP packet calculated by the receiving unit is greater than the time interval that the RTCP report packet should send, or the difference between the time stamp of the last sent packet and the time of sending the last RTCP report packet calculated according to the unit of the sent timestamp is greater than the interval that RTCP should send, construct the RTCP Sender report package for sending.

During the construction of the RTCP control package, the SSRC source synchronization descriptor uses the source synchronization description information on the session. The NTP timestamp uses the current system time plus the number of seconds between 1900 and 1970, in fact, this time is the current number of seconds on January 1, 1900 (see timestamp description ). The RTP timestamp uses snd_last_ts, that is, the timestamp of the last sent stream. The number of sent packets and the number of packet bytes are counted by the RTP stream on the session. In addition, if a packet is received, it contains only one report block. After the data packet is constructed, it is directly sent.

If the current session mode is send-only, call rtp_session_rtcp_recv to receive and process the RTCP package. If the session supports the receiving mode, the RTCP packet is received during the RTP receiving process.

2. Receiving process.

The packet is received by calling the rtp_session_recv_with_ts interface. This interface actually calls rtp_session_recvm_with_ts to receive data from the underlying layer and copy the valid data (excluding the RTP Header) in the returned message block to the user's buffer. The following describes the implementation of rtp_session_recvm_with_ts:

If the receiving is not started, rcv_query_ts_offset is set to the initial time specified by the application, that is, the time when the application queries, and a start time offset is recorded. If the sending is not started or in Recv-only mode, the session last_rcv_time is set to the current time of the system. If the scheduler is set, rcv_time_offset is set to the current time when the scheduler runs after it starts. This is used as the receiving time offset. If the receiving has been started, in order to avoid receiving the same timestamp for multiple times, it is judged that if the time given by the current application parameter is equal to rcv_last_app_ts, that is, the timestamp of the last query by the application, set the read_socket variable to false to avoid continuous reception.

Next, go to the normal processing process. First, set rcv_last_app_ts to the current application time, that is, to update the last received time. If read_socket is set, call the rtp_session_rtp_recv and rtp_session_rtcp_recv interfaces to actually receive data from the underlying socket.

After the data is received in rtp_session_rtp_recv, rtp_session_rtp_parse is called to parse the data packets. If the telephone event package is found in rtp_session_rtp_parse, an event is created and sent to the event queue. For details, see the description of the event section. Updates to related variables in jitter are also processed in this interface, which is completed by calling the jitter_control_new_packet interface. Finally, put the data packet to the receiving queue.

To be further processed. After rtp_session_rtp_recv is selected, the telephone event queue of the session is checked. If it is not empty

If you receive the dial-up packet, you must call the registered callback function and send it to the event queue. After receiving the message, the system returns the result. If no package exists in the queue, proceed with the process:

If the receiving synchronization identifier is set, rcv_ts_offset is set to the timestamp in the received RTP data packet. This is the first timestamp of the stream. The rcv_last_ret_ts variable is set to the time given by the current application. Here is just an initial value. Then clear the synchronization ID. Therefore, the previous offset rcv_ts_offset records the timestamp of the first RTP data packet. The subsequent data packets will not pass through the processing logic here.

Call the jitter_control_get_compensated_timestamp API to calculate the stream timestamp. For details, see the description of the jitter module. If the jitter control on RTP is enabled, the jitter buffer mechanism is used to control data packets. Otherwise, a new data packet is directly obtained from the queue. When jitter is enabled, if the permissive algorithm of the session is enabled, call the rtp_getq_permissive interface to obtain data. In this interface, if the difference between the calculated stream Timestamp and the timestamp recorded in the RTP data packet is less than the 31 power of 2, a packet is displayed and returned from the queue, otherwise, null is returned. If the permissive algorithm is not enabled, call the rtp_getq interface to receive data packets in normal mode. In this interface, we return data packets whose timestamp is equal to or earlier than the calculated timestamp. If there are more than one such data packet, discard the old one, that is, the packet first retrieved from the queue, and the last returned packet is the last retrieved packet. If two data packets have the same timestamp, only one is returned. In addition, if a packet in this interface is discarded, the number of discarded packets is recorded in the reject parameter.

If a data packet is returned in the previous step, the timestamp in the data packet is updated. For details, see the jitter_control_update_corrective_slide interface. Then, update the rcv_last_ts timestamp to the timestamp value when the package arrived, that is, the value before the update. Call the rtp_session_rtcp_process_recv interface to receive and process RTCP. (Previously sent for processing) the modification of the trigger condition and the time amount after the trigger is the same as the sending part. If the sending count in the last rtcp sr report is less than the number of sent packets in the statistic, make_sr is called to construct the SR report package, and the previous statistical count is updated to the value saved in the statistic. If the value is not less than, it indicates that the rtcp sr report package is not required. However, if the number of packets received at the same time is greater than zero, it means that some packets are received, call make_rr to construct the RR package of RTCP. If the packet structure is successful, call rtp_session_rtcp_send to send the packet.

If the scheduling is not started, the package is directly returned to the upper layer without special processing. Otherwise, the scheduling is performed. Similar to the sending part, the rtp_session_ts_to_time interface is called to calculate the scheduling interval of the packet based on the time specified by the application and the time difference between the time when the application receives the first call. At this interval, the time the scheduler runs when the application asks the first package as the next scheduling time of the package. If the time is later than the current time of the scheduler, this time will be used as a wake-up point, waiting for the scheduler to schedule.

The relationship between timestamp values in the receiving and sending processes is shown in:

Figure 5-1

Vi. Implementation of Anti-Jitter

Description of some variables in the jitter struct: (for details about this struct, see Figure 2-8) Where jitt_comp is the user-defined Anti-jitter compensation time, And jitt_comp_ts is the value that converts it to the timestamp unit, adapt_jitt_comp_ts is the compensation time value calculated using the adaptive algorithm. Slide is the average value of the difference between the expected receipt time of the package and the application receipt time, And prev_slide is the last saved slide value. Jitter is the average value of the difference between diff (the timestamp value of the latest package and the time value received locally, used to calculate the slide) and the updated slide, olddiff is the diff value calculated last time, And inter_jitter is the interval jitter. For details, see RTCP protocol (refer to 1 ).

Corrective_step and corrective_slide are used when updating the timestamp in the package.

Adaptive and enabled have been described when introducing the jitter struct.

During Session Initialization, rtp_session_set_jitter_buffer_params is called. This interface sets the jitter buffer parameter. In the code, the default jitter time is set to 80 ms, that is, the interval between four data blocks (for 8 Khz audio sample data). the queue length of the input data packet is set to 100, that is, 100 data packets can be buffered. At the same time, the adaptive (Adaptive) feature of jitter is enabled, that is, adaptive compensation ). In practice, you can also independently call rtp_session_set_jitter_compensation to set the jitter compensation time. You can call rtp_session_enable_adaptive_jitter_compensation to set whether the adaptive compensation function is enabled.

When setting the jitter buffer, jitter_control_init is called to complete jitter initialization. In this interface, jitt_comp is set to the value set by the user. This value is the compensation value. In addition, call jitter_control_set_payload to convert the compensation value to the timestamp unit value and set it to jitter_comp_ts. The conversion depends on the payload clock sampling rate. Set the correction step value (corrective_step) to (160*8000)/Pt-> clock_rate. Most audio sampling rates are 8 kHz, so the correction should be based on 160 timestamp units.

To use jitter, enabled variables must be enabled. To use adaptive variables, enable adaptive variables.

The key to the data sending process is receiving. After receiving the data, it is not directly handed over to the upper-layer application, but placed in the buffer, which is actually a queue. The buffer size is set in the jitter initialization part. The default value is 100 (queue length), which can buffer 100 packets, this will not buffer much data for media streams with hundreds of network packets in one second. In addition, as long as the received packet is parsed, it is first buffered into the queue. If the number of packets exceeds the queue size, the oldest packet is removed, which is also reasonable. The packages passed for the application are obtained from the queue, so the oldest package is obtained. Jitter is required to control whether data packets need to be retrieved and uploaded to the application.

If no jitter buffer control is available for data packets that have been buffered to the local device, we will directly return the data packets. If there is control, we need to determine the packet timestamp, only packages older than the given timestamp (arrived earlier than the given timestamp) are uploaded to the application. The key factor for the control package to be uploaded to the application is the given timestamp value. How does this value come from? In the program, call the jitter_control_get_compensated_timestamp interface for calculation. The basic formula is:

TS = user_ts + slide-adapt_jitt_comp_ts

To better understand the formula above, let's look at how the above values are calculated. First, user_ts, which is the timestamp provided by the application when it receives the stream. It is calculated based on the Application receiving speed and payload type. It is typical, for audio data with a sampling rate of 8 kHz, the increment step value of the timestamp is 160. (sampling is performed every 20 milliseconds ). If there is no delay in network transmission and data packet processing does not require time consumption, user_ts should be consistent with the timestamp value in the RTP package.

Slide, based on the previous introduction, is the average value of the time difference between the expected receipt time of the data packet and the actual local receipt time. After receiving a new RTP packet, this value is updated. The new slide value is the previous value multiplied by 0.99 and the new calculated value multiplied by 0.01.

The Calculation of adapt_jitt_comp_ts depends on the value of jitter. As described earlier, jitter is the average value of the difference between diff and the updated slide. The calculated value is multiplied by 0.99 and the new value is multiplied by 0.01. If the data packet arrives evenly, the diff value and the slide value should be equal, so that the jitter value is zero. On the contrary, if the value of the jitter changes a lot, it means that the interval between each arrival of the data packet is different. To some extent, it indicates that the jitter is relatively large.

Inter_jitter is the interval jitter. For the meaning and calculation, see RTCP sending interval analysis (see 3 ). From the calculation above, we can see that inter-jitter reflects the jitter at two intervals, while slide reflects a relatively long-term even jitter.

If adaptive is enabled, the value of slide is constantly updated, and each time 50 packets are received, adapt_jitt_comp_ts is updated. The new value is one of the larger jitt_comp_ts and 2 * jitter.

For the above calculation process, see interface jitter_control_new_packet.

Now let's look back at the formula above. Slide is actually a value smaller than zero, so we actually try to bring the timestamp closer to the timestamp value in the package, the compensation value serves as a buffer to a certain extent.

For update of the data packet timestamp value, if a data packet is taken out of the queue and the timestamp value of the current data packet is inconsistent with the timestamp value of the previously received data packet, when adaptive is enabled, the timestamp value of the data packet is corrected. The algorithm is as follows:

The current slide minus the previous slide. If the difference value is greater than the corrected step value correction_step, the corrected sliding value correction_slide is added with a correction_setp, And the prev_slide is updated to the value of Slide plus correction_step. If the difference value is less than the negative value of correction_step, convert it to a positive value and perform the same update as before. Then, change the timestamp in the package to the value after adding correction_slide. (Will this change be used later? The data packet has been handed over to the upper layer .)

From the mechanism described above, the jitter buffer mechanism can ensure that data packets are uploaded to the application at a relatively uniform speed by using buffering.

7. event handling

In the RTP Session, the callback function and event queue of each event in the signal table and table are saved. If you receive the event information registered in the signal table, call the registered callback function for processing. For the telephone event package, in addition to calling the callback function, you also need to send it to the event queue, this allows upper-layer applications to perform further processing.

8. Other notes

The comparison Calculation of RTP timestamps is mainly completed through several macros.

Rtp_timestamp_is_newer_than (ts1, ts2) is smaller than or equal to ts2, while rtp_timestamp_is_strictly_newer_than (ts1, ts2) is less than ts2, and there is no equal relationship. However, in actual implementation, the difference value is compared with the 31 power of 2. The principle is as follows:

The difference result is forcibly converted to uinit and then compared to 0x8000000. In this way, if the value is greater than or equal to zero, a success is returned. If the value is less than zero, a failure is returned. It's just a simple comparison.

Concerning the problem of RTCP synchronizing RTP streams:

The media streams transmitted from RTP may be audio streams and video streams. They adopt different encoding methods and different sampling rates, and both streams contain timestamp information. You can use the RTCP report package to synchronize the two streams. The RTCP report package contains absolute NTP-based timestamp information and the same sample timestamp information as the RTP stream, based on the two timestamps, You can synchronize the same media stream. The unique cname information contained in RTCP can associate the audio and video stream information of the same media, although the two streams use different source descriptors SSRC.

About the RTCP sending interval:

In this program, the calculation is only based on the 5-second interval, which is fixed. There are huge limitations, but for point-to-point communication, the problem is not very big. Test the eyebeam program and find that the RTCP package is sent in about 3 seconds. I do not know whether it is calculated or fixed.

This can improve ortp according to the Protocol requirements.

Rules for receiving and sending RTCP packets:

According to the Code, check whether the RTCP package needs to be sent after one RTP packet is sent each time. Check whether the RTCP package can be received at the same time when the RTP package is received. After completion, check whether the RTCP package needs to be sent. But the Protocol rules are not like this.

Concerning the construction of the RTCP package:

The RTCP package must be transmitted to the network in the form of a composite package, and must contain at least two basic packages. The first package must be the report package of the sender or the report package of the recipient, the sdes source description package is also included.

For more information about the RTCP protocol, see the instructions on the RTCP protocol. For more information, see section 1.

9. Use the ortp Library

Ortp provides a test program to test the ortp library, and the test program is also the best example of how to use ortp.

The test program is under the test directory of the source code, including sending the test rtpsend. c. Receive and test rtprecv. c. Test mrtpsend in parallel transmission. c. Parallel Acceptance Test mrtprecv. c. There are also tests related to telephone events. The description here only targets the above four test results for receiving and sending test programs.

Most of the test examples provided in the program are for audio data, and the timestamp added value is also set to 160 (for how to calculate the value, see the description in the timestamp section ), this problem occurs when receiving and sending video data. You need to make some modifications. For details, see the problem list.

Compilation of ortp code on the haisi 3716 platform :. /configure -- Host = arm-hisiv200-linux [specify the cross Environment] -- prefix = [specify the installation directory] Make make install after the completion of the specified directory will create include, lib and share three folders. Include contains the header file that we want to use the ortp library. The lib directory contains the compiled library. The following is a document description under share. You can take the files under the test directory to the installation directory and compile the executable files with the library for testing.

Test Question list:

1. By default, the test program tests audio streams without video streams. The timestamp step value is 160, which is calculated based on the 20 ms sampling period plus 8 KHz sampling rate. As a result, the speed of receiving and sending video streams is extremely slow. Therefore, when receiving video data, you need to adjust the step value to 3600, that is, 90000 sampling rate plus 25 frames per second. You can adjust the test results during actual use.

2. Adjusting the payload type independently does not completely solve the speed problem. The receiving process can be changed according to the received type (that is, the payload type changed event will be generated), so the corresponding adjustment is made, so the receiving process has little impact, however, because the default payload type is set to payload_type_pcmu8000, the sending speed is very slow (based on the 8 KHz sampling rate) adjusted to 33 (payload_type_mp2v.

3. 33. No support is added by default in ortp. You can add it separately based on the avprofile. c file.

4. During the test, we found that the multiple Recv test program could not receive data, and changed the file operation interface to fopen and the ASCII standard file read/write interface. The reason is to be found.

5. Multi-session receiving is currently based on different port numbers.

6. disabling the scheduling mode and block mode can accelerate video stream pushing.

7. Video Stream Transmission speed can be controlled without a scheduler to improve mosaic. Actually sending too fast

No. Use VLC for playback testing.

8. When the scheduler is started, set the application timestamp increment value (user_ts) or adjust this variable.

Adjust the transmission speed of the whole video stream and mosaic.

9. data transmission rate and sampling rate, increase of user_ts timestamp value and buffer size are related.

10. During Concurrent sending and receiving tests (msend and mrecv), you must set the non-blocking mode. Otherwise, the program may be stuck.

10. Reference

1 rfc3550rtp: A transport protocol for real-time applications 2 rfc3551rtp profile for audio and video conferences with minimal control 3 RTCP sending Interval Analysis

Ortp Usage Details 2