Android IOS WebRTC Audio and Video Development Summary (87)--analysis of the implementation of packet loss retransmission Nack in WebRTC

this This paper mainly introduces the realization of WEBRTC in Nack, Weizhenwei, the article was first published in the Wind network , Id:befoio

Support original, reprint must indicate the source, welcome attention to my public number blacker (Id:blackerteam or WEBRTCORGCN).

In WEBRTC, forward error correction (FEC) and packet loss retransmission (NACK) are important methods to resist network errors. FEC adds a redundant error-correcting code to the packet at the sender, and the error-correcting code is sent to the receiving end along with the packet, and the receiver checks and corrects the data based on the error-correcting code. RFC5109[1] Defines the format of the FEC packet. The nack sends the NACK message to the sending side after the packet is detected by the receiving end, and the sending side finds the corresponding packet in the transmit buffer according to the serial number in the NACK message and sends it back to the receiving end. Nack requires the support of the send-side send buffer, rfc5104[2] defines the format of the NACK packet.

In this paper, based on the study of WEBRTC source code, the transmission and reception of video packets is taken as an example, and the implementation of Anck packet retransmission mechanism is deeply analyzed. The main contents include: SDP negotiation Nack, receiving end packet loss determination, NACK message construction, sending, receiving and parsing, RTP packet retransmission. The following are discussed in detail.

I. SDP negotiation NACK

The nack is used as the RTP layer feedback parameter and is associated with the video codec. WEBRTC in the initialization phase, the Peerconnectionfactory object is created, and the Mediaengine is created in the object, where the Videoengine is WebRtcVideoEngine2. When the object is constructed, all video Codec,nack supported by the local side are collected together as Codec properties. During the next SDP negotiation process, the Nack attribute is negotiated in Offer/answer, 1.

Figure 1 SDP negotiation nack and action on video Jitterbuffer

Peerconnection at Createoffer, collect session control information, audio and video codec information and network information on this side. The video codec information is obtained from the WebRtcVideoEngine2. The last offer is to form an SDP message, which is sent to the network via the Peerconnection object.

After receiving an offer, the receiver first calls Setremotedescription, creating a Videoreceivestream object based on local configuration information, and the local Nack configuration information will eventually arrive at Vcmjitterbuffer. The peerconnection then calls Createanswer, generates answer, and ultimately selects codec collections supported by both parties based on the codec information in the offer and the codec information supported on this side. Finally, the setlocaldescription is called with the generated answer as a parameter, and the Videoreceivestream object is recreated downward based on the video codec information in answer. The NACK information is passed down to the final Vcmjitterbuffer, where nack related parameters are set. These parameters play a role in the process of receiving RTP packets, such as determining packet loss, sending NACK packets, and so on. When the answer sends back to the sender, the sending side calls Setremotedescription to perform the same setup process.

Second, the receiver End packet loss determination

The video receive-side packet drops are determined in the worker thread. After the RTP packet arrives at the receiving end, the RTP module reaches the Jitterbuffer object of the VCM module, and finally calls the Vcmjitterbuffer's Insertpacket function to cache and rearrange the packets.

Vcmjitterbuffer stores the serial number of the lost RTP packet in the collection missing_seq_nums. For packets coming from the RTP module, mark their serial number as SEQ1, and the last arrival packet's sequence number is SEQ2. If seq1 > Seq2, then the seq1 order arrives, the packet in the mark (seqnum2, seqnum1) interval is lost and stored in the Missing_seq_nums collection. Note that the loss status here is temporary, if the next packet arrives with seq1 < SEQ2, it means that the packet is out of order, then the missing_seq_nums smaller than SEQ1 's serial number is deleted.

When updating the Missing_seq_nums collection, if the serial number stored in the collection exceeds the preset capacity, the sequence number in the collection is reduced by calling Recycleframesuntilkeyframe (), until the total number of serial numbers in the collection is lower than the preset capacity value.

Third, Nack message sending and receiving

The NACK message construction and sending work of the receiving end is periodically completed in the Moduleprocessthread thread. As shown in procedure 2.

Figure 2 Nack message construction and Dispatch

Moduleprocessthread thread periodically calls Videoreceiver::p the rocess function, which calls vcmjitterbuffer::getnacklist through Vcmreceiver, from Missing_seq _nums the serial number of the RTP packets that were lost over time in the collection. Then call the Rtpstreamreceiver::resendpackets function. The calling process will eventually reach Rtcpsender::sendrtcp, sending a RTCP message of type Nack.

The NACK message is a RTCP extended feedback message of type 205, as shown in 3:

Figure 3 Nack message Format

where pt = 205,fmt = 1,packet Identifier (PID) is the serial number of the Lost RTP packet, Bitmao of Lost Packets (BLP) indicates the loss of the next 16 RTP packets starting from the PID. A NACK message can carry multiple RTP serial numbers, which are processed individually by the nack receiving end.

After the NACK message structure is completed, it is sent to the network layer. The NACK message is a kind of RTCP message, so its sending, receiving and analyzing follow the general flow of RTCP message processing. This section is referenced in document [3].

Four, RTP packet retransmission

After receiving and parsing the NACK message, the receiver handles various types of RTCP messages through a callback mechanism, and for nack messages, Rtpsender resend the RTP packets, as shown in 4:

Figure 4 Transmit-side packet retransmission

Rtcpreceiver after parsing the RTCP, the description structure of the RTCP message is obtained, and then the message semantics is processed by the callback. Nack messages are sent to Rtpsender for processing. Rtpsender finds the corresponding RTP packet in the Rtppackethistory cache based on the serial number contained in the NACK message. If found, the packet is sent to the network.

At this point, a complete nack message loop is completed and the lost RTP packet is sent back to the receiving end.

V. Summary

This paper deeply analyzes the implementation details of WEBRTC internal about packet loss retransmission (NACK), makes an in-depth study of NACK's SDP negotiation, packet loss determination and retransmission, and lays the foundation for continuing to learn the QoS mechanism of WEBRTC.

Reference documents

[1] RFC5109-RTP Payload Format for Generic Forward Error Correction.

[2] rfc5104-rfc 5104-codec Control Messages in the RTP audio-visual profiles with Feedback (AVPF).

[3] WEBRTC RTP/RTCP Protocol implementation of sub-http://www.jianshu.com/p/c84be6f3ddf3.

Android IOS WebRTC Audio and Video Development Summary (87)--analysis of the implementation of packet loss retransmission Nack in WebRTC