Analysis of WEBRTC audio and video analytic process

The WEBRTC audio and video parsing process consists of multiple threads:

1. RTP Network stream receive thread (RTP stream reciever thread)

2. Audio and video decode thread (decode thread)

3. Render threads (render thread)

RTP network stream receive thread (RTP stream reciever thread):

Receive network RTP packets, parse RTP packets, get audio and video packets. The resolved RTP packet is added to the Rtpstreamreceiver::frame_buffer_ or eventually joined Vcmreceiver::jitter_buffer_, decoding the thread from Frame_buffer_ or Jitter_ The frame is removed from the buffer_ to decode.

Related classes: Rtpstreamreceiver (rtp_stream_receiver.cc), Videoreceivestream (video_receive_stream.cc)

Function Call Flow:

Videoreceivestream::D eliverrtp () =

Rtp_stream_receiver_. DELIVERRTP (rtp_stream_receiver.cc) =

Rtpstreamreceiver::receivepacket (rtp_stream_reciver.cc) =

Rtp_receiver_->incomingrtppacket (rtp_receiver_impl.cc) =

Rtp_media_receiver_->parsertppacket (rtp_receiver_video.cc) =

1. Depacketizer->parse (rtp_format_h264.cc) resolves payload_type, such as SPS and PPS.
2. Data_callback_->onreceivedpayloaddata (rtp_stream_reciver.cc) =

A) if (H264) insertspsppsintotracker

Packet_buffer_->insertpacket

b) Video_receiver_->incomingpacket (video_receiver.cc) =

_receiver. Insertpacket (receiver.cc) =

Above a) and B) step is or relationship,

A) Steps:

Packet_buffer_->insertpacket (packet_buffer.cc) =

Received_frame_callback_->onreceivedframe (rtp_stream_receiver.cc) =

Reference_finder->manageframe (rtp_frame_reference_finder.cc) =

Rtpframereferencefinder::manageframe =

a) Rtpframereferencefinder::manageframegeneric =

b) Rtpframereferencefinder::manageframev8=>rtpframereferencefinder::completedframev8 =

c) Rtpframereferencefinder::manageframev9=>rtpframereferencefinder::completedframev9 =

Frame_callback_->oncompleteframe =

Complete_frame_callback_->oncompleteframe =

Frame_buffer_->insertframe

b) Steps :

_receiver. Insertpacket (receiver.cc) =

Jitter_buffer_. Insertpacket (jitter_buffer.cc) =

Decodable_frames_->insertframe (jitter_buffer.cc)

Member Description:

Rtp_receiver_: Rtpreceiver, Subclass Rtpreceiverimpl.

Rtp_media_receiver_:rtpreceiverstrategy pointers, subclasses are Rtpreceiveraudio and Rtpreceivervideo.

Data_callback:rtpdata, sub-class Rtpstreamreceiver.

Video_receiver_:videoreceiver.

Packet_buffer_: Packetbuffer.

_receiver:vcmreceiver.

Received_frame_callback_: Class Rtpstreamreceiver, implement Video_coding::onreceivedframecallback.

Reference_finder_: Class Rtpframereferencefinder.

Frame_callback_: Class Rtpstreamreceiver, implement Video_coding::oncompleteframecallback.

Complete_frame_callback_: Class Videoreceivestream, implement Video_coding::oncompleteframecallback.

Video decoding threads (decode thread):

Reads each frame from the rtpstreamreceiver::frame_buffer_ for decoding.

Related classes: Videoreceivestream (video_receive_stream.cc)

Function Call Flow:

Videoreceivestream::D ecode (video_receive_stream.cc) =

Video_receiver_->decode (video_receiver.cc) =

1. _codecdatabase.getdecoder (frame, _decodedframecallback) =

Ptr_decoder_->registerdecodecompletecallback (_decodedframecallback)

2. if (frame_buffer_->nextframe) video_receiver_. Decode (frame) (frames taken from frame_buffer_ )

　　　　　 Else Video_receiver_. Decode (Kmaxdecodewaittimems) (Take frame from jitter_buffer_ ) =

_receiver. Framefordecoding (take frame) = =

Jitter_buffer_. Nextcompleteframe

_deocoder->decode (_decoder is a specific decoder, such as h264) and =

decoded_image_callback_->decoded (generic_decoder.cc) =

_receivecallback->frametorender (video_stream_decoder.cc) =

Incoming_video_stream_->onframe (incoming_video_stream.cc) =

Render_buffers_->addframe (incoming_video_stream.cc)

Rtp_stream_receiver_->framedecoded (statistics)

Member Description:

Video_receiver_:videoreceiver.

_decodedframecallback: Class Vcmdecodedframecallback.

_receivecallback: Class Vcmreceivecallback, subclass Videostreamdecoder.

INCOMING_VIDEO_STREAM_: rtc::videosinkinterface<videoframe>*, Subclass Incomingvideostream, Webrtcvideoreceivestream.

Render threads (render thread):

The render thread reads the frame from the Incomingvideostream::render_buffers_ and sends it out.

Related classes: Incomingvideostream

Incomingvideostream::incomingvideostreamprocess (incoming_video_stream.cc) =

1. Render_buffers_->frametorender () =

2. External_callback_->onframe (set to Videoreceivestream in videoreceivestream::start) =

Config_.renderer->onframe (video_receive_stream.cc, the Webrtcvideoreceivestream constructor sets the Config_.render for itself, Final pass to Videoreceivestream::config_) =

Sink_->onframe

Sink settings:

Peerconnection::createvideoreceiver (peerconnection.cc) =

Videortpreceiver::videortpreceiver (rtpreceiver.cc) =

Channel_->setsink (Broadcaster_) =

Webrtcvideochannel2::setsink (SSRC, sink) =

Receive_streams_[ssrc]->setsink (sink) (Final setup webrtcvideoreceivestream::sink_)

Member Description:

Render_buffers_:std::list<videorenderframes>

External_callback_: rtc::videosinkinterface<videoframe>*.

SINK_: rtc::videosinkinterface<webrtc::videoframe>*, set by Webrtcvideoreceivestream::setsink.

Broadcaster_: Rtc::videobroadcaster, inherit videosinkinterface<webrtc::videoframe>,broadcast video frames to sinks.

Receive_streams_: std::map<uint32_t, webrtcvideoreceivestream*>.

Analysis of WEBRTC audio and video analytic process