Decoding process of TS Stream-ES-PES-DTS-PTS-PCR

Decoding process of TS Stream-ES-PES-DTS-PTS-PCR

 

 

TS Stream decoding process:

1. Get Pat in ts

2. Obtain PMT From ts

3. You can know the video (audio) type (h264), corresponding PID, and pcr pid transmitted in the current network based on PMT.

4. Set the video filter of the Demux module to the PID and stream type of the corresponding video.

5. the payload data in the ts data packet obtained from the video Demux filter is one piece of PES. The ts header contains the number of packets about which PES the payload belongs. Therefore, the software should copy the data in the payload to the PES buffer to splice a PES package.

6. The merged PES packet header contains PTS and DTS information, and the PES header is es.

7. directly send the es package that is unplugged from the PES header to the decoder for decoding. The decoded data is a frame-by-frame video data, which should be associated with PTS in PES at least for video and audio synchronization.

8. I, B, B, P information is in Es.

 

 

ES is a data stream directly from the encoder. It can be a collectively referred to as an encoded video data stream, audio data stream, or other encoded data streams. The es stream is converted to the PES package after it passes through the PES package. The PES package consists of a packet header and a payload.

In the PES layer, PTS (explicit time tag) and DTS (Decoding time tag) are added to the PES header information for video and audio synchronization. In fact, the Mpeg-2 for video and audio synchronization and system clock recovery time labels in ES, PES and TS three layers, respectively
. On the es layer, video buffer verification vbv (video buffer verifier) is mainly related to synchronization to prevent the buffer overflow or underflow of the decoder. On the PES layer, displays the Time Label PTS (Presentation) in the PES header information.
Time stamp) and decoding time stamp. In the TS layer, the TS header information contains the program clock reference (program clock reference ), used to restore the system time series clock that is consistent with the encoding end.
Time Clock ).

The basic process is as follows: first, the es basic stream produced by MPEG-2 compression encoding, this data stream is very large, and only the I, P, B of these video frame or audio sampling information, then add some synchronization information and package it into a variable-length data packet PES. It turns out to be a stream format and is now a form of data packet segmentation. At the same time, it should be noted that ES is a data stream that only contains one type of content, such as only video or only audio. the packaged PES also only contain one type of Es, for example
Only PES for video elasticsearch and PES for audio elasticsearch. Elasticsearch is an encoding video or audio data stream. Each elasticsearch consists of several access units (Au, each video AU or audio au consists of the header and the encoded data. One AU is equivalent to one video image or one audio.
Frame. It can also be said that each Au is actually the display unit of the encoded data stream, that is, the sampling of a decoded video image or an audio frame. Peg-2 compresses the video to produce I, P, and B frames. Set
Frame sequence I1, P4, B2, B3, P7, B5, B6 frame encoding es. It is changed to PES by packaging and inserting the PTS/DTS mark in each frame. When the PTS/DTS mark is inserted
So you do not need to insert data transmission into B frames. For I and P frames, the sequence of data packets changes after multiplexing. Before the display, the data packets must be stored in the new sort cache of the video decoder, after being sorted and displayed again, you must insert PTS and DTS at the same time.
It is the basis for the new sorting.

 

The existence of the PTS/DTS mark is the key to solving synchronous display of video and audio and preventing overflow or underflow of the decoder input cache.PTS indicates the time when the display unit appears in the system target Decoder (STD-system target decoder). DTS indicates the time when all the bytes of the access unit are removed from the es decoder cache of STD.The video encoding image frame sequence is I1, P4, B2, B3, P7, B5, B6, I10, B8, B9 es, and
After entering pts/DTS, package them into video PES packages. Each PES package has a header that defines the data content in PES and provides scheduled data. Each packet header of I, P, and B has an PTS and DTS, but both PTS and DTs are
In the same way, you do not need to mark the data transmission of B frames. For frames I and P, it must be stored in the re-Sort cache of the Video Decoder before display. After delay (re-sorting), it will be displayed again, the PTS and DTS must be specified separately. For example, the sequence of frames input by the decoder is I1, P4, B2, B3, P7, B5, B6, I10, B8, and B9, p4 must be earlier than B2 and B3, but P4 must be
After B2. B3, P4 needs to re-sort the cache based on the time mark inserted into the data stream in advance to reconstruct the pre-encoding video frame sequence I1, B2, B3, p4, B5, B6, P7, B8, B9, I10. Obviously, the PTS/DTS mark indicates the existence of a dedicated time scale for determining an event or determining Information Decoding. Relying on the dedicated time scale decoder, you can know the time when the identified event or the information is decoded or displayed. For example, the PTS/DTS mark can be used to determine the encoding, multiplexing, decoding, and reconstruction time.

 

 

PCR

 

PCR is included in ts, that is, the header of TS packet may be included. It is used to specify the expected time when the TS packet arrives at decoder. Its effect on SCR is similar.

 

DTS, PTS

For an elasticsearch video, there are many I, P, and B frames, while all P and B frames use I and P frames as the reference. Because frame B is a forward and backward reference
For decode, decode the P or I frame after the B frame first. Therefore, the decode time is inconsistent with the real present time of the frame.
Decode each frame, and then display each frame according to PTS.

Sometimes there are DTS and PTS in the PES packet header. For pts, it represents the first full audio in the PES package payload.
PTS time of access unit or video access unit (not every audio/video access unit carries pts/DTS, so you can specify one in PES as the start point ).

The DTS of PES Baotou is also based on this principle, but note that for video, its DTS and PTS can be different, because B frames are stored so that their order can be reversed. For audio, There is no two-way prediction. His DTS and PTS can be regarded as a sequence. Therefore, you can always use one to only use PTS.

To: http://blog.csdn.net/godspirits/archive/2010/06/07/5653381.aspx