Multimedia Development---H264 coding + packaging + decoding related knowledge in rtsp es, pes, ts ...

1) Es Stream (Elementary stream): Also called the basic stream, which contains the continuous stream of video, audio, or data.

2) PEs stream (Packet Elementary Stream): Also known as packaged basic stream, is the basic stream es stream according to the need to be divided into the length of the packet, and add Baotou to form a packaged basic bitstream pes stream.

3) TS Stream (Transport stream): Also known as the transport Stream, is composed of a fixed length of 188 bytes of the package, containing the independent time base of one or more program, a program can also contain multiple video, audio, and text information es stream; Each ES stream will have a different PID designation. And in order to be able to analyze these ES streams, TS has some fixed PID to send the program and ES stream information in the form of: Pat and PMT table, (in the MPEG-2 system, by video, The standard flow of audio for the actual transmission generated by the ES stream and the secondary data is called the MPEG-2 Transport Stream)

Encapsulation: Is bundled packaging, the screen video files and audio track files packaged together, and according to a certain rules to establish sorting and indexing, easy for the player or playback software to index playback. Includes Avi/ps (program Stream)/TS (Transport Stream)/MKV (Matroska), etc.

4) I frame: Intra-frame encoding frame also known as intra picture,i frame is usually each
The first frame of the GOP (a video compression technique used by MPEG) is moderately compressed as a reference point for random access and can be treated as an image. I-frames can be seen as a product of a compressed image.

5) P Frame: Forward prediction coding frame, also known aspredictive-frame, by sufficiently lower than the image sequence of the previously encoded frame of time redundancy information to compress the amount of transmitted data encoded image, called the prediction frame;

6) B frame: Bidirectional prediction interpolation coding frame also known as bi-directional interpolated Predictionframe, both considering with the source image sequence before the encoded frame, It also takes into account the time redundancy information between the encoded frames behind the source image sequence to compress the encoded image of the transmitted data, also called the bidirectional prediction frame;

7) Pts:presentationtime Stamp. PTS are primarily used to measure when a decoded video frame is displayed;

8) Dts:decode TimeStamp. DTS primarily identifies when the bit stream in memory is read into the decoder to decode.

The order of DTS and the order of PTS should be the same without the existence of B-frames.

9)

I frame: itself can be extracted by the video decompression algorithm into a single complete picture.

P frame: You need to refer to an I frame or b frame in front of it to generate a complete picture.

B frame: Refer to its previous I or P frame and a P-frame behind it to generate a complete picture.

A GOP is formed between two I frames, and the size of the BF can be set by parameter in x264, i.e. the number of B between I and P or two p.

It can be explained that if a frame B exists, the last frame of a GOP must be p.

(10) Different for DTS and pts:

DTS is primarily used for decoding video, which is used in the decoding phase. PTS are used primarily for video synchronization and output. Used when display. In the absence of B
frame in case. The output order of DTS and PTS is the same.

Here is an example of a GOP of 15, with its decoded reference frame and its decoding sequence in it:

such as: I frame
The decoding does not depend on any other frames. And the decoding of P frame depends on the I frame or p in front of it.
Frame. The decoding of frame B depends on the nearest I frame or p frame and the nearest p
Frame.

(11) The approximate process of video decoding is as follows:

(12) Calculation of PTS

Method one, according to the IPB type of the front and back frame, we can know the actual display order of the frame, use the frame rate in the SPS information obtained earlier, and the frame count frame_count to calculate the PTS. This method requires a few frames of cache (typically caching the length of a group).

ipbbipbbipb... Frame type
1 2 3 4 5 6 7 8 9 10 11 ... The first few frames
1 4 2 3 5 8 6 7 9 12 10 ... frame display order

Between an I-frame and the next I-frame, is a group.
From visible, the display order of frames of type P is after the last B-frame behind.
So to get the PTS of the 7th frame, at least to know the type of his next frame, in order to learn his display sequence.

8th Frame pts=1000 (ms) *7 (frame display order) * frame rate

Method Two, each slice information inside, all records have PIC_ORDER_CNT_LSB, the current frame in this group display order. With this pic_order_cnt_lsb, you can calculate the PTS of the current frame directly. This method does not require frame caching.

Calculation formula:

pts=1000* (i_frame_counter+ pic_order_cnt_lsb) * (Time_scale/num_units_in_tick)

I_frame_counter is the frame sequence of the most recent I-frame position, through the I-frame count + the frame order in the current group, to get the actual display sequence position of the frame, multiply the frame rate, and then multiply 1000 (milliseconds) of the base_clock (basic clock frequency), get pts.

I p b b I p b b i p b
... Frame type
1 2 3 4 5 6 7 8 9 10 11 ... The first few frames
1 4 2 3 5 8 6 7 9 12 10 ... Frame Display Order
0 6 2 4 0 6 2 4 0 6 2
... pic_order_cnt_lsb

Be careful to note that the pic_order_cnt_lsb inside the slice is incremented by 2.
The frame rate recorded in the SPS in H264 is usually twice times the actual frame rate time_scale/num_units_in_tick=fps*2

Therefore, the actual calculation formula should be this way
PTS=1000* (I_FRAME_COUNTER*2+PIC_ORDER_CNT_LSB) * (Time_scale/num_units_in_tick)
or a
PTS=1000* (I_FRAME_COUNTER+PIC_ORDER_CNT_LSB/2) * (TIME_SCALE/NUM_UNITS_IN_TICK/2)

So, the 11th frame of PTS should be so calculated
1000* (9*2+2) * (Time_scale/num_units_in_tick)

RTP Packaging H264 Timestamp, because the H264 standard description is 90000/frame rate, here the PTS Base_clock are calculated according to 1000 (milliseconds), if multiplexed into TS, Base_clock is 90k, so also should be multiplied by 90.

PIC_ORDER_CNT_LSB:

Lsb:least significant bits POC low

Poc:picture oder counter

Pic_order_cnt_type: Indicates the encoding method of the POC (picture Order Count) and the order in which the POC identity image is played.

Http://www.cppblog.com/dvb-dvb/archive/2009/08/12/99268.html

http://blog.csdn.net/ljh081231/article/details/5828997

Frame rate Solution for http://www.chinavideo.org/archiver/?tid-5256.html 264-bit stream

http://bbs.csdn.net/topics/390368092 RTP frame rate, timestamp

Http://70565912.blog.51cto.com/1358202/533736/pts Get Code implementation

http://zhongcong386.blog.163.com/blog/static/1347278042012113033745404/

Http://www.xuebuyuan.com/660091.html

Http://video.vomont.com/client.html

http://www.phpv.net/html/122.html hack TV Download

http://wxinpeng.iteye.com/blog/2106068 RTSP Public network address

Http://www.foko.cn/admin/upload/201323123111.pdf-Mei-chan IPC

Http://www.fanli7.net/a/bianchengyuyan/_NET/20130415/297725.html

http://blog.csdn.net/ljh081231/article/details/8124535 CCD

Multimedia Development---H264 coding + packaging + decoding related knowledge in rtsp es, pes, ts ...