H264 frame structure Analysis and frame judging

http://blog.csdn.net/dxpqxb/article/details/7631304

H264 Nalu (NAL unit) to support the transmission of encoded data in a packet-switched technology-based network.

NALU defines the basic format that can be used for packet-based and bitstream-based systems, while giving out information, thus providing video encoding and interface to the outside world.



H264 three different forms of data in the encoding process:

Sodb Data bit string--> the most primitive encoding data, namely VCL data;

RBSP raw byte sequence load--> the end bit (rbsp trailing bits a bit "1") is added to the back of the Sodb for byte alignment;

EBSP Extended byte sequence payload-----rbsp byte (0X03) is added to the base of this: When adding Nalu to Annexb, you need to add the start code NALU before each set of Startcodeprefix, If the Nalu corresponding slice is the beginning of a frame, it is represented by a 4-bit byte, ox00000001, otherwise the ox000001 is represented by a 3-bit byte (part of a frame). In addition, in order for the Nalu body to not include the start code conflict, at the time of encoding, each encounter two bytes consecutive 0, insert a byte of 0x03. 0X03 is removed when decoding. Also known as shelling operations.

Encoding process:

1. Packaging the Sodb of the VCL output into Nal_unit,nalu is a general encapsulation format, which can be applied to the sequential byte stream and the IP packet switching mode.

2. For different transport networks (circuit-switched | packet switching), the Nal_unit is packaged into a package format for different networks (such as encapsulating Nalu into RTP packets).



---------------------------------------------------

Process one, VCL data encapsulated into Nalu

---------------------------------------------------

The bit stream Sodb (String of Data bits) output from the VCL layer is processed in the following three steps, between Nal_unit:

1.SODB byte alignment is encapsulated into rbsp (Raw byte Sequence Payload).

2. In order to prevent the rbsp byte-stream and the SCP (start_code_prefix_one_3bytes,0x000001) in the ordered Word stream mode, there is a race case, the loop detects the first three bytes of the rbsp, In the case of byte contention, add Emulation_prevention_three_byte (0x03) before the third byte, specifically:

Nal_unit (numbytesinnalunit) {

Forbidden_zero_bit

Nal_ref_idc

Nal_unit_type

numbytesinrbsp = 0

for (i = 1; i < Numbytesinnalunit; i++) {

if (i + 2 < numbytesinnalunit && next_bits) = = 0x000003) {

rbsp_byte[numbytesinrbsp++]

rbsp_byte[numbytesinrbsp++]

i + = 2

Emulation_prevention_three_byte

} else

rbsp_byte[numbytesinrbsp++]

}

}

3. Anti-byte competition after processing the RBSP plus one byte header (forbidden_zero_bit+ nal_ref_idc+ nal_unit_type), encapsulated into Nal_unit.

------------------------------------------------

Process two, RTP packaging for Nalu

------------------------------------------------

One, Nalu packaged into RTP in three ways:

1. Single NAL unit mode
That is, a RTP packet consists of only one complete nalu. In this case, the RTP NAL header Type field and the original H.
The Nalu header Type field is the same.

2. Combo Packet mode
This may be a RTP package consisting of multiple NAL units. There are 4 combinations: stap-a, Stap-b, MTAP16, MTAP24.
The type values here are 24, 25, 26, and 27, respectively.

3. Shard Packet Mode
Used to encapsulate a NALU unit into multiple RTP packets. There are two types of fu-a and fu-b. The type values are 28 and 29, respectively.

Remember the definition of the previous nal_unit_type, 0~23 is for H264 use, 24~31 unused, in RTP packaging, if a nalu placed in a RTP packet, you can use Nalu Nal_unit_type, However, when you need to package multiple Nalu into a single RTP package, or if you need to package a NALU into multiple RTP packets, define a new type to identify it.

Type Packet type name
---------------------------------------------------------
0 undefined-
1-23 NAL Unit single NAL unit packet per h.
Stap-a Single-time Aggregation Packet
Stap-b Single-time Aggregation Packet
MTAP16 multi-time Aggregation Packet
MTAP24 multi-time Aggregation Packet
Fu-a Fragmentation Unit
Fu-b Fragmentation Unit
30-31 undefined

Two or three ways to package the specific format

1. Single NAL unit mode

For NALU packets with a length less than the MTU size, a single NAL cell pattern is generally used.
For an original H. Nalu unit is usually composed of [start code] [Nalu Header] [Nalu Payload] three parts, where Start code is used to indicate that this is a

The start of the Nalu unit must be "00 00 00 01" or "XX", Nalu only one byte, followed by Nalu unit content.
When packaging, remove the "00 00 01" or "00 00 00 01" Of the start code, the other data packets to the RTP packet.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| F| nri| Type | |
+-+-+-+-+-+-+-+-+ |
| |
| Bytes 2..N of a single NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :... OPTIONAL RTP Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

If there is a nalu of H. S:

[xx] A0 1E 0E 2F ...]

This is a sequence parameter set of NAL cells. [00 00 00 01] is the start code of four bytes, 67 is the NALU header, and 42 begins with the Nalu content.

Package into RTP package will be as follows:

[RTP Header] [A0 1E 0E 2F]

That is, just get rid of the 4-byte start code.

2 Combo Packet mode

Second, when the length of the Nalu is special, several NALU units can be sealed in a RTP packet.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Stap-a NAL HDR | Nalu 1 Size | Nalu 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Nalu 1 Data |
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Nalu 2 Size | Nalu 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Nalu 2 Data |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :... OPTIONAL RTP Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

3 Fragmentation Units (FUs).

When the length of the Nalu exceeds the MTU, the Nalu unit must be partitioned into packets. Also known as Fragmentation Units (FUs).

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU Indicator | FU Header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| FU Payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :... OPTIONAL RTP Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 14. RTP Payload format for FU-A

FU indicator has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| F| nri| Type |
+---------------+
Fu indicates the type of byte of the field type=28 represents Fu-a. The value of the NRI field must be set according to the value of the NRI domain of the Shard nal cell.

The format of the FU header is as follows:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| s| e| r| Type |
+---------------+
S:1 bit
When set to 1, the start bit indicates the start of the Shard nal unit. When the following FU load is not the beginning of the fragment NAL unit load, the starting bit is set to 0.
E:1 bit
When set to 1, the end bit indicates the end of the Shard nal cell, that is, the last byte of the load is also the last byte of the Shard nal cell. When the following FU load is not the last shard of the Shard nal unit, the end bit is set to 0.
R:1 bit
The reserved bit must be set to 0 and the receiver must ignore the bit.
Type:5 bits

Three, unpacking and settlement package

Unpacking: When the encoder needs to be encoded in the original nal according to Fu-a, the original NAL unit head and the fragment of the Fu-a cell header has the following relationship:
The first three bits of the original NAL head are the first three bits of the FU indicator, the last five bits of the original NAL head are the last five digits of the FU header, and the remaining digits of the FU indicator and FU headers are determined according to the actual situation.

Unpacking: When the receiving end receives Fu-a shard data, it is necessary to restore all the Shard package combinations to the original NAL package, the relationship between the Fu-a cell header and the restored nal is as follows:
The eight bits of the restored NAL head are made up of the last five bits of FU indicator's first three FU headers, namely:
Nal_unit_type = (Fu_indicator & 0xe0) | (Fu_header & 0x1f)

H264 frame structure Analysis and frame judging