H.264 basic and RTP packets

I. Basic concepts of h264

1. NAL, slice and frame meanings and Their Relationships

1 frame data can be divided into multiple slice.
Data in each slice only uses the data of its own slice for intra-frame prediction, and has no dependency with other slice data.
NAL is used to pack the encoded data. For example, each slice data can be stored in the nal package.
I frame is independently encoded and independent from other frame data.
P frame depends on I frame data.
B frame depends on I frame, P frame, or other B frame data.

A frame can be divided into multiple slice codes, while a slice code is packaged into a nal unit. However, in addition to the slice code stream, the nal Unit, it can also accommodate other data, such as the sequence parameter set SPs.

NAL refers to the network extraction layer, which contains some network-related information.
Slice means slice. In 264, the image is divided into one frame or two fields, and the frame can be divided into one or several slilce ); A piece consists of a macro block (MB. Macro Block is the basic unit for encoding.

2. 1 (encoding band of non-IDR image), 2 (encoding Band Data Segment A), and 3 (encoding band data segment B) in nal nal_unit_type), 4 (encoding Band Data Segment c), 5 (IDR image encoding band) Types
Three encoding modes: I _slice, p_slice, and B _slice
The five types in nal nal_unit_type indicate what information is next and how to block specific data.
I _slice, p_slice, and B _slice indicate I-type slice, p-type slice, and B-type slice. here, I _slice is the intra-frame prediction mode encoding; p_slice is the unidirectional prediction encoding or intra-frame mode; B _slice is the bidirectional prediction or intra-frame mode.

3. Are there three types of frames: I frame, P frame, and B frame?
The differences between I frame, P frame, and B frame are as follows: I _slice, p_slice, B _slice, slice, and frame, which are already known in question 1.

4. At last, what frames are 6 (SEI), 7 (SPS), and 8 (PPS) in nal nal_unit_type?
NAL nal_unit_type is a sequence parameter set (SPS), image parameter set (PPS), and enhancement information (SEI. It indicates that the subsequent data information includes the sequence parameter set (SPS), image parameter set (PPS), and enhancement information (SEI ).

Ii. Detailed explanation of h264 RTP packets --- reprint

H.264 video RTP load format

1. Network abstraction layer unit type (NALU)

The NALU header consists of one byte. Its syntax is as follows:

+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| F | NRI | type |
+ --------------- +

F: 1 bit.
Forbidden_zero_bit. The H.264 specification specifies that this digit must be 0.

NRI: 2 bits.
Nal_ref_idc. 00 ~ 11. It seems to indicate the importance of this NALU. For example, the NALU decoder of 00 can discard it without affecting the playback of the image.

This attribute.

Type: 5 bits.
Nal_unit_type. Type of the NALU unit. The description is as follows:

0 is not defined
1-23 nal unit Single nal unit package.
24 STAP-A single time combo pack
25 STAP-B single time combo pack
26 mtap16 time Combination Package
27 mtap24 time Combination Package
28 unit of FU-A Fragment
29 unit of FU-B Fragment
30-31 Not Defined

2. Packaging Mode

The following is the structure of the RTP Header specified in RFC 3550.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| V = 2 | p | x | CC | M | Pt | sequence number |
+- +-+
| Timestamp |
+- +-+
| Synchronization source (SSRC) identifier |
+ = + + = +
| Contributing source (CSRC) identifiers |
|... |
+- +-+

Payload type (PT): 7 bits
Serial number sequence number (SN): 16 bits
Timestamp: 32 bits

H.264 the payload format defines three basic payload structures. The receiving end may use RTP payload
The first byte to recognize them. This byte is similar to the format of the NALU header, and The Nal unit type field of this header Structure
Which structure is represented,

The structure of this byte is as follows. It can be seen that it has the same structure as the NALU header of H.264.
+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| F | NRI | type |
+ --------------- +
Field type: the type of the nal unit in the RTP payload. The difference between this field and the type field in H.264 is that when type
Is 24 ~ 31 indicates that this is a nal unit in special format, while in H.264, only 1 ~ 23 is a valid value.

24 STAP-A single time combo pack
25 STAP-B single time combo pack
26 mtap16 time Combination Package
27 mtap24 time Combination Package
28 unit of FU-A Fragment
29 unit of FU-B Fragment
30-31 Not Defined

Possible structure types are:

1. Single nal unit Mode
A rtp packet is composed of only one complete NALU. In this case, the RTP nal header type field and the original H.264
The NALU header type fields are the same.

2. Combined Packet Mode
A rtp package may be composed of multiple NAL units. There are four combination methods: STAP-A, STAP-B, mtap16, mtap24.
The type values here are 24, 25, 26, and 27.

3. multipart Packet Mode
It is used to encapsulate a NALU unit into multiple RTP packets. Two types of FU-A and FU-B exist. The type values are 28 and 29, respectively.

2.1 single nal unit Mode

For packages whose NALU length is smaller than MTU size, the single nal unit mode is generally used.
For an original H.264 NALU unit, it is usually composed of three parts: [start code] [NALU header] [NALU payload]. The start code indicates that this is

The start of the NALU unit must be "00 00 00 01" or "00 00 01". The NALU header is only one byte, followed by the content of the NALU unit.
Remove the start code of "00 00 01" or "00 00 01" during packaging, and use the RTP package of other data packets.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| F | NRI | type |
+-+ |
|
| Bytes 2.. n of a single nal unit |
|
| +-+
|:... Optional RTP padding |
+- +-+

If a NALU of H.264 is like this:

[00 00 00 01 67 42 A0 1E 23 56 0e 2f...]

This is a serial parameter set nal unit. [00 00 01] is the start code of four bytes, 67 is the NALU header, and the data starting from 42 is the content of NALU.

The RTP package is encapsulated as follows:

[RTP Header] [67 42 A0 1E 23 56 0e 2f]

That is, you only need to remove the start code of 4 bytes.

2.2 combined Packet Mode

Secondly, when the length of NALU is particularly small, several NALU units can be enclosed in a RTP package.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| RTP Header |
+- +-+
| STAP-A nal HDR | NALU 1 size | NALU 1 HDR |
+- +-+
| NALU 1 Data |
::
++-+ -+
| NALU 2 size | NALU 2 HDR |
+- +-+
| NALU 2 Data |
::
| +-+
|:... Optional RTP padding |
+- +-+

2.3 fragmentation units (FUS ).

When the length of NALU exceeds MTU, The NALU unit must be split into packets, also known as fragmentation units (FUS ).

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| Fu indicator | fu header |
+-+ |
|
| Fu payload |
|
| +-+
|:... Optional RTP padding |
+- +-+

Figure 14. RTP payload format for FU-A

The fu indicator octet has the following format:

+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| F | NRI | type |
+ --------------- +

The fu header has the following format:

+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| S | E | r | type |
+ --------------- +

3. SDP parameters

The following describes how to represent an H.264 stream in SDP:

. "M =" the media name in the row must be "video"
. The encoding name in the "A = rtpmap" line must be "h264 ".
. The clock frequency in the "A = rtpmap" row must be 90000.
. Other parameters are included in the "A = fmtp" line.

For example:

M = video 49170 RTP/AVP 98
A = rtpmap: 98 h264/ 90000
A = fmtp: 98 profile-level-id = 42a01e; sprop-parameter-sets = z0iacpztbymi, amljia =

The following describes some common parameters.

3.1 packetization-mode:
Indicates the supported encapsulation mode.
When the value of packetization-mode is 0 or does not exist, the single NALU unit mode must be used.
When the value of packetization-mode is 1, the non-interleaved packet mode must be used.
When the value of packetization-mode is 2, the interleaved packet mode must be used.
This parameter cannot be set to another value.

3.2 sprop-parameter-sets:
This parameter can be used to transmit the sequence parameter set of H.264 and The Nal unit of the image parameter. The value of this parameter is base64 encoded. Different parameter sets are separated.

3.3 profile-level-ID:
This parameter indicates the profile type and level of the H.264 stream. Three bytes represented by base16 (hexadecimal). The first byte indicates the profile type of H.264.

Three bytes indicate the profile level of H.264:

3.4 max-Mbps:
The value of this parameter is an integer that indicates the maximum macro block processing speed per second.

Developer Diary of a foreign x264 Library:

Http://x264dev.multimedia.cx/archives/249