A detailed description of the basic and RTP packaging of H.

I. H264 Basic Concepts

1, NAL, slice and frame meaning and mutual relations

1 frame data can be divided into multiple slice.
The data in each slice, in-frame prediction uses only the data of its own slice, and no dependency on other slice data.
The NAL is used to carry the encoded data into large packets. For example, each slice data can be placed in a NAL package.
I-frame is self-coded and does not depend on other frame data.
P frame relies on the I frame data.
b frame relies on I frame, P frame, or other B frame data.

A frame can be divided into multiple slice to encode, and a slice code is packaged into a nal unit, but the NAL unit can accommodate other data, such as the sequence parameter set SPS, in addition to the slice encoded stream.


NAL refers to the network extraction layer, which contains some network-related information
Slice is the meaning of the film, 264 in the image into a frame (frame) or two fields (field), and the frame can be divided into one or several slices (slilce), the slices are composed of macro blocks (MB). A macro block is a basic unit of encoding processing.

2, NAL Nal_unit_type 1 (non-IDR image encoding strip), 2 (coded stripe data Block a), 3 (coded stripe data partition block B), 4 (coded stripe data block C), 5 (IDR image encoding stripe) type
Three coding modes with slice species: I_slice, P_slice, B_slice
The five types of NAL Nal_unit_type represent what information is then and how it is divided.
I_slice, P_slice, b_slice represent I-type slices, P-type slices, and B-type slices. I_slice is the intra-frame predictive mode encoding, P_slice is one-way predictive encoding or intra-frame mode, and B_slice is bidirectional prediction or intra-frame mode.

3, there are 3 types of frame: I frame, P frame, B frame what is the mapping relationship between.
I-Frame, P-frame, B-frame relationship with I_slice, P_slice, B_slice,slice, and frame differences are already understood in question 1.

4, finally, NAL Nal_unit_type in the 6 (SEI), 7 (SPS), 8 (PPS) belong to what frame it.
NAL Nal_unit_type is the concept of a sequence parameter set (SPS), image parameter set (PPS), and enhanced information (SEI) that does not belong to a frame. Indicates that the following data information is a sequence parameter set (SPS), image parameter set (PPS), enhanced information (SEI).

Second, H264 RTP package Detailed---reproduced

H. A video RTP payload format

1. Network Abstraction Layer Unit type (NALU)

The Nalu header consists of a byte with the following syntax:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| F|  nri| Type |
+---------------+

F:1 a bit.
Forbidden_zero_bit. This one must be 0, as stipulated in the H.

Nri:2 a bit.
NAL_REF_IDC. taking 00 ~ 11, it seems to indicate the importance of this nalu, such as 00 of the Nalu decoder can discard it without affecting the playback of the image. But generally not too concerned

This property.

Type:5 a bit.
Nal_unit_type. The type of this Nalu unit. The following are briefly described:

0 Not defined
1-23 NAL Unit Single NAL unit package.
Stap-a Single-time combo package
Stap-b Single-time combo pack
MTAP16 Multi-time combo pack
MTAP24 Multi-time combo pack
Fu-a Shards of cells
Elements of Fu-b shards
30-31 Not defined

2. Packaging mode

The following is the structure of the RTP header as specified in RFC 3550.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| v=2| p|  X| CC |     m|       PT | Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| Contributing Source (CSRC) identifiers |
| ....                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Load Type Payload type (PT): 7 bits
  serial numbers Sequence number (SN): Bits
  timestamp timestamp:32 bits
& nbsp;  the
  H. Payload format defines three different basic load (PAYLOAD) structures. The receiving end may identify them through the first byte of RTP payload 
 . This byte is similar to the NALU header format, and the NAL cell type field of the header structure,
 , indicates which structure is represented,

The structure of this byte is as follows, it can be seen that it is the same as H. NALU head structure.
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| F|  nri| Type |
+---------------+
Field type: The type of NAL unit in this RTP payload. The difference between this field and the Type field in H. S is that when the type
A value of 24 ~ 31 means that this is a special format of the NAL unit, while in H. 1~23 is a valid value.

Stap-a Single-time combo package
Stap-b Single-time combo pack
MTAP16 Multi-time combo pack
MTAP24 Multi-time combo pack
Fu-a Shards of cells
Elements of Fu-b shards
30-31 Not defined

Possible types of structures are:

1. Single NAL unit mode
That is, a RTP packet consists of only one complete nalu. In this case, the RTP NAL header Type field and the original H.
The Nalu header Type field is the same.

2. Combo Packet mode
This may be a RTP package consisting of multiple NAL units. There are 4 combinations: stap-a, Stap-b, MTAP16, MTAP24.
The type values here are 24, 25, 26, and 27, respectively.

3. Shard Packet Mode
Used to encapsulate a NALU unit into multiple RTP packets. There are two types of fu-a and fu-b. The type values are 28 and 29, respectively.

2.1 Single NAL Unit mode

For NALU packets with a length less than the MTU size, a single NAL cell pattern is generally used.
For an original H. Nalu unit is usually composed of [start code] [Nalu Header] [Nalu Payload] three parts, where Start code is used to indicate that this is a

The start of the Nalu unit must be "00 00 00 01" or "XX", Nalu only one byte, followed by Nalu unit content.
When packaging, remove the "00 00 01" or "00 00 00 01" Of the start code, the other data packets to the RTP packet.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| F|  nri|                                               Type | |
+-+-+-+-+-+-+-+-+                                               |
| |
| Bytes 2..N of a single NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :... OPTIONAL RTP Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

If there is a nalu of H. S:

[xx] A0 1E 0E 2F ...]

This is a sequence parameter set of NAL cells. [00 00 00 01] is the start code of four bytes, 67 is the NALU header, and 42 begins with the Nalu content.

Package into RTP package will be as follows:

[RTP Header] [A0 1E 0E 2F]

That is, just get rid of the 4-byte start code.

2.2 Combo Packet Mode

Second, when the length of the Nalu is special, several NALU units can be sealed in a RTP packet.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Stap-a NAL HDR | Nalu 1 Size | Nalu 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Nalu 1 Data |
:                                                               :
+               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Nalu 2 Size | Nalu 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Nalu 2 Data |
:                                                               :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :... OPTIONAL RTP Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.3 Fragmentation Units (FUs).

When the length of the Nalu exceeds the MTU, the Nalu unit must be partitioned into packets. Also known as Fragmentation Units (FUs).

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   FU Indicator |                               FU Header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
| |
| FU Payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :... OPTIONAL RTP Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 14. RTP Payload format for FU-A

The FU indicator octet has the following format:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| F|  nri| Type |
+---------------+

The FU header has the following format:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| s| e|  r| Type |
+---------------+

3. SDP Parameters

The following describes how to represent a h: stream in the SDP.

. The media name in the "m=" line must be "video"
. The encoding name in the "A=rtpmap" line must be "H264".
. The clock frequency in the "A=rtpmap" line must be 90000.
. Other parameters are included in the "A=fmtp" line.

Such as:

M=video 49170 RTP/AVP 98
A=rtpmap:98 h264/90000
A=fmtp:98 profile-level-id=42a01e; sprop-parameter-sets=z0iacpztbymi,amljia==

Some common parameters are described below.

3.1 Packetization-mode:
Represents the supported packet pattern.
When the value of Packetization-mode is 0 o'clock or does not exist, a single Nalu cell mode must be used.
Non-interleaved (non-interleaved) encapsulation mode must be used when the value of Packetization-mode is 1.
When the value of Packetization-mode is 2, the Interleaved (interleaved) packet mode must be used.
This parameter is not allowed to take other values.

3.2 Sprop-parameter-sets:
This parameter can be used to transmit the sequence parameter set and the image parameter NAL unit of H. The value of this parameter is encoded using BASE64. Separate parameter sets are separated by "," numbers.

3.3 Profile-level-id:
This parameter is used to indicate the profile type and level of the H. 3 bytes represented by BASE16 (hexadecimal). The first byte represents the profile type of H.

The three bytes represent the profile level of H +:

3.4 max-mbps:
The value of this parameter is an integral type, which indicates the maximum processing speed of the macro block per second.

A developer's diary of a x264 Library abroad:

http://x264dev.multimedia.cx/archives/249