The RTP package for the H264 of RTSP learning

1. H264 Basic Concepts

1.1, NAL, slice and frame introduction and interrelated

NAL refers to the network extraction layer, which put some network-related information.

Slice is the meaning of the film, 264 in the image into a frame (frame) or two fields (field), and the frame can be divided into one or several slices (slilce), the slices are composed of macro blocks (MB). A macro block is a basic unit of encoding processing.

A frame can be divided into multiple slice to encode, and a slice is packaged into a NAL unit after encoding. The data in each slice, in-frame prediction uses only the data of its own slice, and no dependency on other slice data. NAL is used to package encoded data, and each slice data can be placed in a NAL package. However, the NAL unit can accommodate other data, such as the sequence parameter set SPS, in addition to the slice encoded code stream.

I-frame is self-coded and does not depend on other frame data.

P frame relies on the I frame data.

b frame relies on I frame, P frame, or other B frame data.

1.2, NAL nal_unit_type

There are 1 (coding strips for non-IDR images), 2 (coded stripe data Block a), 3 (coded stripe data block B), 4 (coded stripe data block C), 5 (coded stripe for IDR image), five types of Nalnal_unit_type

Slice has three encoding modes: I_slice, P_slice, B_slice

The five types of nalnal_unit_type represent what information is represented next and how it is divided into chunks. I_slice, P_slice, b_slice represent I-type slices, P-type slices, and B-type slices. The I_slice is the intra-frame predictive mode encoding, the P_slice is one-way predictive encoding or intra-frame mode, and the B_slice is bidirectional prediction or intra-frame mode.

1.3, Frame of 3 types: I frame, P frame, B frame between the mapping relationship:

The differences between the I frame, P frame, and B frame relationships with I_slice, P_slice, B_slice,slice, and frame are illustrated in 1.1.

1.4, NAL nal_unit_type parameter set

The frames in the Nalnal_unit_type are divided into: Sequence parameter set (SPS), image parameter set (PPS), enhanced information (SEI), which is not part of the frame concept, indicating that subsequent data information is a sequence parameter set (SPS), image parameter set (PPS), enhanced information (SEI).

2. Video RTP payload Format

2.1. Network Abstraction Layer Unit type (NALU)

The Nalu header consists of a single byte, which is in the following format:

Description :

F:1 bit. Forbidden_zero_bit. This one must be 0 and set to 1 to indicate a syntax violation in the H.

Nri:2 bit. NAL_REF_IDC, taking 00 ~ 11, indicates the importance of this nalu, indicating that the NAL unit is not used for reconstructing reference images of inter-frame image prediction. Such nal units can be discarded without risking the referential image integrity. A value greater than 0 indicates that the decoding of the NAL unit requires maintaining the integrity of the reference image. However, in general, this attribute is not very concerned.

Type:5 bit. Nal_unit_type. The type of this NALU unit is summarized as follows:

0 Not defined

1-23 NAL Unit Single NAL unit package.

Stap-a Single-time combo package

Stap-b Single-time combo pack

MTAP16 Multi-time combo pack

MTAP24 Multi-time combo pack

Fu-a Shards of cells

Elements of Fu-b shards

30-31 Not defined

2.2. RTP Package Format

The following is the structure of the RTP header as specified in RFC 3550:

RTP Header Information Setup Description:

Markerbit (M): The bit is set when the 1 bit RTP payload is the last package of the NAL unit. The decoder can use this bit as an indication of the last package, but it can not rely on this property.

Note: Aggregation packages that ship multiple NAL units have only one m-phase association.

Payload type (PT): 7 bits nal unit load type that represents the type of parameter set included in the NAL Nal_unit_type above for the current load.

SequenceNumber (SN): bits are used according to RFC 3550 settings, and for single Nalu and non-interleaved packaging, the sequence number is used to decode the Nalu sequence.

The timestamp:32 bits RTP timestamp is set to the sample time stamp of the content. You must use a clock frequency of up to khz.

If the NAL cell does not have his own time attribute (that is, parameter set and SEI NAL units), the RTP timestamp is set to access the RTP timestamp of the unit's primary encoded image.

The receiver should ignore any image time Sei messages contained in the Access unit (only one display timestamp), instead, the receiver should synchronize the display process with the RTP timestamp.

The H. Payload format defines three different basic load (PAYLOAD) structures, which can be identified by the first byte of the RTP Payload, a byte similar to the Nalu header, and the NAL of the header structure The cell Type field indicates which structure is represented, and the structure of the byte is as follows:

It can be seen that it is the same as the NALU head structure of H.

Field type: The type of the NAL unit in this RTP payload, and the difference between this field and the Type field in H. 24 is that when the value of type is 31, it means that this is a special format of the NAL unit, and that only the 1~23 is a valid value in H.

Stap-a Single-time combo package

Stap-b Single-time combo pack

MTAP16 Multi-time combo pack

MTAP24 Multi-time combo pack

Fu-a Shards of cells

Elements of Fu-b shards

30-31 Not defined

Possible types of structures are:

(1). Single NAL unit mode

That is, a RTP packet consists of only one complete nalu. In this case, the RTP NAL header Type field is the same as the original H. Nalu Header Type field.

(2). Combo Packet mode

This may be a RTP package consisting of multiple NAL units. There are 4 combinations: stap-a, Stap-b, MTAP16, MTAP24, and the type values are 24, 25, 26, and 27, respectively.

(3). Shard Packet Mode

Used to encapsulate a NALU unit into multiple RTP packets. There are two types of fu-a and fu-b, and the type values are 28 and 29, respectively.

2.3. Single NAL unit mode

For NALU packets with a length less than the MTU size, a single NAL cell pattern is generally used:

For an original H. 00 Nalu unit is usually composed of [start code] [Nalu Header] [Nalu Payload] Three parts, where start code is used to indicate that this is the beginning of a NALU unit, must be "00 00 01" or "xx", Nalu head only one byte, followed by Nalu unit content. When packaging, remove the "00 00 01" or "00 00 00 01" Of the start code, the other data packets to the RTP packet.

If there is a nalu of H. S:

[xx] A0 1E 0E 2F ...]

This is a sequence parameter set of NAL cells. [00 00 00 01] is the start code of four bytes, 67 is the NALU header, and 42 begins with the Nalu content.

Package into RTP package will be as follows:

[RTP Header] [A0 1E 560E 2F]

That is, just get rid of the 4-byte start code.

2.4. Combination packet mode

Second, when the length of the Nalu is special, several NALU units can be sealed in a RTP packet:

2.5, Fragmentation Units (FUs).

When the length of the Nalu exceeds the MTU, the Nalu unit must be partitioned into packets. Also known as Fragmentation Units (FUs), the fu-a shard pattern commonly used is shown in the following table:

The fu-a consists of a 1-byte shard indicator unit, a 1-byte shard cell header, and a shard element load.

The Shard indicator cell format is as follows:

The Type field is a value of (fu-a) or (fu-b). The value of the NRI field must be set according to the value of the NRI domain of the Shard nal cell.

The head format of FU is as follows table:

S:1 bit. When set to 1, the start bit indicates the start of the Shard nal unit. When the following FU load is not the beginning of the fragment NAL unit load, the starting bit is set to 0.

E:1 bit. When set to 1, the end bit indicates the end of the Shard nal cell, i.e. the last byte of the load is also the last byte of the Shard nal cell. When the following FU load is not the last shard of the Shard nal unit, the end bit is set to 0.

R:1 bit. The reserved bit must be set to 0 and the receiver must ignore the bit.

Type:5 bits. The NAL unit load type that represents the type of parameter set contained in the NAL Nal_unit_type above for the current load.

3. H264 Common SDP parameter description

The following describes how to represent a h: stream in the SDP.

. The media name in the "m=" line must be "video"

. The encoding name in the "A=rtpmap" line must be "H264".

. The clock frequency in the "A=rtpmap" line must be 90000.

. Other parameters are included in the "A=fmtp" line.

Such as:

M=video 49170 RTP/AVP 98

A=rtpmap:98 h264/90000

A=fmtp:98 profile-level-id=42a01e;sprop-parameter-sets=z0iacpztbymi,amljia==

Some common parameters are described below. 

3.1, Packetization-mode:

Represents the supported packet pattern.

When the value of Packetization-mode is 0 o'clock or does not exist, a single Nalu cell mode must be used.

Non-interleaved (non-interleaved) encapsulation mode must be used when the value of Packetization-mode is 1.

When the value of Packetization-mode is 2, the Interleaved (interleaved) packet mode must be used.

This parameter is not allowed to take other values.

 3.2, Sprop-parameter-sets:

This parameter can be used to transmit the sequence parameter set and the image parameter NAL unit of H. The value of this parameter is encoded using BASE64. Separate parameter sets are separated by "," numbers.

3.3, Profile-level-id:

This parameter is used to indicate the profile type and level of the stream, 3 bytes represented by BASE16 (hex), the first byte represents the profile type of H. x, and the third byte represents the profile level of H.

3.4, max-mbps:

The value of this parameter is an integral type, which indicates the maximum processing speed of the macro block per second.