H264 NAL RTP Packaging

1. Network Abstraction Layer Unit type (NALU)

Nalu is H264 used for network transmission of the unit type, a complete Nalu unit is generally 0x000001 or 0x00000001 start, followed by Nalu head and NALU data; When we were transmitting in the network, Will remove the start 0x000001 or 0x00000001, and you will typically need to replace them with the head of the RTP payload (1 bytes);

Where NALU data is rbsp data;

The Nalu header consists of a byte with the following syntax:

     +---------------+      |0|1|2|3|4|5|6|7|      +-+-+-+-+-+-+-+-+      | F| nri|  Type   |      +---------------+

f:1 a bit. Forbidden_zero_bit. This one must be 0, as stipulated in the H.

nri:2 a bit. NAL_REF_IDC. Importance indication, mark the importance of the NAL unit, the greater the value, the more important, the decoder in the decoding process, you can lose the importance of 0 Nalu, take 00 ~ 11.

NAL_REF_IDC. The semantics of 0 values and non-0 values are consistent with the H. In other words, a value of 00 indicates that the contents of the NAL unit are not used to reconstruct inter-frame image predictions that affect images. Such nal units can be discarded without risking the image integrity.       A value greater than 00 indicates that the decoding of the NAL unit requires maintaining the integrity of the referenced image. In addition to the above designation, according to this RTP load specification, a NRI value greater than 00 indicates a relative transmission priority, as determined by the encoder. Mane can use this information to protect more important nal units. The highest transmission priority is 11, followed by 10, 01;00 the lowest. (Note: The processing of any nonzero NRI at the H. Therefore, the receiver does not have to manipulate the value of NRI when transmitting the NAL unit to the decoder)

The encoder must set the NRI value according to the H. A specification. When the Nal_unit_type range is 1 to 12. In particular, the H. 0 specification requires that the value of the NRI of the NAL unit Nal_unit_type to 6,9,10,11,12 be 7, and for Nal_unit_type equal to 8 (indicating the set of sequential parameter sets or image parameter sets), the NAL unit The encoder should be set to NRI 11 (binary format), and for Nal_unit_type equal to 5 of the main encoded image of the encoder NAL unit (indicating that the encoder belongs to an IDR image), the H. B encoder should be set NRI to one; for mapping other Nal_ Unit_types to NRI values, can be used, and in some environments valid, other mappings can also, depending on the application and the use of H.264/AVC Annex A profile. (Note: Data partitioning is not available in some profiles, that is, in main or baseline profiles.) Therefore, the NAL unit type 2, 3, 4 appears only in the video stream conforming to the data partition is allowed by the profile case, will not appear in the flow that conforms to main/baseline profiles)

Table 2. Example of the NRI value of the encoder and the encoded slice of the main coded reference image data partition

      NAL Unit type     Content of NAL unit               NRI (binary)      --------------- -------------------------------------------------       1               NON-IDR Coded slice                             10        2               Coded Slice Data partition a                        3               Coded Slice Data partition B       &Nbsp;               4               Coded Slice Data partition c                 01

(Note: Non-reference image NRI value is 00)

The encoder should set the redundant coded reference image of the encoder and the chip partition NAL unit NRI value is 01 (binary format); for NAL unit type 24~29 NRI is defined in the second figure below; for Nal_unit_ The NRI value for the type range of nal cells from 13 to 23 does not have a recommended value because these values are reserved for Itu-t,iso/iec. There is no recommended value for NRI values for nal cells that are nal_unit_type 0 or 30,31, because the semantics of these values are not specified in this document.

type:5 a bit. Nal_unit_type. The type of this Nalu unit. The following are briefly described:

0 The 1-23 NAL unit is not defined as a single NAL unit package. The combination package of Stap-a single time Stap-b single time combination package of more than MTAP16 multiple time combination package MTAP24 multiple time combination package of Fu-a Shard Unit 30-31 of Fu-b shard not defined

(c denotes importance)

2. Packaging mode

The following is the structure of the RTP header as specified in RFC 3550.

 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- +-+-+-+-+-+-+-+-+-+      | v=2| p|  X| CC |     m|       PT |      Sequence Number |                           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+      |      Timestamp |           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+      |      Synchronization source (SSRC) identifier |            +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+      |      Contributing Source (CSRC) identifiers |                             |      ....                              | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Load type Payload type (PT): 7 bits serial number Sequence numbers (SN): bits timestamp Timestamp: bits H The. Payload format defines three different basic load (PAYLOAD) structures. The receiving end may identify them through the first byte of the RTP Payload. This byte is similar to the NALU header format, and the NAL cell type field of the header structure indicates which structure is represented.

The structure of this byte is as follows, it can be seen that it is the same as H. NALU head structure.

       +---------------+      |0|1|2|3|4|5|6|7|      +-+-+-+-+-+-+-+-+      | F| nri|  Type   |      +---------------+

Field type: This is the type of NAL unit in RTP payload. The difference between this field and the Type field in H: 24 to 31 indicates that this is a special format of the NAL unit, and that only 1~23 is a valid value in H.

Possible types of structures are:

1. Single NAL unit mode i.e. one RTP packet consists of only one complete nalu. In this case, the RTP NAL header Type field is the same as the original H. Nalu Header Type field.

2. The combined packet mode may consist of multiple NAL units consisting of one RTP package. There are 4 combinations: stap-a, Stap-b, MTAP16, MTAP24. The type values here are 24, 25, 26, and 27, respectively.

3. The Shard packet mode is used to encapsulate a NALU unit into multiple RTP packets. There are two types of fu-a and fu-b. The type values are 28 and 29, respectively.

2.1 Single NAL Unit mode

For NALU packets with a length less than the MTU size, a single NAL cell pattern is generally used.

For an original H. Nalu unit is usually composed of [start code] [Nalu Header] [Nalu Payload] three parts, where Start code is used to indicate that this is a

The start of the Nalu unit must be "00 00 00 01" or "XX", Nalu only one byte, followed by Nalu unit content. When packaging, remove the "00 00 01" or "00 00 00 01" Of the start code, the other data packets to the RTP packet.

     0                   1                   2                   3     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    | F| nri|  Type   |                                               |    +-+-+-+-+-+-+-+-+                                               |    |                                                               |    |               Bytes 2..N of a single NAL unit                 |    |                                                               |                               | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    |                               :... OPTIONAL RTP padding        |    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 1.  RTP payload format for single NAL unit packet

This generally applies to nal packets less than 1460 bytes of the case, but not too small, too small to need to pack, discussed later; the RTP payload of a single NAL unit package is as above;

In other words, the NAL head and NAL data are placed directly in the RTP payload position, because at this time the RTP payload head and the NAL head is the same, so only need to use one of them;

If there is a nalu of H. S:

[xx] A0 1E 0E 2F ...]

This is a sequence parameter set of NAL cells. [00 00 00 01] is the start code of four bytes, 67 is the NALU header, and 42 begins with the Nalu content.

Package into RTP package will be as follows:

[RTP Header] [A0 1E 0E 2F]

That is, just get rid of the 4-byte start code.

2.2 Combo Packet Mode

Second, when the length of the Nalu is special, several NALU units can be sealed in a RTP packet.

     0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6                          7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |    RTP Header | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    |         Stap-a NAL HDR | Nalu 1 Size |    Nalu 1 HDR |                         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    |    Nalu 1 Data | :                                                               :    +               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               -+-+-+-+-+-+    | | Nalu 2 Size |    Nalu 2 HDR |                         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    |    Nalu 2 Data |                               :                                                               :    | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    | :...    OPTIONAL RTP Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 2.  An example of a RTP packet including an stap-a               containing the Single-time aggregation units

This is generally used for multiple small nal packages packaged into a large nal package, of course no more than 1460 bytes; The RTP payload structure of this case is roughly the same.

Note:

Hdr=header represents the head;

Stap-a NAL HDR is actually a byte of the head of RTP payload, the value should be 00011000b (stap-a type = 24, binary is 11000);

Nalu size occupies two bytes; Nalu head one byte; Note that the data portion is not aligned by 32 bytes, while the Nalu size does not contain 2 bytes of itself;

2.3 Fragmentation Units (FUs).

When the length of the Nalu exceeds the MTU, the Nalu unit must be partitioned into packets. Also known as Fragmentation Units (FUs).

     0                   1                   2                   3     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    | FU Indicator  |   FU Header   |                               |    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |    |                                                               |    |                         FU payload    |                               | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    |                               :... OPTIONAL RTP padding        |    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 3. RTP Payload format for FU-A

A NAL packet is split into multiple nal packages, which is typically a NAL packet that exceeds 1460 bytes, and the payload structure of the case is as above.

Note:

One of the FU indicator corresponds to the head of the RTP payload, here should be 0x00011100b (fu-a=28, binary representation that is 11100);

FU indicator is defined as follows:

        +---------------+       |0|1|2|3|4|5|6|7|       +-+-+-+-+-+-+-+-+       | F| nri|  Type   |       +---------------+

The FU header is defined as follows:

        +---------------+      |0|1|2|3|4|5|6|7|      +-+-+-+-+-+-+-+-+      | s| e| r|  Type   |      +---------------+

For the first packet of the Shard, set S is 1, and for the last packet of the Shard, set E to 1,r to 0, and the Type field to the type in the Nalu header, which can be used to differentiate the frame type;

Note that the FU payload does not transmit Nalu's head, Nalu's head consists of Fu indicator (top 3) and Fu header (last five): Nal_unit_type = (Fu_indicator & 0xe0) | (Fu_header & 0x1f);

Technorati Tags: h264,rtp pack, pack, unpack unpacking

Reference article:

http://blog.csdn.net/heanyu/article/details/6109957 (NRI description Reference)

Http://www.cnblogs.com/frkang/p/3352251.html (RTP packet description is clearer)

https://tools.ietf.org/html/rfc6184 (Standard, RTP Payload Format for H. Video)

http://m.blog.csdn.net/blog/yangguangmeng/25562921 (I, P, B-Frame, SPS, pps judgment)

http://blog.csdn.net/ljzcom/article/details/9834405 (Miscellaneous)

http://www.360doc.com/content/13/0124/08/9008018_262076786.shtml (Miscellaneous)

HTTP://BLOG.CSDN.NET/JWYBOBO2007/ARTICLE/DETAILS/7054140 (Combination packet example seems to be wrong, STAP-A head is not 78 should be 24)

Http://www.cnblogs.com/likwo/p/3533392.html (places to be aware of in practice)

Http://blog.sina.com.cn/s/blog_3f0f1d3701013mta.html (Example Analysis H264 RTP payload)

H264 NAL RTP Packaging