RTP Packaging Process

1. Network abstraction layer unit type (NALU)

The NALU header consists of one byte,
Its syntax is as follows:

+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| F | NRI | type |
+ --------------- +

F: 1 bit.
Forbidden_zero_bit. The H.264 specification specifies that this digit must be 0.

NRI: 2 bits.
Nal_ref_idc. 00 ~ 11. It seems to indicate this nalu
For example, 00 NALU decoder can discard it without affecting image playback.

This attribute.

Type: 5 bits.
Nal_unit_type. Type of the NALU unit. The description is as follows:

0 is not defined
1-23 nal unit Single nal
Unit package.
24 STAP-A
Single-time Combination Package
25 STAP-B single time combo pack
26 mtap16 time Combination Package
27 mtap24 time Combination Package
28 unit of FU-A Fragment
29 unit of FU-B Fragment
30-31 Not Defined

2. Packaging Mode

The following is the structure of the RTP Header specified in RFC 3550.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| V = 2 | p | x | CC | M | Pt | sequence number |
+- +-+
| Timestamp |
+- +-+
| Synchronization source (SSRC) identifier |
+ = + + = +
| Contributing source (CSRC) identifiers |
|... |
+- +-+

Payload type (PT): 7 bits
Serial number sequence number (SN): 16 bits
Timestamp: 32 bits

H.264 Payload
The format defines three basic payload structures.
The receiving end may recognize them through the first byte of RTP payload.
This byte is similar to nalu
Header Format,
And the nal of this header Structure
The unit type field indicates which structure is represented.

 

The structure of this byte is as follows. It can be seen that it has the same structure as the NALU header of H.264.
+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| F | NRI | type |
+ --------------- +
Field type: the type of the nal unit in the RTP payload. The difference between this field and the type field in H.264 is that when type
Is 24 ~ 31 indicates that this is a nal unit in special format, while in H.264, only 1 ~ 23 is a valid value.

24 STAP-A single time combo pack
25 STAP-B single time combo pack
26 mtap16 time Combination Package
27 mtap24 time Combination Package
28 unit of FU-A Fragment
29 unit of FU-B Fragment
30-31 Not Defined

Possible structure types are:

1. Single nal
Unit Mode
A rtp packet is composed of only one complete NALU. In this case, the RTP nal header type field and the original H.264
The NALU header type fields are the same.

2. Combined Packet Mode
A rtp package may be composed of multiple NAL units. There are four combination methods: STAP-A, STAP-B, mtap16, mtap24.
The type values here are 24, 25, 26, and 27.

3. multipart Packet Mode
It is used to encapsulate a NALU unit into multiple RTP packets. Two types of FU-A and FU-B exist. The type values are 28 and 29, respectively.

2.1 single nal
Unit Mode

For packages whose NALU length is smaller than MTU size, the single nal unit mode is generally used.
For an original H.264 NALU unit, it is usually composed of three parts: [start code] [NALU header] [NALU payload]. The start code indicates that this is

The beginning of the NALU unit,
It must be "00 00 00 01" or "00 00 01". The NALU header is only one byte, followed by the content of the NALU unit.
Remove the start code of "00 00 01" or "00 00 01" during packaging, and use the RTP package of other data packets.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| F | NRI | type |
+-+ |
|
| Bytes 2.. n of a single nal unit |
|
| +-+
|:... Optional RTP padding |
+- +-+

If a NALU of H.264 is like this:

[00 00 00 01 67 42 A0 1E 23 56 0e 2f...]

This is a sequential parameter set nal
Unit. [00 00 01] is the start code of four bytes, and 67 is the nalu
Header. The data starting from 42 is NALU.
Content.

The RTP package is encapsulated as follows:

[RTP Header] [67 42 A0 1E 23 56 0e 2f]

That is, you only need to remove the start code of 4 bytes.

 

2.2 combined Packet Mode

Secondly, when the length of NALU is particularly small, several NALU units can be enclosed in a RTP package.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| RTP Header |
+- +-+
| STAP-A nal HDR | NALU 1 size | NALU 1 HDR |
+- +-+
| NALU 1 Data |
::
++-+ -+
| NALU 2 size | NALU 2 HDR |
+- +-+
| NALU 2 Data |
::
| +-+
|:... Optional RTP padding |
+- +-+

Example:

If a NALU of H.264 is like this:

[00 00 00 01 67
42 A0 1E 23 56 0e 2f...]

[00 00 00 01 67 42 A0 1E 23 56 0e 2f...]

This is a sequential parameter set nal
Unit. [00 00 01] is the start code of four bytes, 67 is the NALU header, and 42 is the content of NALU.

The RTP package may be encapsulated as follows:

[RTP Header] [78, STAP-A nal HDR, one byte] [length, two bytes] [67 42 A0 1E 23 56 0e 2f...] [length, two bytes] [67 42 A0 1E 23 56 0e 2f...]

 

2.3 fragmentation units (FUS ).

When the length of NALU exceeds MTU, The NALU unit must be split into packets, also known as fragmentation units (FUS ).

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 4 5 6 7 8 9 0 1
+- +-+
| Fu indicator | fu header |
+-+ |
|
| Fu payload |
|
| +-+
|:... Optional RTP padding |
+- +-+

Figure 14. RTP payload format for FU-A

The fu indicator octet has the following format:

+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| F | NRI | type |
+ --------------- +

The fu header has the following format:

+ --------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+-+
| S | E | r | type |
+ --------------- +

 

Example:

Int geplayback: sendnalupacket (byte * slicedata, int slicesize, bool isend,

Bool isvideo, int type, time_t pts, int64 timestamp)
{
// When NALU is smaller than the maximum RTP Package Size
If (slicesize <1350 ){
Return sendpacket (slicedata, slicesize, isend, true, type, PTS, timestamp );
}

// If a NALU is larger than the maximum RTP package size, you need to split it and package it for sending.
Byte buffer [1500];

Byte nalheader = slicedata [0]; // NALU Header
Byte * Data = slicedata + 1;
Int leftover = slicesize-1;
Bool isstart = true;

While (leftover> 0 ){
Int size = min (1350, leftover );
Isend = (size = leftover );

// Construct the Fu Header
Buffer [0] = (nalheader & 0x60) | 28; // Fu indicator

Buffer [1] = (nalheader & 0x1f); // Fu Header
If (isstart ){
Buffer [1] | = 0x80;
}
If (isend ){
Buffer [1] | = 0x40;
}

Memcpy (buffer + 2, data, size );
Sendpacket (buffer, size + 2, isend, true, type, PTS, timestamp );

Leftover-= size;
Data + = size;
Isstart = false;
}

Return slicesize;
}