Summary of the TS stream that can be played by iPad packaged with h264 and aac

It took me nearly three weeks to implement HTTP live streaming on the crtmpserver to package h264 and aac into ts streams and play them on the iPad through HTML5, because there is no ready-to-useCodeFor reference, the packaging code is all handwritten. For the packaging format, refer to ISO/ice 18318-1.pdf. During this period, I encountered many problems and took a lot of detours. The standard-compliant ts may not be able to play on the iPad, but the TS played on the iPad must be in line with the standard. It can be said that it is a special case in the TS standard. The main problems and key points are summarized as follows:

[Packaging process and related diagram :]

[Notes for videos from h264 to TS :]

<1>. If the h264 package is greater than 65535, you can set pes_packet_length to 0. For details, see the description of pes_packet_length in ISO/ice 13818-1.pdf 49/174.
Package PES and directly read the content of an h264 frame. In this case, we set the value of pes_packet_length to 0000.
The length of the PES package is not specified. ISO/ice 13818-1.pdf 49/174 is described, which is mainly convenient.
When the length of an h264 frame is greater than pes_packet_length (2 bytes), the maximum length is 65535.
In this case, we set the length of pes_packet_length to 0000, even if the length of the h264 video frame is
It does not need to store more than 65535 bytes in multiple PES packages. It turns out that this can be done, and the iPad can play back.
-----------------

<2> the stream_id In the PES header does not seem to matter. You can change it to a few. However, 0xe0 is set in the sample files that can be played by iPad.
And http://developer.apple.com/library/ios/#documentation/AudioVideo/Conceptual/HTTP_Live_Streaming_Metadata_Spec/2/2.html in Apple
In the table provided by 2.4 PES stream format, 0xbd is used to indicate private_stream_id_1. I do not know whether this description is specific to ID3, but it is recommended to set it to 0xeo, this is the case for several ts sample files I have seen.
-----------------

<3> for videos, PTS and DTS both need to be set to the same. If only PTS is used, it cannot be used without DTs. You can assign the PTS value to DTS. The pts_dts_flag value should be set to 0x03, that is, the binary '11'
-----------------

<4> stream_type = 0x1b in. PMT indicates h264, stream_type = 0x0f indicates AAC
-----------------

<5>. the iPad can play back without PCR. You can set pcr_pid In the PMT header to 0x1fff to indicate no PCR. For details, see ISO/ice 13818-1.pdf 65/174 and set pcr_flag in all subsequent adaptation_field files: 00, and the value of the PCR part does not need to be written in adaptation_field.
-----------------

<6> when the video frame of each frame is packaged into PES, The Nal 00 00 01 09 XX must be added at the beginning. Otherwise there will be problems
The official website of Apple has the following instructions:
Https://developer.apple.com/library/ios/#documentation/networkinginternet/conceptual/streamingmediaguide/FrequentlyAskedQuestions/FrequentlyAskedQuestions.html

10. These settings are the current recommendations. There are also certain requirements.
The current mediastreamsegmenter tool works only with the MPEG-2 transport streams as defined in ISO/IEC 13818.
The transport stream must contain H.264 (MPEG-4, Part 10) video and aac or MPEG audio.
If AAC audio is used, it must have ADTs headers. H. 264 video access units must use access unit delimiter NaLS,
And must be in unique PES packets.

Among them, H.264 video access units must use access unit delimiter NaLS's access unit delimiter nal is 09 In the h264 document, but I am not quite clear about its usage. According to the one byte following my experiment 09, it seems that everything can be set, but it cannot be set to 00.
-----------------

<7>. PTs Calculation Method

// 1 S = 90000 time scale, a frame should be 90000/video_frame_rate timescale
Static uint32_t video_frame_rate = 30;
Static uint32_t video_pts_increment = 90000/video_frame_rate; // divide by the Frame Rate in one second to obtain the time consumed for each frame. The unit is timescale.
Static uint64_t video_pts = 0;

In future calculations, for each generated frame of TS, the PTS value of this frame should be video_pts + = video_pts_increment. The pseudocode is as follows:

While (..)
{
Readh264 (hsf-_buffer)
Packetts (hsf-_buffer, out_ts_buffer, video_pts );
Video_pts + = video_pts_increment;
}

<8>. Only PTS is required for audio, and DTS is not required.
The Calculation Method of audio PTS is the same as above, but not through frame rate, but through sampling rate.
Uint32_t audio_pts_increment = (90000 * audio_samples_per_frame)/audio_sample_rate;
The above pseudo code will not be pasted.

<9>. For the insertion time of Pat and PMT, in fact, Pat and PMT are to be inserted into the TS Stream consecutively, not after the insertion. Because ts must be able to start playing at any time. At present, I insert Pat and PMT packets every four frames of video (these four h264 frames may be packaged into many ts packages, but more than four packets.

<10>. The continuity_counter In the TS header of PES audio and video data packets must be continuous. from small to large, the loop starts from 0 when it increases to 15. The first package may not start from 0. However, the subsequent continuity_counter must be continuous and increase by step 1. The growth of continuity_counter is independent from the growth of all the packages with the same PID. For example, if the value of continuity_counter in the First Pat package is 0, the value of continuity_counter In the second Pat package is 1. The continuity_counter value of the first PMT package is 2, and the continuity_counter value of the second PMT package is 3.

[Audio AAC to ts considerations :]

AAC does not need to set DTS, As long as PTS is enough, and DTS may not be able to play back

When AAC is packaged into PES, you must set the length of pes_packet_length for playing on the iPad, while the video can be set to 0, but the audio must be set to the correct length value, the length of AAC cannot exceed 65535, so it is impossible to cause pes_packet_length overflow.
Otherwise, the iPad cannot play the video. However, QQ audio and video can be played. This may be a feature of the iPad.

// Data_len: Specifies the length of the AAC audio bare stream.
// 3: PES Header
// 5: for audio, there will be 5-byte PTS
_ Ts_packet_pes_header.pes_packet_length_16bit = data_len + 3 + 5;

[About CRC32 VerificationAlgorithm:]
Static uint32_t crc32table [1, 256] = {
0x00000000, 0x04c11db7, 0x09823b6e, 0x0d4326d9, 0x130476dc, 0x17c56b6b,
0x1a864db2, 0x1e475005, 0x2608edb8, 0x22c9f00f, 0x2f8ad6d6, 0x2b4bcb61,
0x350c9b64, 0x31cd86d3, 0x3c8ea00a, 0x384fbdbd, 0x4c11db70, 0x48d0c6c7,
0x4593e01e, 0x00002fda9, 0x5f15adac, 0x5bd4b01b, 0x0000796c2, 0x52568b75,
0x6a1936c8, 0x6ed82b7f, 0x639b0da6, 0x675a1011, 0x791d4014, 0x7ddc5da3,
0x709f7b7a, 0x745e66cd, 0x9823b6e0, 0x9ce2ab57, 0x91a18d8e, 0x95609039,
0x8b27c03c, 0x8fe6dd8b, 0x82a5fb52, 0x8664e6e5, 0xbe2b5b58, 0xbaea46ef,
0xb7a96036, 0xb3167d81, 0xad2f2d84, 0xa9ee3033, 0xa4ad16ea, 0xa06c0b5d,
0xd4326d90, 0xd0f37027, 0xddb056fe, 0xd9714b49, 0xc7361b4c, 0xc3f706fb,
0xceb42022, 0xca753d95, 0xf23a8028, 0xf6fb9d9f, 0xfbb8bb46, 0xff79a6f1,
0xe13ef6f4, 0xe5ffeb43, 0xe8bccd9a, 0xec7dd02d, 0x34867077, 0x30476dc0,
0x3d044b19, 0x39c556ae, 0x278206ab, 0x23431b1c, 0x2e003dc5, 0x2ac12072,
0x128e9dcf, 0x164f8078, 0x1b0ca6a1, 0x1fcdbb16, 0x018aeb13, 0x054bf6a4,
0x0808d07d, 0x0cc9cdca, 0x7897ab07, 0x7c56b6b0, 0x71159069, 0x75d48dde,
0x6b93dddb, 0x6f52c06c, 0x6211e6b5, 0x66d0fb02, 0x5e9f46bf, 0x5a5e5b08,
0x571d7dd1, 0x53dc6066, 0x4d9b3063, 0x495a2dd4, 0x44366b0d, 0x40d816ba,
0xaca5c697, 0xa864db20, 0xa527fdf9, 0xa1e6e04e, 0xbfa1b04b, 0xbb60adfc,
0xb6238b25, 0xb2e29692, 0x8aad2b2f, 0x8e6c00008, 0x832f1041, 0x87ee0df6,
0x99a95df3, 0x9d684044, 0x902b669d, 0x94ea7b2a, 0xe0b41de7, 0xe4750050,
0xe9362689, 0xedf73b3e, 0xf3b06b3b, 0xf771768c, 0xfa325055, 0xfef34de2,
0xc6bcf05f, 0xc27de8, 0xcf3ecb31, 0xcbffd686, 0xd5b88683, 0xd1799b34,
0xdc3abded, 0xd8fba05a, 0x690ce0ee, 0x6dcdfd59, 0x608edb80, 0x644fc637,
0x7a089632, 0x7ec98b85, 0x738aad5c, 0x774bb0eb, 0x4f040d56, 0x4bc510e1,
0x46863638, 0x000072b8f, 0x5c007b8a, 0x58c1663d, 0x5500000e4, 0x51435d53,
0x251d3b9e, 0x21dc2629, 0x2c9f00f0, 0x285e1d47, 0x36194d42, 0x32d850f5,
0x3f9b762c, 0x3b5a6b9b, 0x0315d626, 0x07d4cb91, 0x0a97ed48, 0x0e56f0ff,
0x1011a0fa, 0x14d0bd4d, 0x19939b94, 0x1d528623, 0xf12f560e, 0xf5ee4bb9,
0xf8ad6d60, 0xfc6c70d7, 0xe22b20d2, 0xe6ea3d65, 0xeba91bbc, 0xef68060b,
0xd727bbb6, 0xd3e6a601, 0xdea580d8, 0xda649d6f, 0xc423cd6a, 0xc0e2d0dd,
0xcda1f604, 0xc960ebb3, 0xbd3e8d7e, 0xb9ff90c9, 0xb4bcb610, 0xb07daba7,
0xae3afba2, 0xaafbe615, 0xa7b8c0cc, 0xa0000dd7b, 0x9b3660c6, 0x9ff77d71,
0x92b45ba8, 0x9675461f, 0x8832161a, 0x8cf30bad, 0x81b02d74, 0x857130c3,
0x5d8a9099, 0x594b8d2e, 0x5408abf7, 0x50c9b640, 0x4e8ee645, 0x4a4ffbf2,
0x470cdd2b, 0x43csc09c, 0x7b827d21, 0x7f436096, 0x7200464f, 0x76c15bf8,
0x68860bfd, 0x6c47164a, Zero X 61043093, 0x65c52d24, 0x119b4be9, 0x155a565e,
0x18197087, 0x1cd86d30, 0x029f3d35, 0x065e2082, 0x0b1d065b, 0x0fdc1bec,
0x00003a651, 0x3352bbe6, 0x3e119d3f, 0x3ad08088, 0x2497d08d, 0x2056cd3a,
0x2d15ebe3, 0x29d4f654, 0xc5a92679, 0xc1683bce, 0xcc2b1d17, 0xc8ea00a0,
0xd6ad50a5, 0xd26c4d12, 0xdf2f6bcb, 0xdbee767c, 0xe3a1cbc1, 0xe760d676,
0xea23f0af, 0xeee2ed18, 0xf0a5bd1d, 0xf464a0aa, 0xf9278673, 0xfde69bc4,
0x89b8fd09, 0x8d79e0be, 0x803ac667, 0x84fbdbd0, 0x9abc8bd5, 0x9e7d9662,
0x933eb0bb, 0x97ffad0c, 0xafb010b1, 0xab710d06, 0xa6322bdf, 0xa2f33668,
0xbcb4666d, 0xb8757133, 0xb5365d03, 0xb1f740b4
};

Uint32_t CRC32 (const uint8_t * data, int Len)
{
Int I;
Uint32_t CRC = 0 xffffffff;
For (I = 0; I <Len; I ++)
CRC = (CRC <8) ^ crc32table [(CRC> 24) ^ * Data ++) & 0xff];
Return CRC;
}

[About padding byte in TS stream]

The TS stream has two different filling forms, because each packet of TS requires 188 bytes. When there are less than 188 bytes, you must add 188 bytes, this involves filling.

<1>. if the TS package carries PSI data (Pat, PMT, etc.), it is filled with 0xff after the last valid byte of the package until 188 bytes are satisfied.
The decoder discards these bytes. For details, refer to iso_iec 13818-1.pdf 60/174.

<2>. if the TS package carries PES data, you must use the adaptation_field field when less than 188 bytes exist, that is, specify the adaptation_field_control value in the TS package to control the bearer relationship between payload and adaptation_field, and specify the adaptation_field_length in PES to specify the number of bytes to be filled, we recommend that you use ff to fill in the byte value at will, and the adaptation_field can only be used in PES. Iso_iec 13818-1.pdf 39/174