Use librtmp for H264 and AAC live

libx264 version is
LIBFAAC version is 1.28

1, the division of the frame1.1 H-Frames

the definition of each frame is for H. xx or xx.

For example, the following H264 file fragment this contains three frames of data:

E0 9F 96 10 00 (C0)
In the F1 of the 2A (XX)
3C (CE) 5D DC E9 BD
E6 D9 B7 2C D8 D9 EE EF ...

The first frame is the xx-C0-DA-E0-9F, which was F1 2A.
the second frame is xx 3C 80
The third frame is the FF FF 5D DC E9 BD E6 D9 B7 2C D8 D9 EE EF.

frame type:
nal_slice = 1
NAL_SLICE_DPA = 2
NAL_SLICE_DPB = 3
NAL_SLICE_DPC = 4
NAL_SLICE_IDR = 5
Nal_sei = 6
Nal_sps = 7
Nal_pps = 8
Nal_aud = 9
Nal_filler =,

We only need to know four frame types when we send RTMP data, other types I've made it into non-keyframes.
nal_sps (7), SPS frame
Nal_pps (8), PPS frame
Span style= "FONT-SIZE:16PX;" > NAL_SLICE_IDR (5), Keyframe
Nal_slice (1 ) non-keyframe

The way the frame type is judged to be the lower four bits of the first byte after the interface character.
the frame type for the first frame is: 0x67 & 0x1F = 7, which is an SPS frame
the frame type for the second frame is: 0x68 & 0x1F = 8, which is a PPS frame
the frame type for the third frame is: 0x06 & 0x1F = 6, which is an SEI frame

The above is the use of frame qualifiers to divide frames, and to determine the type of each frame.

Note: If the image is compressed into a H264 frame, we do not have to frame, because each compressed output is clear the size of the frame (including the qualifier), each time the result of the compression may be multi-frame function. A specific discussion will be given.

1.2 AAC Frames

for AAC frames it is defined as FF F1

Here I don't have an example, you can see the following frame structure by looking at the AAC binary file:
FF F1 9F FD DE (6C)-----------

Note: For AAC, the first 7 bytes of each frame are the description of the frame, that is, the AAC's naked data is removed from the previous 7 bytes, and we want to remove the 7 bytes when we send RTMP. Similarly, if we are sending RTMP on one side of the compression, we also do not need to define frames, because the output of LIBFAAC each time the compression is complete is a complete frame data, we only need to package the frame to send.

As described above, if we just compress one side to send the compression results to the RTMP server, then we can not define the frame, if we are sending H264 and AAC files, then we need to define the frame.

2. Encoding information for video and audio

if we simply send the compressed data package to the RTMP server, then the RTMP server is not able to decode and play the data, before we send the encoded information of the audio and video to the RTMP server. Many people may be struggling to find the following three coding parameters. In fact, it is very simple to want to get.

2.1 (H264) SPS

for H264, the SPS is the first frame after encoding. If you are reading the H264 file, the length of the data between the first frame qualifier and the second frame qualifier is 4.

2.2 (H264) PPS

for H264, PPS is the second frame after encoding. If you are reading the H264 file, it is the data between the second frame qualifier and the third frame qualifier, and the length is not fixed.

2.3 (AAC) Audiodecoderspecificinfo

This is a length of 2 bytes, which can be obtained by calculating or calling functions. recommended by calling Faacencgetdecoderspecificinfo (Fh,&spec,&len);
in general, dual channel 44100 sampling, this value is 0x1210

Use of 3.librtmp

/* Assign and initialize */rtmp = Rtmp_alloc (); Rtmp_init (rtmp);/* Set url*/if (Rtmp_setupurl (rtmp,rtmp_url) = = FALSE) {    log (Log_err, "Rtmp_setupurl () failed!");    Rtmp_free (rtmp);    return-1;} /* Set writable, that is, publish stream, this function must be used before connection, otherwise invalid */rtmp_enablewrite (rtmp),/* Connection Server */if (Rtmp_connect (rtmp, NULL) = = FALSE) {    log (log _err, "Rtmp_connect () failed!");    Rtmp_free (rtmp);    return-1;} /* Connect stream */if (Rtmp_connectstream (rtmp,0) = = FALSE) {    log (Log_err, "Rtmp_connectstream () failed!");    Rtmp_close (rtmp);    Rtmp_free (rtmp);    return-1;}

/* Defines the length of the header, Rtmp_max_header_size defines the value as 18*/in Rtmp.h #define Rtmp_head_size   (sizeof (Rtmppacket) +rtmp_max_header_ SIZE) rtmppacket * packet;unsigned char * body;/* allocates packet memory and initialization, Len is packet length */packet = (Rtmppacket *) malloc (Rtmp_head_size+len); memset (packet,0,rtmp_head_size);/* Package Body */packet->m_body = (char *) packet + Rtmp_head_size;body = (unsigned char *) Packet->m_body;packet->m_nbodysize = len;/* * This omits the package body fill */packet->m_hasabstimestamp = 0;packet->m_ Packettype = Rtmp_packet_type_video; /* Here are two types of audio, one is video */packet->m_ninfofield2 = Rtmp->m_stream_id;packet->m_nchannel = 0x04;packet->m_ Headertype = Rtmp_packet_size_large;packet->m_ntimestamp = timeoffset;/* Send */if (rtmp_isconnected (RTMP)) {    ret = Rtmp_sendpacket (rtmp,packet,true); /*true is placed in the send queue, false is not put into the send queue, directly send */}/* free memory */free (packet);

/* Close and Release */rtmp_close (RTMP); Rtmp_free (RTMP);

4. Package Type 4.1 H. Encoded information frame

the encoded information frame of H. E is sent to the RTMP server called AVC sequence HEADER,RTMP the server only receives the SPS in the AVC sequence header, PPS can resolve subsequent sent H264 frames.

int Send_video_sps_pps () {rtmppacket * packet;    unsigned char * body;    int i;    Packet = (Rtmppacket *) malloc (rtmp_head_size+1024);    memset (packet,0,rtmp_head_size);    Packet->m_body = (char *) packet + rtmp_head_size;    BODY = (unsigned char *) packet->m_body;    memcpy (Winsys->pps,buf,len);    Winsys->pps_len = Len;    i = 0;    body[i++] = 0x17;    body[i++] = 0x00;    body[i++] = 0x00;    body[i++] = 0x00;    body[i++] = 0x00;    /*avcdecoderconfigurationrecord*/body[i++] = 0x01;    body[i++] = sps[1];    body[i++] = sps[2];    body[i++] = sps[3];    body[i++] = 0xFF;    /*sps*/body[i++] = 0xe1;    body[i++] = (Sps_len >> 8) & 0xFF;    body[i++] = Sps_len & 0xFF;    memcpy (&body[i],sps,sps_len);    i + = Sps_len;    /*pps*/body[i++] = 0x01;    body[i++] = (Pps_len >> 8) & 0xFF;    body[i++] = (pps_len) & 0xFF;    memcpy (&body[i],pps,pps_len);    i + = Pps_len; Packet->m_packettype = Rtmp_packet_type_vidEO;    Packet->m_nbodysize = i;    Packet->m_nchannel = 0x04;    Packet->m_ntimestamp = 0;    Packet->m_hasabstimestamp = 0;    Packet->m_headertype = Rtmp_packet_size_medium;    Packet->m_ninfofield2 = rtmp->m_stream_id;    /* Call the Send interface */Rtmp_sendpacket (rtmp,packet,true);        Free (packet); return 0;}

How to obtain SPS and PPS?

As already mentioned, the 1th frame of the H264 is the SPS frame, and PPS is the 2nd frame.

we will call the following interface when we encode

Size = X264_encoder_encode (cx->hd,&nal,&n,pic,&pout); int i,last;for (i = 0,last = 0;i < n;i++) {    if (Nal[i].i_type = = Nal_sps) {        Sps_len = nal[i].i_payload-4;        memcpy (Sps,nal[i].p_payload+4,sps_len);    } else if (Nal[i].i_type = = Nal_pps) {        Pps_len = nal[i].i_payload-4;        memcpy (Pps,nal[i].p_payload+4,pps_len);        /* Send SPS pps*/        Send_video_sps_pps ();        } else {/        * send normal frame *        /Send_rtmp_video (nal[i].p_payload,nal[i].i _payload);    }    Last + = Nal[i].i_payload;}

I don't have to know the exact meaning of the SPS, pps:)

4.2 H. A key frame

4.3 h, non-key frame

int send_rtmp_video (unsigned char * buf,int len) {int type;    Long Timeoffset;    Rtmppacket * packet;    unsigned char * body;  Timeoffset = GetTickCount ()-start_time;            /*start_time time stamp for start of live//* Remove Frame Qualifier */if (buf[2] = = 0x00) {/*00 xx 01*/buf + = 4;    Len-= 4;            } else if (buf[2] = = 0x01) {/*00 01*/buf + = 3;    Len-= 3;    } type = buf[0]&0x1f;    Packet = (Rtmppacket *) Base_malloc (rtmp_head_size+len+9);    memset (packet,0,rtmp_head_size);    Packet->m_body = (char *) packet + rtmp_head_size;    Packet->m_nbodysize = len + 9;    /*send video packet*/BODY = (unsigned char *) packet->m_body;    memset (body,0,len+9);    /*key frame*/body[0] = 0x27;    if (type = = Nal_slice_idr) {body[0] = 0x17;   } body[1] = 0x01;    /*nal unit*/body[2] = 0x00;    BODY[3] = 0x00;    BODY[4] = 0x00;    BODY[5] = (len >>) & 0xFF;    BODY[6] = (len >>) & 0xFF; BODY[7] = (Len ≫> 8) & 0xFF;    BODY[8] = (len) & 0xFF;    /*copy data*/memcpy (&body[9],buf,len);    Packet->m_hasabstimestamp = 0;    Packet->m_packettype = Rtmp_packet_type_video;    Packet->m_ninfofield2 = winsys->rtmp->m_stream_id;    Packet->m_nchannel = 0x04;    Packet->m_headertype = Rtmp_packet_size_large;    Packet->m_ntimestamp = Timeoffset;    /* Call the Send interface */Rtmp_sendpacket (rtmp,packet,true); Free (packet);}

here to illustrate:
For example, X264_encoder_encode output 6 frames.
SPS frames, pps frames, keyframes, non-keyframes, non-keyframes, non-keyframes, respectively.
send the result should be, Sps,pps together become a frame call send function, Left 4 frames, remove each nal of the delimiter, send each nal respectively.

in libx264 each call X264_encoder_encode output n frames, we want to find the SPS and PPS from these n frames, the remaining sub-all send Nal,sps and PPS frame qualifier is 00 00 00 01, and the normal frame It could be 00 00 00 01 or maybe 00 00 01.

If there is no SPS frame and PPS frame in the X264_encoder_encode, then the result is removed from the first frame, so the frame is called as a whole to send the function, and their type is determined by the first frame type.

In addition, the 1th frame of the H264 stream must be an SPS frame (containing a frame qualifier of 8 bytes), and the 2nd frame must be a PPS frame.

4.4 AAC encoded information

int cap_rtmp_sendaac_spec (unsigned char *spec_buf,int    Spec_len) {Rtmppacket * packet;    unsigned char * body;    int Len;  len = Spec_len;    /*spec data length, generally 2*/packet = (Rtmppacket *) Base_malloc (rtmp_head_size+len+2);    memset (packet,0,rtmp_head_size);    Packet->m_body = (char *) packet + rtmp_head_size;    BODY = (unsigned char *) packet->m_body;    /*af + AAC RAW data*/body[0] = 0xAF;    BODY[1] = 0x00; memcpy (&body[2],spec_buf,len);    /*SPEC_BUF is the AAC sequence header data */Packet->m_packettype = Rtmp_packet_type_audio;    Packet->m_nbodysize = len+2;    Packet->m_nchannel = 0x04;    Packet->m_ntimestamp = 0;    Packet->m_hasabstimestamp = 0;    Packet->m_headertype = Rtmp_packet_size_large;    Packet->m_ninfofield2 = rtmp->m_stream_id;    /* Call the Send interface */Rtmp_sendpacket (rtmp,packet,true); return TRUE;} 

for audio decoding parameters the AAC sequence header is passed

here's how to get the AAC sequence header

Char *buf;int len;faacencgetdecoderspecificinfo (Fh,&buf,&len); memcpy (spec_buf,buf,len); Spec_len = len;/* Release system memory */free (BUF);

In addition if you are open AAC file to send, then you can try to calculate this value yourself, in fact, it is very simple, open FAAC source code to see the implementation of Faacencgetdecoderspecificinfo, that is, a few shift things:).

for the general situation 44100Hz dual channel, this value is 0x1210, lazy is directly use this value bar.

4.5 AAC General Data

as mentioned earlier, the normal data for sending AAC should be modified because the first 7 bytes of AAC (including the frame qualifier) are useless for RTMP servers.

void * Cap_dialog_send_audio (unsigned char * buf,int len) {long timeoffset;    Timeoffset = GetTickCount ()-start_time;    BUF + = 7;    Len + 7;        if (Len > 0) {rtmppacket * packet;        unsigned char * body;        Packet = (Rtmppacket *) malloc (rtmp_head_size+len+2);        memset (packet,0,rtmp_head_size);        Packet->m_body = (char *) packet + rtmp_head_size;        BODY = (unsigned char *) packet->m_body;        /*af + AAC RAW data*/body[0] = 0xAF;        BODY[1] = 0x01;        memcpy (&body[2],buf,len);        Packet->m_packettype = Rtmp_packet_type_audio;        Packet->m_nbodysize = len+2;        Packet->m_nchannel = 0x04;        Packet->m_ntimestamp = Timeoffset;        Packet->m_hasabstimestamp = 0;        Packet->m_headertype = Rtmp_packet_size_medium;        Packet->m_ninfofield2 = rtmp->m_stream_id;        /* Call the Send interface */Rtmp_sendpacket (rtmp,packet,true);    Free (packet); } return 0;}

at this point all processes are finished.

a few things to note:
librtmp Multi-threaded sending can sometimes cause problems, do not know what the problem, it is best to change the queue to send. The filled packet is sent to other threads by message or other means, and the sending thread is unified.

5. Reference documentation

• "Video File Format specification Version 10" Http://www.adobe.com/content/dam/Adobe/en/devnet/flv/pdfs/video_ File_format_spec_v10.pdf
• "H264 video via rtmp live" http://blog.csdn.net/firehood_/article/details/8783589
• rtmpdump-2.3

Use librtmp for H264 and AAC live