Decoding H264 video streams using FFmpeg on the iOS platform

Source:http://www.aichengxu.com/view/37145

In the iOS platform using ffmpeg decoding H264 video stream, there is a need for friends to refer to below.

For mainstream video transmission protocols such as video files and RTSP, FFmpeg provides a avformat_open_input interface that can be opened directly by passing in a file path or URL. Reading video data, decoder initial parameter settings, etc., can be done by invoking the API.

However, for h264 streams, there is no encapsulation format and you cannot use Libavformat. So many jobs need to be done by hand.

The H264 flow here refers to Annexb, which is the format in which each NAL unit starts with a starting code of 00 00 00 01 or 00 00 01. You can refer to this article for the H264 bitstream format.

The first is to manually set Avcodec and Avcodeccontext:

AVCodec *codec = avcodec_find_decoder(AV_CODEC_ID_H264);AVCodecContext *codecCtx = avcodec_alloc_context3(codec);avcodec_open2(codecCtx, codec, nil);

Many of the information needed for decoding is saved in Avcodeccontext, such as the length and width of the video, but we don't know it yet.

This information is stored in the SPS (set of sequence parameters) and PPS (set of image parameters) of the H264 stream.

For each NAL unit, the last 5 bits of the first byte after the starting code represent the type of the NAL unit. 7 represents sps,8 on behalf of PPS. Usually after the SPS and PPS is the IDR frame, without the information on the front frame can be decoded , represented by 5来.

Methods for detecting NAL unit types:

- (int)typeOfNalu:(NSData *)data{    char first = *(char *)[data bytes];    return first & 0x1f;}

The 264 decoder extracts the video information when it decodes SPS and PPS and saves it in the Avcodeccontext. However, it is not possible to pass SPS and PPS only, and it is necessary to pass the following IDR frames to the decoder to decode correctly.

Can write a simple detection, if you receive the SPS, the back of the PPS and IDR frames are received, and then passed to the decoder .

Initializes a avpacket and Avframe, and then decodes the data block that is connected to the SPS, PPS, and IDR frames to avpacket.

We assume that the data block containing SPS, PPS, IDR frames is stored in videodata with len length.

char *videoData;int len;AVFrame *frame = av_frame_alloc();AVPacket packet;av_new_packet(&packet, len);memcpy(packet.data, videoData, len);int ret, got_picture;ret = avcodec_decode_video2(codecCtx, frame, &got_picture, &packet);if (ret > 0){    if(got_picture){    //进行下一步的处理    }}

In this way, the H264 stream can be decoded successfully, and the decoded data is saved in the Avframe.

I wrote a objective-c class to perform a series of steps to receive the video stream, decode it, and play it.

The receiving of video data is directly received by the socket, using the Open source project Cocoaasyncsocket.

As indicated in the project name, this is an asynchronous socket class. The operation of the read-write socket is performed in a separate dispatch queue, and the corresponding delegate method is automatically called after execution, where it is further processed.

The read H264 stream uses the Gcdasyncsocket
- (void)readDataToData:(NSData *)data withTimeout:(NSTimeInterval)timeout tag:(long)tag
method, which is to stop reading and call the delegate method when it reads the same content as the bytes in data. The data parameter passed in is 00 00 13 bytes. So each read-in Nalu starts with no start code, and at the very end there is the next Nalu start code. Therefore, after each reading, the end of the start code is staged, and then the main body is connected to the last staged start code, which constitutes the complete nalu.

VideoPlayer.h:

//videoPlayer.h#import <Foundation/Foundation.h>@interface videoPlayer : NSObject- (void)startup;- (void)shutdown;@end

Video PLAYER.M:

//VideoPlayer.m#import "VideoPlayer.h" #import "GCDAsyncSocket.h" #import "libavcodec/avcodec.h" #import "libswscale/ swscale.h "Const int Header = 101;const int Data = 102; @interface videoplayer () <gcdasyncsocketdelegate>{Gcdasyn CSocket *Socket;    NSData *startcodedata; NSData * LastStartcode;    FFmpeg avframe *frame;    Avpicture picture;    Avcodec *codec;    Avcodeccontext *codecctx;    Avpacket packet;    struct Swscontext *img_convert_ctx;    Nsmutabledata *keyframe;    int outputwidth; int outputheight;} @end @implementationVideoplayer-(id) init{self = [super init];        if (self) {avcodec_register_all ();        frame = Av_frame_alloc ();        codec = Avcodec_find_decoder (av_codec_id_h264);        Codecctx = AVCODEC_ALLOC_CONTEXT3 (codec);        int ret = AVCODEC_OPEN2 (Codecctx, codec, nil);        if (ret! = 0) {NSLog (@ "Open codec failed:%d", ret); }Socket= [[Gcdasyncsocket alloc]initwithdelegate:self Delegatequeue:dispatch_get_main_queue ()]; Keyframe = [[Nsmutabledata alloc]init]; Outputwidth = 320; Outputheight = 240; unsigned char startcode[] = {0,0,1}; Startcodedata = [NSDataDataWithbytes:startcode Length:3]; } return self;} -(void) startup{nserror *error = nil; [Socket connecttohost:@ "192.168.1.100" onport:9982 withtimeout:-1 error:&error]; NSLog (@ "%@", error); if (!error) {[SocketReaddatatodata:startcodedata withtimeout:-1 tag:0]; }}-(void) Socket: (Gcdasyncsocket *) sock didreaddata: (NSData *)DataWithtag: (Long) tag{[socket readdatatodata:startcodedata withtimeout:-1 tag:data]; if (tag = = Data) {int type = [self typeofnalu:data]; I F (Type = = 7 | | type = = 8 | | type = = 6 | | type = = 5) {//sps PPS SEI IDR [keyframe appendData: LastStartcode]; [keyframe appendbytes:[data bytes] length:[data length]-[self startcodelenth:data]]; if (type = = 5 | | type = = 1) {//idr P frame if (type = = 5) {int nallen = (int) [keyframe length]; Av_new_packet (&pack ET, Nallen); memcpy (packet.Data, [keyframe bytes], nallen); Keyframe = [[Nsmutabledata alloc] Init];//reset keyframe}else{nsmutabledata *nalu = [[Nsmutabledata alloc]initWithData: LastStartcode]; [Nalu appendbytes:[data bytes] length:[data length]-[self startcodelenth:data]]; int nallen = (int) [Nalu length]; Av_new_packet (&packet, Nallen); memcpy (packet.Data, [Nalu bytes], nallen); } int ret, got_picture; NSLog (@ "decode start"); RET = Avcodec_decode_Video2 (Codecctx, Frame, &got_picture, &packet); NSLog (@ "decodeFinish"); if (Ret < 0) {NSLog (@ "decode error"); return;} if (!got_picture) {NSLog (@ "didn ' t get Picture"); return;} static int Sws_flags = Sws_fast_bilinear; Outputwidth = codecctx->width; Outputheight = codecctx->height; if (!img_convert_ctx) Img_convert_ctx = Sws_getcontext (Codecctx->width, Codecctx->height, codecCtx->pix_fmt , Outputwidth, Outputheight, pix_fmt_yuv420p, sws_flags, NULL, NULL, and NULL); Avpicture_alloc (&picture, pix_fmt_yuv420p, Outputwidth, outputheight); ret = Sws_scale (Img_convert_ctx, (const uint8_t* const*) frame->Data, Frame->linesize, 0, Frame->height, Picture.data, picture.linesize); [Self display]; NSLog (@ "Show frameFinish"); Avpicture_free (&picture); Av_free_packet (&packet); }} [self Savestartcode:data];} -(void) display{}-(int) Typeofnalu: (NSData *) data{char first = * (char *) [data bytes]; return first & 0x1f;} -(int) Startcodelenth: (NSData *)Data{Char temp = * ((char *) [data bytes] + [data length]-4); return temp = = 0x00? 4:3;} -(void) Savestartcode: (NSData *) data{int startcodelen = [self startcodelenth:data]; Nsrange Startcoderange = {[Data length]-Startcodelen, Startcodelen};Last  startcode = [data Subdatawithrange:startcoderange];} -(void) shutdown{if (socket) [socket disconnect];} -(void) dealloc{//Free Scaler if (img_convert_ctx) Sws_freecontext (IMG_CONVERT_CTX),//Free the YUV frame if (frame) av_fr Ame_free (&frame); Close the codec if (CODECCTX) avcodec_close (CODECCTX);} @end 

Playing the decoded YUV Video in the project uses OpenGL, and the part that is played here is omitted.

Decoding H264 video streams using FFmpeg on the iOS platform