Analysis of H.264 decoding process -4__h.264

x264 Open Source project to achieve H.264 video coding, but does not provide the corresponding decoder. FFmpeg Open Source Multimedia Codec collection brings together nearly all the media formats in the market codec source code. One of the h264.c is a normal decoding x264 code stream of the independent source files, the use of the procedures and the above coding or decoding codec application cases are basically the same. This section describes how H264.C implements the H.264 video decoding process in a top-down manner.

The h264.c source file has thousands of lines, a large amount of code, and is not easy to browse, analyze, and migrate. At the same time, the file also relies on other source files, the organization is more complex, implementation platform because it is not based on Windows VC + +, to compile, tracking and so bring a lot of inconvenience. The "chapter7\ffmpeg_h264" directory in the CD-ROM provides a project case "ffmpeg_h264" for H.264 video decoding from the FFmpeg extracted C language. "ffmpeg_h264" file features a clear classification, in the vc++2005 platform to compile, and eliminate the MMX support, pure C language engineering easy to transplant embedded platform. Starting from main function, the following step is to deeply analyze the process of realizing H.264 video decoding.
FFMPEG_H264 Engineering 1 decoding main program

Function main realizes the console application of H.264 video decoding, including all the process of decoding work. Its function call relationship is shown in the figure.

In the above applications, the topological structure flow is basically the same as the MPEG-1 video decoding process, including Avcodec_init, Avcodec_register_all, Avcodec_find_decoder, Avcodec_all_context , Avcodec_open, Avcodec_all_frame, Avcodec_decode_video, Avcodec_close and other ffmpeg of the major public modules, which Avcodec_decode_video is the core of the decoder , the annotation is made in the diagram. The code for the main function is implemented as follows:

int main () {FILE * inp_file;

    FILE * OUT_FILE;
    int i;         int Nallen; /*nal length * * unsigned char* Buf;       /*h.264 Code Flow * * int got_picture; /* Whether to decode a frame image/int consumed_bytes;

    /* Decoder consumption of the length of the code stream * * Cnt=0 int;           Avcodec *codec;        /* codec codec*/avcodeccontext *c;         /* Codec codec context*/avframe *picture;
    /* decoded image/* Output and Output file * * * Inp_file = fopen ("e:\\bitavc\\busn_dsp.264", "RB");

    Out_file = fopen ("E:\\BITAVC\\DSP_DEC.YUV", "WB");
    Nallen = 0;

    /* Allocates memory and initializes to 0*/Buf = (unsigned char*) calloc (500*1024, sizeof (char)); 

    Initialization of/*codec, initialization of some constant table/avcodec_init (); 

    /* Registered codec*/Avcodec_register_all ();

    /* Find H264 codec*/CODEC = Avcodec_find_decoder (codec_id_h264); 

    if (!CODEC) return 0; 

    /* Initialize codec default parameter */c = Avcodec_alloc_context ();

    if (!c) return 0; /*1.  

    Open codec, where the H.264 decoder is initialized, and the Decode_init local function is called/if (Avcodec_open (c, codec) < 0) return 0; /* to AVFRAME application space, and clear 0 */Picture = Avcodec_alloc_frame ();

    if (!picture) return 0;

        /* Loop decoding/while (!feof (inp_file)) {/* get a NAL packet from the stream/Nallen = getnextnal (Inp_file, Buf); /*2.

        NAL decoding, call decode_frame local function * * consumed_bytes= avcodec_decode_video (c, Picture, &got_picture, Buf, Nallen);
        cnt++;

        /* Output current decoding information/printf ("no:=%4d, length=%4d\n", cnt,consumed_bytes);
            /* Return <0 to decode the data header, return >0, to decode a frame image/if (Consumed_bytes > 0) {/* To extract decoded image from two-dimensional space * * For (i=0 i<c->height; i++) fwrite (picture->data[0] + i * picture->linesize[0], 1, C->widt
            h, Out_file); For (i=0 i<c->height/2; i++) fwrite (picture->data[1] + i * picture->linesize[1], 1, C->widt
            H/2, Out_file); For (i=0 i<c->height/2; i++) fwrite (picture->data[2] + i * picture->linesize[2], 1, C->widt
     H/2, Out_file);   }/* Close File/if (inp_file) fclose (inp_file);

    if (out_file) fclose (out_file); /*3. 
        Turn off codec, release resources, call decode_end local function/if (c) {avcodec_close (c);
        Av_free (c);
    c = NULL;
        /* Release Avframe space */if (picture) {av_free);
    Picture = NULL;
        /* Release memory/if (BUF) {free (BUF);
    Buf = NULL;
return 0;
 }

In the process described above, the steps 1~8 the basic steps for FFmpeg to work when coding or decoding. The feature of the analysis function name will find that the codec processing function is prefixed with "avcodec_". In order to use the H.264 decoder codec, you first need to define codec as follows:

Avcodec H264_decoder = {
    "H264",//             decoder name
    codec_type_video,//   decoder type, video
    codec_id_h264,      // Decoder ID
    sizeof (h264context),//        decoder context size
    decode_init,//            decoder initialization
    NULL,                   //encoder, disabled
    Decode_end,         //Destroy decoder
    decode_frame,       //decoding
    0,                      //codec compatibility, disable
};

Therefore, the actual decoding processing module adopts Decode_init, Decode_frame, decode_end three local functions to achieve. and the main function uses the public function interface to access the functions here. Real for example registers codec.

/** * Simple call to register all the
 codecs. 
 *
/void Avcodec_register_all (void)
{
    static int inited = 0;

    if (inited!= 0) return
    ;
    inited = 1;

    Register_avcodec (&h264_decoder);
    Av_register_codec_parser (&h264_parser);
}

Therefore, the codec decoder actually calls the three child functions in h264.c to complete the decoding task. 2 Configuration Decoder

The above is the registration and definition decoder, where the FFmpeg avcodec_open module opens the codec and actually invokes the Decode_init local function to create and configure the H.264 decoder.

int Avcodec_open (Avcodeccontext *avctx, Avcodec *codec)
{
    int ret;
    if (AVCTX->CODEC)  return-1;    /*codec pointer effective *

    /Avctx->codec = codec;       /* Point codec*/
    avctx->codec_id = codec->id;    /*codec name * *
    avctx->frame_number = 0;
#if 1
    if (codec->priv_data_size > 0) {/*CODEC structure is not 0*//
        * For codec application space/
        Avctx->priv_data = av_ Mallocz (codec->priv_data_size);
        if (!avctx->priv_data) 
            return-enomem;
    } else {
        avctx->priv_data = NULL;
    }

    initialization of the #endif/*codec
    /ret = Avctx->codec->init (AVCTX);
    * If failed, release the context structure body/
    if (Ret < 0) {
        av_freep (&avctx->priv_data);
        return ret;
    }

    return 0;
}

The initialization of the above implementation decoder includes setting the default decoding parameters, image width and height, initializing the I-frame predictive function, and then the planar pattern of pixel format I420, and finally the initialization of VLC table for variable length encoding. 3 H.264 video decoding

The video decoding module is completed by FFmpeg's Avcodec_decode_video (). Since the H.264 stream is nal packaged, it is necessary to invoke the Getnextnal () module to read from the stream file the parsing of a NAL packet, that is, to find the NAL header ID 0x00 00 00 01, then pass the data between the two head IDs to the VCL decoder Avcodec_decode_video () to decode:

/** * Decode a frame. * @param buf bitstream buffer, must be ff_input_buffer_padding_size larger then the actual read bytes * because some Opti  mized bitstream readers read or the bit at once and could read over the "End * @param buf_size the size of the" buffer in  Bytes * @param got_picture_ptr Zero if no frame could be decompressed, otherwise, it is non zero * @return-1 if error, 
 Otherwise return the number of * bytes used.
                         */int Avcodec_decode_video (Avcodeccontext *avctx, avframe *picture, int *got_picture_ptr,

    uint8_t *buf, int buf_size) {int ret;

    /* Currently no decoded logo */*got_picture_ptr= 0;                /*h.264 decoding/ret = Avctx->codec->decode (AVCTX,//decoder context picture,                Decoded image got_picture_ptr,//Image decoding logo buf,          Code stream buf_size to be decoded); Code Stream Length/* If you use theMMX and other instructions, you need to call the function */Emms_c ();

    /* Decoding is the image, not the header data */if (*GOT_PICTURE_PTR) avctx->frame_number++;
return ret;
 }

The above VCL decoder ffmpeg the common function interface decode according to the incoming stream and length, namely the actual decoding function Decode_frame realize H.264 decoding. If the MMX instruction is used in the decoding process and the float/double type is used after exiting, the MMX state will need to be emptied using the EMMS instruction (Empty MMX) to restore the previous CPU state. The actual decoding function Decode_frame realize the real decoding work of VCL, the topology of the module is shown in the figure.

The Decode_frame decoding video coding layer VCL data, the core of the module is decode_nal_units.

static int Decode_frame (Avcodeccontext *avctx,/* Decoder context/void *data,  /* Decoding the structure of the image/int *data_size,/* Structure size * * uint8_t *BUF,/* stream space/int buf_size)/* Code flow length/{H264co          ntext *h = avctx->priv_data;              /* Decoder Context * * Mpegenccontext *s = &h->s;                   /*mpegenccontext pointer */avframe *pict = data;                          /* Image space */int buf_index;                 /* Current Stream position * * s->flags= avctx->flags;                   /*CODEC logo * * * s->flags2= avctx->flags2;
    /*codec's logo 2*/if (buf_size = 0) {/* stream is not empty/return 0;  /* Truncate stream decoding/if (s->flags&codec_flag_truncated) {int next= find_frame_end (&s->parse_context,
        BUF, buf_size); if (Ff_combine_frame (&s->parse_contExt, Next, &buf, &buf_size) < 0) return buf_size; /* Decode special data in the code stream/if (s->avctx->extradata_size && s->picture_number==0) {if (0 < Decode_n
    Al_units (H, S->avctx->extradata, s->avctx->extradata_size)) return-1;
    /* Normal decoding/* Buf_index=decode_nal_units (H, buf, buf_size);

if (Buf_index < 0) return-1;
    #if 0/*b Frame */if (S->pict_type==b_type | s->low_delay) {*pict= * (avframe*) &s->current_picture;
    else {*pict= * (avframe*) &s->last_picture;  #endif/* No decoded output image, Solution dock data */if (!s->current_picture_ptr) {Av_log (H->s.avctx, Av_log_debug, "error, no
        Frame\n ");
    return-1; 
    /* have decoded image output/*pict= * (avframe*) &s->current_picture;
    ASSERT (Pict->data[0]);
    /* Indicates the decoding output/*data_size = sizeof (avframe);
/* Returns the current consumption of the stream/return Get_consumed_bytes (S, Buf_index, buf_size);
 }

In the decoding process, firstly, if the code stream is truncated, then the decoded code stream is combined with a frame image, and the code flow is determined whether there are special data such as Huffman table, then the normal decoding nal, the final output decoding image and the length of the code stream consumed. The core of which is nal decoding decode_nal_units. 4 nal Packet decoding

The basic unit in H.264 code flow is represented by individual nal packets, so the decoded processing function
Decode_nal_units () implements the decoding NAL package. The topology of the module is shown in the diagram:

The NAL decoding in the above diagram mainly includes the sequence parameter set SPS, the image parameter set PPS, the instant decoding refresh slice IDR, the image title Slice_header, the image slice slice and so on the decoding decode_nal_units realization process is as follows:

/*nal unit decoding/static int decode_nal_units (H264context *h,//decoder handle uint8_t *buf,//code flow NULL
    between int buf_size) {//code stream length Mpegenccontext * Const S = &h->s;
    Avcodeccontext * Const avctx= s->avctx;

    int buf_index=0; /* Loop decoding/for (;;)   {int consumed; The consumed length int dst_length; Target length int bit_length;   Bit length uint8_t *ptr; Temporary pointer/* Search prefix start code: 0x 01*/for (; Buf_index + 3 < buf_size; buf_index++) {/* The first search must succeed, otherwise
                Think code outflow Error * * (buf[buf_index] = = 0 && buf[buf_index+1] = = 0 && buf[buf_index+2] = = 1)
        Break
        }/* Follow-up still has data */if (buf_index+3 >= buf_size) break;
        /* Index after moving/buf_index+=3;
        /* Network Abstraction Layer NAL solution Package/ptr= decode_nal (H, buf + Buf_index, &dst_length, &consumed, Buf_size-buf_index);
        if (ptr[dst_length-1] = = 0) dst_length--; /* Determine the exact end position of the code stream */bit_length= 8*dst_length-decode_rbsp_trailing (ptr + dst_length-1); if (S->avctx->debug&ff_debug_startcode) {Av_log (H->s.avctx, Av_log_debug, "NAL%d at%d length%d\
        N ", H->nal_unit_type, Buf_index, dst_length);
        /* Index after move/buf_index + = consumed;

        if (s->hurry_up = = 1 && h->nal_ref_idc = 0) continue;             Switch (h->nal_unit_type) {*/* decodes NAL type/Case nal_idr_slice://idr slice IDR (h); Case Nal_slice://Image Slice SLICE decode/* Initialization stream pointer/init_get_bits (&AMP;S-&GT;GB, PTR, Bit_leng
            TH);
            h->intra_gb_ptr= h->inter_gb_ptr= &s->gb;
            s->data_partitioning = 0;
            /* Decoded image slice head/if (Decode_slice_header (h) < 0) return-1; if (h->redundant_pic_count==0 && s->hurry_up < 5)/***************** image piece data decoding/Dec Ode_slIce (h);
        /***************** image chip data decoding/break;
            Case NAL_DPA://Data partition a init_get_bits (&AMP;S-&GT;GB, PTR, bit_length);
            H->intra_gb_ptr= h->inter_gb_ptr= NULL;
            s->data_partitioning = 1;
            /* Decoded image slice head/if (Decode_slice_header (h) < 0) return-1;
        Break
            Case NAL_DPB://Data partition B init_get_bits (&AMP;H-&GT;INTRA_GB, PTR, bit_length);
            H->intra_gb_ptr= &h->intra_gb;
        Break
            Case NAL_DPC://Data partition C init_get_bits (&AMP;H-&GT;INTER_GB, PTR, bit_length);
            H->inter_gb_ptr= &h->inter_gb; if (h->redundant_pic_count==0 && h->intra_gb_ptr && s->data_partitioning && s->
            Hurry_up < 5) Decode_slice (h);
        Break
        Case Nal_sei://Supplemental enhanced information break; Case NAL_SPS://sequence parameter set SPS/* Initialization Code Flow * * Init_get_bits (&AMP;S-&GT;GB, PTR, bit_length);
            /*sps Decoding * * Decode_seq_parameter_set (h);
            if (s->flags& codec_flag_low_delay) s->low_delay=1;
            avctx->has_b_frames=!s->low_delay;
            printf ("Decode sps\n");
        Break
            Case NAL_PPS://Image parameter set PPS/* Initialization code Flow * * Init_get_bits (&AMP;S-&GT;GB, PTR, bit_length);
            /*pps Decoding * * Decode_picture_parameter_set (h);
            printf ("Decode pps\n");
        Break
        Case Nal_picture_delimiter:break;
        Case Nal_filter_data:break;
        Default:av_log (Avctx, Av_log_error, "Unknown NAL Code:%d\n", H->nal_unit_type);
        }/* Image frame Type/* s->current_picture.pict_type= s->pict_type;
    /* Key frame type initialization/s->current_picture.key_frame= S->pict_type = = I_type; /* No decoded output image/if (!s->curRENT_PICTURE_PTR) return buf_index;
    No frame/* Modify image sequence poc*/h->prev_frame_num_offset= h->frame_num_offset;
    h->prev_frame_num= h->frame_num;
        if (s->current_picture_ptr->reference) {h->prev_poc_msb= h->poc_msb;
    h->prev_poc_lsb= h->poc_lsb; }/* Tag reference frame */if (s->current_picture_ptr->reference) execute_ref_pic_marking (H, H->mmco, H->mmco_
    index);
    else assert (h->mmco_index==0);
    /* Code flow Fault-tolerant end */ff_er_frame_end (s);
    /* Decoding one frame finished, extended image * * Mpv_frame_end (s);
return buf_index;
 }

The above process first resolves to get nal prefix code 0x 00 00 01, then function decode_nal () implementation nal unpack, according to the NAL type of resolution to perform different decoding: SPS, PPS, IDR, Slice_header, Slice, etc., after decoding, Mark reference frame, code stream fault tolerance end, and finally extend the decoding image boundary. The core of nal decoding is function Decode_slice (). 5 Image Slice decoding

H.264 's coding is based on slice, and the types of films are divided into I_type, P_type and B_type. FFmpeg H.264 decoder in this level of image chip decoding is not piecewise type, that is, unified processing. Because the entropy encoding of H.264 is divided into two kinds of cabac and CAVLC, the corresponding entropy decoding is also treated differently. The module Decode_slice () realizes the decoding function of the image slice, the function's topological structure is shown in the figure.

The image slice decoding is divided into two types of arithmetic entropy decoding Cabac and Huffman entropy decoding CAVLC, each of which includes the functions of entropy decoding, inverse quantization and inverse transformation, motion compensation and so on. The implementation process for the Decode_slice () module is as follows:

/* Decode image Slice/static int decode_slice (H264context *h) {Mpegenccontext * const S = &h->s; const int part_mask= s->partitioned_frame? (ac_end|
    Ac_error): 0x7F;
    s->mb_skip_run=-1;
        /*CABAC decoding */if (H->PPS.CABAC) {int i;
        /* Structured Code Flow * * Align_get_bits (&S->GB);  /* Initialize CABAC status/* Ff_init_cabac_states (&h->cabac, Ff_h264_lps_range, Ff_h264_mps_state, Ff_h264_lps_state,
        64); /* Fill Code Stream decoding structure * * Ff_init_cabac_decoder (&H->CABAC, S->gb.buffer + get_bits_co
        UNT (&S->GB)/8, (S->gb.size_in_bits-get_bits_count (&S->GB) + 7)/8);
            /* Calculates the initial state of fixed 400/for (i= 0; i < 399; i++) {int pre; if (H->slice_type = =