H.264 reference image management in decoderPeter Lee 2005.11.24 videosky.9126.com [preface] Because H.264 adopts multi-Reference Frame Prediction Technology, it is complicated to manage reference images. Go50 only explains how to use it and does not explain why it is used. After reading it, you may have many questions. This article provides a detailed explanation of the problem of reference image management, aiming to further unveil its haze. :) Note: to facilitate the management of reference images from a macro perspective, this article does not consider B frame and field mode. PS: in H.264, The codecs must maintain the synchronization of the reference image list. Therefore, the reference image management in the encoder and decoder is basically the same. Before reading this article, you can first look at the I previously written "jm85 multi reference frame problem": http://blog.csdn.net/sunshine1314/archive/2004/11/08/172498.aspx [outline]
I. Meanings of Related Variables
Ii. decoding process (see image management for details)
Iii. Marking of reference images
Iv. Initialization of reference images
V. reordering of reference images
I. Meanings of Related VariablesDPB (decoded picture buffer): stores all the rebuilt images. listx [I]: The reference frame list contains reference frames that point to the rebuilt images in DPB and are sorted. Listx [0] is the list of forward reference frames. Both P and B are used. listx [1] is the list of backward reference frames. Only B frames are used. maxframenum: 1 <(active_sps-> log2_max_frame_num_minus4 + 4); frame_num: indicates the serial number of the Reference Frame (indicating the decoding sequence ). See the example in Figure 1. Figure 1 example of frame_num Value Note: When frame B is not used as a reference, we can ignore frame B, and then I/P frames increase sequentially. Frame B for reference, frame_num is exactly the same as the encoding/decoding sequence. Picnum: the serial number of the short-term reference image. The calculation process is as follows: // Note: frame_num is the serial number of the current image if (listx [I]-> frame_num) {framenumwrap = listx [I]-> frame_num-frame_num;} else framenumwrap = listx [I]-> frame_num; listx [I]-> picnum = framenumwrap; longtermpicnum: sequence Number of the long-term reference image, which is specified externally.
Ii. decoding process (see image management for details)Figure 2 initialization of the decoding process reference image list: place the required reconstruction images in DBP to listx [0] and sort them. Refer to the image list for resorting: Reordering listx [0] based on the syntax elements in the current slice code stream. Reference image tags: the reconstructed image in DBP can be marked as: no reference, short-term reference, or long-term reference.
Ii. Marking of reference imagesNote: The purpose of marking the reference image is to be consistent with the actual decoding process, because the reference frame is not required for 1st frames, and only the reference frame is saved. Skip this step if the current image is not for reference. 1. Save the reconstructed image to DPB. If the current image is an IDR frame, it indicates whether to clear DPB Based on the read syntax element. In addition, it is determined that the IDR frame is used as a long-term reference or a short-term reference. Non-IDR frame. The current frame is inserted as a short-term reference. 2. There are two methods for reference image marking: Automatic Sliding Window and custom marking. The method is determined by the corresponding syntactic elements. Automatic sliding window method: the FIFO method. When DPB is full, the short-term reference image with the minimum picnum in DPB (that is, the earliest reference image) is removed from DPB. Custom markup (adaptive_ref_pic_marking_mode): specifies the type of reference image set for a reference image in the syntax element or removes it. The command is as follows:
Table
1
Custom reference image tagging commands
| Command |
Tag operation |
| 0 |
End tag operation |
| 1 |
Mark a short-term reference as a non-reference (that is, remove) |
| 2 |
Mark a long-term reference as a non-reference (that is, remove) |
| 3 |
Mark a short-term reference as a long-term reference |
| 4 |
Specify the maximum number of long-term reference images |
| 5 |
Clear DBP and disable long-term reference |
| 6 |
Save the current image as a long-term reference |
Iii. Initialization of reference images1. Place the required reconstruction images in DBP to listx [0] 2. Sort listx [0]. Put the short-term references first, and then the long-term references. Sort the short-term references in descending order by picnum, for long-term reference, sort by longtermpicnum in ascending order. For details, see the example in table 2:
Table 2
Reference Image initialization and sorting instance
Iv. reordering of reference imagesWhy re-sorting: reduce the bitrate. When a reference image with a large serial number (ref_idx) is used in a large proportion, a smaller serial number of the reference image can reduce the bit rate. This process is re-ordered. The general process of reordering the reference image is as follows: traverse the reference image in listx [0], read the syntax element reordering_of_pic_nums_idc for each traversal to a reference image, and update its reference image number. The reordering_of_pic_nums_idc command is as follows: 0/1: Specify the serial number of the short-term reference image, and use the offset and offset Offset Method 2: Specify the serial number of the long-term reference image 3: Terminate the re-sorting. Reference Image Re-sorting example:
