LCU mode selection Process analysis (CTU depth selection and CU segmentation)

1. Rate distortion cost calculation ModelHEVC's largest coding unit is LCU, 64x64 cu, to a LCU select the best cu coding depth, need to traverse all 64x64 to 8x8 division, a total of 85 CU, through the calculation of the cost of distortion to choose this LCU the best way to split. For each CU, iterate through all the selectable predictive modes within and between frames, choosing the best PU Prediction model based on the cost of distortion. For each PU prediction mode, the method of Tu segmentation is determined by the current CU size PU prediction mode and the maximum TU partition depth, and the selection process also needs to calculate the cost of distortion. It can be seen that in the HEVC coding process, any method of partitioning or prediction mode selection, need to calculate the rate of distortion cost, HM Encoder, the rate of distortion cost calculation model has the following two kinds:
1, non-RDO model can be used in intra-frame prediction, motion estimation, the selection of the best MVP and the selection of the best motion parameter set in the merge mode. 2. The RDO model is used to calculate the rate distortion cost of all the selection of the method of partitioning. 2.LCU Encoding Depth selection processWhen the size of the LCU is 64x64 and the maximum encoding depth is 3, the selection process for the LCU encoding depth can be illustrated as shown below.

The first step, as shown in Figure a), is to traverse all inter-frame and intra-frame prediction modes for code unit A with a size of 64x64 depth of 0, resulting in an optimal predictive mode with a depth of 0 o'clock and a rate-distortion-cost Ra.
The second step, as shown in Figure B, a CU division of a, get four sub-cu:b0,b1,b2,b3, at this time the coding depth of 1, and the coding unit B0 traverse all inter-frame and intra-frame prediction mode, the B0 optimal prediction method and rate distortion cost Rb0.
The third step, as shown in Figure C, for further CU division of the B0, get four sub-cu:c0,c1,c2 and C3, at this time the coding depth of 2, and the coding unit C0 traverse all possible prediction modes, the C0 optimal prediction mode and rate distortion cost Rc0.
Fourth step, as shown in Figure D, the C0 do further CU division, get four sub-cu:d0,d1,d2 and D3, at this time the coding depth of 3, has reached the maximum coding depth, can not be further CU division. In order to select D0, D1, D2 and D3, the corresponding optimal prediction mode and rate distortion cost Rd0, RD1, Rd2 and Rd3 are calculated, the sum of the cost of four CU is computed and compared with RC0, and the lower value is chosen as the cost of the optimal rate distortion of c0 (recorded as MIN-RC0), the corresponding prediction method and segmentation method are the best prediction and segmentation method of C0.
Fifth step, follow the fourth step, in order to C1, C2 and C3 to divide and predict the mode of selection, respectively, the corresponding optimal prediction method and the rate of distortion cost min-rc1, MIN-RC2 and MIN-RC3, and calculate the current coding depth of four CU rate distortion cost, compared with Rb0, The cost of the low rate distortion (recorded as min-rb0) is obtained, and the corresponding prediction mode and segmentation method are the optimal prediction mode and segmentation method of B0.
Sixth step, follow the second to the fifth step, the B1, B2 and B3 are divided and the prediction mode selection, respectively, the corresponding optimal prediction method and rate distortion cost min-rb1, MIN-RB2, MIN-RB3, and calculate the current coding depth of four CU rate distortion cost of the sum, and Ra By comparison, the cost of low rate distortion (recorded as Min-ra) is obtained, and the optimal partitioning method and the optimal prediction mode of the LCU are found out.
3.CU Predictive mode selection process

In the selection process of the best partitioning method of LCU, for each CU of coding depth, make PU prediction mode selection, a 2nx2n Cu,pu frame prediction mode with Skip, 2nx2n, 2NxN, nx2n, NxN, 2NxnU, 2NxnD, nlx2n and NR X2n. The CU partition method for the skip mode is 2NX2N,NXN mode, which is only used when the CU size is 8x8. After four modes are collectively known as amp mode, the amp mode can be closed directly in the parameter configuration as needed. In the case of AMP mode open, the selection of these four modes is related to the previous selection results. The size of the in-frame mode PU can be 2nx2n or NXN,NXN only for CU with a depth of maximum coding depth. For each
PU, there are 35 kinds of intra-frame prediction mode. The mode selection process for PU is shown in Figure 3.3:
The first step is to calculate the rate distortion cost of the skip mode, and the skip mode is applied to the newly introduced motion merging technology in HEVC. The prediction block is found by using each motion parameter set in the candidate list of motion parameters, and the difference is made with the original block and transformed, quantified and entropy encoded. Finally, we calculate the cost of the distortion by the number of bits and the distortion of the reconstructed block and the original block, and find the best merge-index.
The second step, calculate the rate distortion cost of the 2NX2N mode, first use the AMVP technology to get the MVP list, then use the non-RDO method to calculate the price distortion cost to get the best MVP; Finally, the motion search, find the prediction block, and the original block to do the difference and transform, quantization and entropy coding, Calculates the cost of the distortion using the number of encoded bits and the SSD that reconstructs the block and the original block. Compare the rate distortion cost of skip mode to find the best PU mode in Skip and 2nx2n.
The third step, calculate the rate distortion cost of 2NxN and nx2n mode, first use AMVP technology, find the best MVP and make motion search to find prediction block, then use the technique of motion merging to get the best merge-index corresponding rate distortion cost, and determine the best prediction mode according to the price distortion cost. Finally, the residual error of the best mode is transformed quantization and entropy coding using RDO to calculate the current rate distortion cost of PU mode, and compare with the rate distortion cost of the best PU mode obtained in the second step, get the best pu mode.
Fourth step, the rate distortion cost of the AMP mode is calculated, the amp mode includes four kinds of segmentation methods, but in order to reduce the coding complexity, the encoding process does not traverse all the patterns, but is based on the best Pu mode selected in the first three steps and the best segmentation mode of the CU on the previous layer, and the model decision is chosen partially. Select the following table:

  Fifth step, intra-frame prediction mode selection, for I_slice or non-i_slice need to encode the quantization coefficient is not all 0 CU, need to do intra-frame prediction of the segmentation method and intra-frame prediction mode selection, the selection process is as follows:  
1, the predicted unit PU size and CU The size of the uniform is 2nx2n, with the non-RDO model to calculate the cost of distortion, from the 35 intra-frame prediction mode to find the less-than-cost N-mode Prediction model as candidate mode, the size of n is related to the size of the PU, as shown in table 3-2.  
2, the current PU on the upper side and the left adjacent PU used in the intra-frame prediction mode, that is, the current PU most likely predictive mode (MPM) into the candidate mode of the step, at this time the candidate Mode K is (n+2).  
3, using the RDO model to calculate the rate of distortion cost in the K best candidate mode to select the best brightness of the intra-frame prediction mode.  
4, according to the brightness of the intra-frame prediction mode selection results, the color of the intra-frame prediction mode selection.  
5, if the encoding depth of the current encoding unit is the maximum encoding depth, the current encoding unit is divided into 4 nxn pu, for each PU repeat 1~4 step, find each PU best intra-frame prediction mode and the best prediction mode corresponding to the rate of distortion cost.  
6, according to the previous steps to obtain the value of distortion cost, determine the current CU optimal partitioning method and intra-frame prediction mode.


Sixth step, calculate the rate distortion cost of the PCM mode.


As described above, for a LCU want to get the best CU partition, need to traverse all of its Cu, for each CU also need to traverse a variety of prediction mode, get its minimum rate distortion cost. In order to speed up the coding, the optimization of the mode selection process can be studied and analyzed from three aspects, such as the choice of CU coding depth, the selection of PU mode and the choice of TU partition mode. Because the PU mode selection and the Tu division method is chosen to calculate the current depth, the rate of the coding CU distortion cost, reduce the number of CU traversal, but also reduce the number of times the choice of Pu mode and the number of TU division choice. Therefore, it is most effective to optimize the algorithm based on the CU coding depth selection process.
This digest from the paper: Next generation Video coding standard HEVC inter-frame Predictive optimization algorithm _ Zhang