H.264 Rate Distortion Optimization

In H.264, Rate Distortion Optimization plays an important role.
Formula for rdo: J (mode) = SSD + λ * R (ref, mode, MV, residual)
SSD refers to the mean square sum of the difference between the reconstruction block and the source image, and λ is the Laplace multiplier;
R is the actual bitrate of Macro Block encoding in this mode, including the bit rate of the reference frame, mode, motion vector, and residual.
Of course, only R (mode, residual) is supported ).
However, because SSD needs to reconstruct the image, it must involve transformation quantization, entropy encoding, reverse transformation inversion, and reconstruction. The calculation amount is quite large.
So there is an alternative formula:
J (mode) = sad + λ * R (ref, mode, MV)

J (mode) = satd + λ * R (ref, mode, MV)
Sad is the absolute deviation between the prediction block and the source image in this mode.
Satd: the sum of the absolute values of the residual values of the 4x4 blocks transformed by Harman. Therefore, satd reflects the residual bit rate to some extent. Therefore, residual is ignored in the Replacement Formula R.

For (j = 0; j <16; j)
{
For (I = 0; I <16; I)
{
M1 [I] [J] = imgy_org [img-> opix_y + J] [img-> opix_x + I]-IMG-> mprr_2 [k] [J] [I]; calculate the Residual Block of the current macro block
M0 [I % 4] [I/4] [J % 4] [J/4] = m1 [I] [J];
}
}
Current_intra_sad_2 = 0; // NoSad

Start handicap here
For (JJ = 0; JJ <4; JJ)
{
For (II = 0; II <4; ii)
{
For (j = 0; j <4; j) first one-dimensionalHadamard Transformation

{
M3 [0] = M0 [0] [II] [J] [JJ] + M0 [3] [II] [J] [JJ];
M3 [1] = M0 [1] [II] [J] [JJ] + M0 [2] [II] [J] [JJ];
M3 [2] = M0 [1] [II] [J] [JJ]-M0 [2] [II] [J] [JJ];
M3 [3] = M0 [0] [II] [J] [JJ]-M0 [3] [II] [J] [JJ];


M0 [0] [II] [J] [JJ] = m3 [0] + m3 [1];
M0 [2] [II] [J] [JJ] = m3 [0]-m3 [1];
M0 [1] [II] [J] [JJ] = m3 [2] + m3 [3];
M0 [3] [II] [J] [JJ] = m3 [3]-m3 [2];
}

For (I = 0; I <4; I)
{

M3 [0] = M0 [I] [II] [0] [JJ] + M0 [I] [II] [3] [JJ];
M3 [1] = M0 [I] [II] [1] [JJ] + M0 [I] [II] [2] [JJ];
M3 [2] = M0 [I] [II] [1] [JJ]-M0 [I] [II] [2] [JJ];
M3 [3] = M0 [I] [II] [0] [JJ]-M0 [I] [II] [3] [JJ];

Second One-Dimensional Hadamard Transformation

M0 [I] [II] [0] [JJ] = m3 [0] + m3 [1];
M0 [I] [II] [2] [JJ] = m3 [0]-m3 [1];
M0 [I] [II] [1] [JJ] = m3 [2] + m3 [3];
M0 [I] [II] [3] [JJ] = m3 [3]-m3 [2];
For (j = 0; j <4; j)
If (I + J )! = 0)
Current_intra_sad_2 + = ABS (M0 [I] [II] [J] [JJ]);Take the absolute value sum as the cost of the transformed AC Residual Difference

}

}
}

For (j = 0; j <4; j)
For (I = 0; I <4; I)
M4 [I] [J] = M0 [0] [I] [0] [J]/4;

// Hadamard of DC koeff
For (j = 0; j <4; j)The next two for loops perform Hadamard transformation on the DC residual of the front macro block and take the transformed value to the absolute value.
Sum as the cost
{
M3 [0] = M4 [0] [J] + M4 [3] [J];
M3 [1] = M4 [1] [J] + M4 [2] [J];
M3 [2] = M4 [1] [J]-M4 [2] [J];
M3 [3] = M4 [0] [J]-M4 [3] [J];

M4 [0] [J] = m3 [0] + m3 [1];
M4 [2] [J] = m3 [0]-m3 [1];
M4 [1] [J] = m3 [2] + m3 [3];
M4 [3] [J] = m3 [3]-m3 [2];
}

For (I = 0; I <4; I)
{
M3 [0] = M4 [I] [0] + M4 [I] [3];
M3 [1] = M4 [I] [1] + M4 [I] [2];
M3 [2] = M4 [I] [1]-M4 [I] [2];
M3 [3] = M4 [I] [0]-M4 [I] [3];

M4 [I] [0] = m3 [0] + m3 [1];
M4 [I] [2] = m3 [0]-m3 [1];
M4 [I] [1] = m3 [2] + m3 [3];
M4 [I] [3] = m3 [3]-m3 [2];

For (j = 0; j <4; j)
Current_intra_sad_2 + = ABS (M4 [I] [J]);
}
If (current_intra_sad_2 <best_intra_sad2)
{
Best_intra_sad2 = current_intra_sad_2;
* Intra_mode = K; // update best intra Mode

}
}
}
Best_intra_sad2 = best_intra_sad2/2;

Return best_intra_sad2;
}

The above is a section in the source program. intra_16 * 16 is not a calculation.Sad

Value, but calculate satd.
Among them, M1 is the residual macro block, and M0 is also, but for the convenience of calculation of Hadamard transformation below, it is expressed as the form of M0 [4] [4] [4] [4] [4, the first two [4] [4] represent 8x8 coordinates, and the last two [4] [4] represent 4x4 coordinates in 8x8.
The program first performs Hadamard transformation on the residual, then puts forward all the DC components, and then performs Hadamard transformation on the DC components,
The final result is satd.
I don't understand either of the following two points:
1 Why do I divide the DC component by 4?
2. Why did best_intra_sad2 divide it by 2?

This is mainly because satd transformation is not a normalized matrix,The amplitude of the transformed system value increases, so the corresponding values of/2 and/4 are required.

The Hadamard transform itself has a/2 operation. Therefore, each transformation must perform a/2 operation on all coefficients. The find_sad_16x16 function is executed twice.
Hadamard Transformation: first, the 256 coefficients are performed once, and then all DC coefficients are performed again. Therefore, the DC coefficient should be/4, while the AC coefficient should be
/2. In the find_sad_16x16 function, M4 [I] [J] = M0 [0] [I] [0] [J]/4 is the coefficient of DC.
/4, and finally: best_intra_sad2 = best_intra_sad2/2; can be considered as a disguised form of AC Coefficient
/2. But here it is equivalent to all coefficients/2, so the DC coefficient is more than once/2. I don't know the reason for this one more time.

This issue was discussed in the 264 Park group. There is a conclusion on/2 of Hadamard transformation. However, no basis has been found for the second division of the DC coefficient.
The definition of Level 4 Hadamard transformation includes this/2. See http://en.wikipedia.org/wiki/Hadamard_transform
. One more explanation here
Assume that the Hadamard transformation does not have/2, and the transformation matrix is:
1 1 1 1
1-1 1-1
1 1-1-1
1-1-1 1
In this case, a column vector v = (1, 1, 1, 1) is transformed, that is, the column vector V is left multiplied by the transformation matrix, and the transformed vector V '= (4, 0, 0, 0 )'.

Now, observe V and V. In the Euclidean space, the "size" of a vector is measured by its length, which is obtained by calculating the inner product. So
Len (v) = SQRT (1 ^ 2 + 1 ^ 2 + 1 ^ 2 + 1 ^ 2) = 2
Len (V') = SQRT (4 ^ 2 + 0 ^ 2 + 0 ^ 2 + 0 ^ 2) = 4
It can be seen that if there is no/2, the length of the vector changes before and after the transformation. Such transformations violate the definition of orthogonal transformations.

Therefore, the Hadamard transformation, as an orthogonal transformation, must have the normalization of this/2.

A: By extension, the length of the vector before and after the integer DCT transformation also changes. Why is it not divided by 2?

DCT transformation (non-integer) is also a normalized integer transformation is also orthogonal transformation, so it will certainly meet the normalization. Did firstime forget to consider scaling matrix.

According to the 113 page of Bi Houjie's book, the transformation matrix is formula 6.15 (in this case, the scaling matrix has not been separated yet ?) :
A
B c-B
A-
C-B-c
A = 1/2, B = (2/5) ^ 0.5. This matrix does not have the same length for the column vector v = (1, 1, 1, 1) '.