The block filter of H.

Block effect and its cause

When we watch the video, in the scene of vigorous motion, we can often observe that the small squares appear in the image, and the small squares show discontinuous effect at the boundary (such as), this phenomenon is called the Block effect (blocking artifact).

First, we need to figure out why the block effect occurs. In the coding process, the DCT transform of the pixel residuals was obtained, and the DCT coefficients were correlated with each pixel, which represented the basic tones and details of the transformed data. After DCT transformation, the DCT coefficients are quantified, quantization can effectively remove the space redundancy between adjacent pixels, which means that some details of the element data will be erased. The ideal scenario is to quantify the amount of detail that is not recognized by the human eye, but in the case of low bitrate, the details of the original data are lost too much. Moreover, the DCT transform is based on the block, will be 8x8 or 4x4 pixel residuals after the conversion of 8x8 or 4X4DCT coefficients, if the low bit rate quantization, will make the adjacent two blocks of the correlation between the difference, resulting in block effect.

The motion compensation of H + + increases the block effect caused by the transformation quantization. Because the matching of the motion compensation block is impossible to be absolutely accurate, the residual size of each block is different, especially when the reference frame of the adjacent two blocks is not large, and the gap between the motion vector or the reference block is too big, the data generated on the block boundary is more obvious.

There are two main forms of block effect: one is that the high-frequency coefficients of DCT are quantized to 0, which makes the strong edge jagged at cross-boundary, called trapezoid noise, and the other one often appears in the flat area, because quantization causes the DC coefficient of the brightness block to be changed, which causes the change of the base hue of the transform block. This is called lattice noise.

The location of the de-block filter in the decoder

To mitigate and eliminate block effects in video images, filters are often used to filter the pixels at the block boundary to smooth out the mutation of the pixel value, which is called a de-block filter (deblocking filter).

The content of the block filter is specified in the Standard 8.7 section, which is called the Loop filter. Loop filter is placed in the decoded image reconstruction loop. In a codec environment where loop filtering is enabled, both the encoder and the decoder are filtered after the image has been reconstructed. In the encoder, the filtered image is used as a reference image for subsequent coded motion compensation, and in the decoder, the filtered image is output and displayed as a reference image for subsequent image decoding and reconstruction.

Pre-filtering preparation 1. Filter parameters

In the standard, the de-block filter is applied to the brightness and chroma macro block filtering, the syntax element DISABLE_DEBLOCKING_FILTER_IDC is used to control whether the block filter is open, it has a value of three: 0~2.

• 0: Turn on the block filter function, the block filter can cross the slice boundary.
• 1: Turn off the block filtering function.
• 2: The block filtering function is turned on, but the filter can only be performed on a macro block within the same slice range.

2. Filter boundary

The block filter is based on the macro block, including the brightness macro block and the Chroma macro block. The width height of the brightness macro block is 16x16 macro block, and the Chroma macro block has several different formats, and the block filter boundary is as follows:

The thick lines in the figure are filtered boundaries, where the thick red line is the horizontal boundary (horizontal edge), the thick blue line is the vertical boundary (Vertical edge), and the filter boundary divides the macro block into multiple 4x4 blocks. One thing to note is that if Transform_8x8_mode_flag is 1, then the luminance macro block and the 4:4:4 Chroma macro block will take the 8x8 DCT, when the brightness macro block and the filter boundary of the 4:4:4 Chroma macro block will split the macro block into an 8x8 block.

The filtering boundary can also be subdivided according to whether the filtering process uses macro blocks other than the current macro block. The top boundary of the macro block, the left boundary due to the edge of the macro block, the filter must use the adjacent macro block, and the rest of the filter boundary in the filter will only use the current macro block pixels inside.

• Left MB Edge: The leftmost macro block filtering boundary of the macro block in the vertical direction
  • If the current macro block is the leftmost macro block of the image, the left edge filter is not required
  • If it is specified that only the current slice macro block can be filtered (disable_deblocking_filter_idc=2), and the current macro block and its left macro block is not the same slice, there is no need to filter the left edge
• Vertical Internal Mb Edge: The internal filtering boundary of the macro block in the vertical direction. Different macro block types contain different numbers of vertical internal filter boundaries

  Macro Block Format	Y	4:2:0 CB/CR	4:2:2 CB/CR	4:4:4 CB/CR
  Number of vertical internal filter boundaries	3	1	1	3

• Top MB Edge: Macro block filtering boundary at the top of the macro block in the horizontal direction
  • Top edge filtering is not required if the current macro block is the topmost macro block of the image
  • If it is specified that only the current slice macro block can be filtered (disable_deblocking_filter_idc=2), and the current macro block is not the same slice as the macro block above, then the top boundary filtering is not required.
• Vertical Internal Mb Edge: The internal filtering boundary of the macro block in the vertical direction. Different macro block types contain different numbers of vertical internal filter boundaries

  Macro Block Format	Y	4:2:0 CB/CR	4:2:2 CB/CR	4:4:4 CB/CR
  Number of horizontal internal filter boundaries	3	1	3	3

Filtering the macro-block filtering of the brightness macro block, filtering the filtering boundary of a macro block, also need to follow a certain sequence.

• Vertical boundary filtering is performed first, from left to right
• Horizontal boundary filtering, from top to bottom

3. Filter source Pixel selection

The source pixels used to filter the block are distributed on both sides of the boundary, with 4 pixels, as shown in

The 4x4 or 8x8 blocks where p and q pixels are located are respectively p blocks and Q blocks.

If the currently encoded image is encoded as a frame or field, the filtered source pixels can be directly followed by the positions on either side of the boundary. However, if the image is encoded in a frame-field adaptive manner (mbaff), the pixels on either side of the boundary need to be positioned to get the correct source pixels.

As we have discussed earlier, the block effect is caused by the DCT quantization of block, the purpose of the block filter is to eliminate the block effect, so the block filter needs to correctly locate the quantization of DCT transform. In the Frame-field adaptive coding environment, the macro block can be encoded in the way of frame or field, but after the macro block is reconstructed, all the frames are macro blocks, so we need to locate the pixel of the block where the DCT transform quantization is based on the actual situation.

The following is an example of a vertical boundary filtering pixel location in a Frame field adaptive coding environment

The following is an example of a horizontal boundary filtering pixel location in a frame-field adaptive coding environment

In addition, in the Frame field Adaptive coding environment, if the current macro block is a Frame macro block, its upper macro block is a field macro block, then the top boundary filtering requires two boundary filtering

Filtering process 1. Estimating boundary strength

For filtering boundaries, we first need to roughly estimate the pixel gap on both sides of the boundary based on the location of the boundary and the information of the macro block, which we call the pixel gap as the boundary strength (bs,boundary strength).

Judging conditions	Boundary Strength BS
P Block or Q block is in-frame encoding mode, and the block edge is the macro block edge	4
P Block or Q block is in-frame encoding mode	3
Residual transformation coefficients for p blocks or Q blocks contain non 0 coefficients	2
The residual transformation coefficients of P block or Q block do not contain a non-0 coefficient, and the number of reference frames or motion vectors (forward and backward) of P block and Q block is different	1
The residual transformation coefficients of P block or Q block do not contain a non-0 coefficient, and the reference frame of P block and Q block and the number of motion vectors (forward and backward) are the same	0

The above table is used for the calculation of luminance BS, and the BS of the Chroma macro block follows its corresponding luminance macro block. As the table description is not detailed, please refer to standard 8.7.2.1 for details.

2. Distinguish between true and false boundaries

After roughly estimating the intensity of the filtering boundary, we need to distinguish between the block effect (false boundary) or the original boundary of the video image (the true boundary) caused by the DCT quantization of the block. If the real boundary is not to be filtered, if it is a false boundary, then block filtering is required. The distinction between true and false boundaries is based on the following two assumptions:

• The difference between pixels on the real boundary is usually greater than the pixel value on both sides of the false boundary.
• For a true boundary with a small difference in pixel values, even with a block filter, the subjective effect of the two sides is not much affected.

Therefore, the de-block filter should follow the following principles:

• In a flat area, even small pixels are easy to detect, so you can change more pixels by using a stronger block filter.
• For complex areas, to maintain image detail, use a weaker block filter to change fewer pixels.

Assumed to be the luminance value distribution of the pixel points, in which case the difference between the pixel points is very large, according to the above hypothesis, between P0 and Q0 is the real boundary of the object, so there is no need to filter.

Standard H. Two sets the threshold α and β to determine the true and false boundary, α represents the boundary threshold between blocks and blocks, and β represents the threshold of the inner boundary of a block. For the difference of pixel points on both sides of the boundary, if the following three conditions are satisfied, it will be judged as the false boundary that needs filtering, otherwise it is decided that the real boundary is not needed.

|p0-q0| <α[indexa]

|p1-p0| <Β[INDEXB]

|q1-q0| <Β[INDEXB]

where α and β can be obtained from the table by Indexa and Indexb. Indexa and Indxeb are the indexes of tables, and they are calculated as follows

Indexa = CLIP3 (0, Wuyi, Qpaverage + Filteroffseta)

Indexb = CLIP3 (0, Wuyi, Qpaverage + filterqffsetb)

where qpaverage = (qpp+qpq+1)/2,filteroffseta and FILTEROFFSETB are offsets, offsets are used to adjust the filter strength. When we need to increase the intensity of filtering, we can remove the block effect caused by suboptimal motion estimation, Improper coding mode selection, and improve the subjective quality of the image. When the filtering strength is reduced, the negative offset can protect the image details from being blurred by the smoothing effect of the filter. The offset will be transferred in the slice header information, please refer to the deblocking related syntax elements in the syntax structure analysis of H.

Index	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21st	22	23	24	25
α		0	0		0		4	7		12
β		0	0		0		2	3		4

Index	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46	47	48	49	50	51
Alpha	15	17	20	22	25	28	32	36	40	45	50	56	63	71	80	90	101	113	127	144	162	182	203	226	255	255
β		8	9			13	15		17

It is known from the above that α and β depend on the size of QP, Indexa, Indexb and α, β corresponding values are shown in the table below

The larger the QP is visible (the larger the index), the greater the alpha and beta. The larger the QP means that the greater the quantization error, the more obvious the block effect, so the threshold value should also be larger to increase the filtering effect, whereas the threshold should be smaller.

3. Filtering operations

In the preceding we discussed 5 kinds of boundary strength BS, boundary filtering is required when the boundary strength is not 0 o'clock. There are two kinds of filters for boundary filter of H.

• BS = three-way, with weak intensity filter, first change p0, q0 two pixels, and then use threshold β to determine whether need to adjust P1 and Q1
• BS = 4, at this time there are two kinds of intensity filter, strong filter can change 6 pixels (P0, p1, p2, q0, Q1, Q2), weak filter only change two points on the boundary (P0, q0)

(1) BS = filter operation at the time of the

① first filter the two pixel points on the boundary P0 and q0, it needs to input P1, P0, q0, Q1, the filtering process is as follows

1. First to get the difference Δ, the calculation of the difference: δ= ((q0-p0) <<2 + (P1-Q1) + 4) >> 3
2. Then we need to limit the difference δ, to ensure that the difference in a certain range, this range is mainly through the table to get, for details, please check the standard 8.7.2.3
3. Use the difference δ to calculate the new P0, Q0, which is the filtered value

② then filters the pixel points P1 and Q1 in the block respectively. The 4:2:0 and 4:2:2 Chroma macro block boundaries are not required to perform this part of the filter. If you are calculating P1, you will need to enter P2, p1, P0, q0, and Q1, P0, q0, Q1 if Q2.

In addition, only satisfying |p2-p0|<β can filter the P1, because satisfying this condition is that the internal P1 of P block has a false boundary, and the filtering process of P1 is as follows

1. To get the difference Δ first, the difference is calculated as: δ= (P2 + ((p0+q0+1) >>1) − (p1<<1)) >> 1
2. Then we need to limit the difference δ, to ensure that the difference in a certain range, this range is mainly through the table to get, for details, please check the standard 8.7.2.3
3. Calculate the new P1 with the difference value

The Q1 filtering process is also a similar step.

(2) The filter operation of BS = 4 o'clock

In the intra-frame prediction coding, it tends to encode the simple area of the texture with 16x16 luminance prediction mode (such as Blue sky, white wall, etc.), in order to achieve the purpose of fast coding. Although this method only causes a slight block effect on the boundary of the macro block, in this case, even a small intensity value table will visually produce a steep ladder-like feeling (color block layering), so for this content smooth macro block boundaries need to adopt a strong filter; If there is a lot of detail in the macro block boundary at this time, Instead, strong filtering should not be done. The threshold method is still used to determine whether there is a real boundary, if there is no large amount of detail information, strong filtering can be done, the other is weak filtering.

Here the filter is a good understanding of the tap Filter, p, q block on the filtering process is similar, here is the case of P block.

For points of the P block, if the following formula is satisfied, the detail information is not considered to be much:

$\left\{\begin{matrix}
|p0-q0|&<& (\alpha >> 2) +2\\
|p2-p0|&<&\beta
\end{matrix}\right.$

Using Strong filtering

$\left\{\begin{matrix}
P0 &= & (p2+2p1+2p0+2q0+q1+4) >>3\\
P1 &= & (p2+p1+p0+q0+2) >>2 \ \
P2 &= & (2p3+3p2+p1+p0+q0+4) >>3
\end{matrix}\right.$

Otherwise use weak filter, only change P0 point

$p 0 = (2p1+p0+q1+2) >>2$

Reference documents

ITU-T Rec. H. 04-2013 advanced Video coding for generic audiovisual services

Chen, Liu Jing, Cao Xixin: In-depth understanding of video codec technology--based on the standard and reference model of H.

The block filter of H.