easypr--Development Details (3) Gaussian blur, grayscale, and Sobel operators

In the previous article, we learned about all the steps in the platelocate process. In this article, we analyze the first 3 steps, namely Gaussian blur, grayscale and Sobel operator.

One, Gaussian Blur

1. Target

The image denoising is prepared for the edge detection algorithm.

2. Effects

The first step in our license plate positioning is Gaussian blur.

　　

Figure 1 Gaussian Blur effect

3. Theory

Detailed explanation can see this: Nanyi speak Gaussian Blur.

Gaussian Blur is a very well-known image processing technology. As the name implies, its general application is to blur the image, but at the same time Gaussian blur is also applied in the preprocessing phase of the image. Before understanding the Gaussian blur, first look at the average fuzzy algorithm. The average fuzzy algorithm is very simple. In the figure below, the values of each pixel take the average of all the pixels around them (8 total).

Figure 2 Average Blur diagram

In the image above, the pixel value of the red dot on the left is originally 2, and after blurring, it becomes 1 (the mean value of all surrounding pixels). In the average blur, the weights of the surrounding pixels are the same, all 1. If the weights of the surrounding pixels are different, and the values of the two-dimensional Gaussian distributions are the same, then the Gaussian blur is called.

In the above blur process, each pixel takes the average of the surrounding lap, also known as the blur radius of 1. If you take around three laps, it is called a radius of 3. If the radius increases, the effect will be more deeply blurred.

4. Practice

In Platelocate, this is called Gaussian blur.

    Gaussian Blur. The number in size affects the effect of license plate positioning.
    Gaussianblur (SRC, Src_blur, Size (M_gaussianblursize, m_gaussianblursize), 
        0, 0, Border_default);

The parameter of the size field specifies the radius of the Gaussian blur. The value is the m_gaussianblursize variable of the Cplatelocate class. Because the Gaussian blur of OpenCV only receives an odd number of radii, the variable is an even value that throws an exception.

This gives the OPENCV's Gaussian Blur API (in English, 2.48 or more).

Gaussian blur this process must be necessary. The author's answer is necessary, if we comment on this code and modify it slightly, rerun it. You will find that the platelocate process has changed in the closed operation. The final result is as follows.

Figure 3 Results with no Gaussian blur

As you can see, the rectangle where the license plate is located produces a skew. The final candidate "license plate" tiles are as follows:

Figure 4 "license plate" tiles without Gaussian blur

If we use the edge detection algorithm without Gaussian blur, we get the candidate "license plate" to reach 8. This will not only increase the processing time of license plate judgment, but also increase the probability of judging errors. Since the license plate in the obtained license plate is skewed, if our character recognition algorithm requires a level license plate block, then almost certainly we will not get the correct character recognition effect.

The radius in the Gaussian blur also causes a noticeable change in the result. Some pictures, the Gaussian blur radius is too high, the license plate will not be located. Some pictures, the Gaussian blur radius is low, the license plate also can not be located. Therefore, the radius of Gaussian blur is neither too high nor too low. A static constant default_gaussianblur_size with a value of 5 in the Cplatelocate class, indicating the radius of the recommended Gaussian blur. This value is one of the most comprehensive positioning rates for nearly thousands images that have been tested. In the constructor of the Cplatelocate class, M_gaussianblursize is given a value of default_gaussianblur_size, so the radius of the default Gaussian blur is 5. If it is not a special case, you do not need to modify it.

After several experiments, it must be admitted that preserving the Gaussian blur process with a radius value of 5 is the best practice. In order to respond to special needs, the Cplatelocate class should also provide a method to modify the value of the Gaussian radius, calling the code (assuming a Gaussian blur radius of 3 is required) as follows:

    Cplatelocate plate;
    Plate.setgaussianblursize (3);

At present, the processing steps of EASYPR are Gaussian blur, then grayscale. Judging from the present experimental results, the Gaussian blur process based on color is more likely to detect the edge point than the Gaussian blur process after gray.

Second, grayscale processing

1. Target

Prepare a grayscale environment for edge detection algorithms.

2. Effects

The effect of grayscale is as follows.

Figure 5 Grayscale Effect

3. Theory

In the grayscale processing step, the most controversial is the loss of information. Undoubtedly, the original platelocate process to face the picture is a color picture, and from this step, will face is a grayscale image. In the preceding, it has been said that this step is beneficial and the disadvantage is to be discussed.

Undoubtedly, for the computer, the color image is more difficult to deal with than the grayscale image, many image processing algorithms only apply to grayscale images, such as the Sobel operator mentioned later. In this case, you have no choice but to turn the image into a grayscale image and then process it, unless you redesign the algorithm. On the other hand, the most abundant details are lost after being transformed into grayscale images. You know, the real world is colored, and the human identification of things is based on a colored frame. It is even possible to say that because our naked eye can distinguish between colors, so our ability to differentiate, discern, and remember things is very strong.

The pros and cons of removing color in the license plate positioning link are also the same. Converting to grayscale images, while benefiting from the use of a variety of proprietary algorithms, loses the distinction between the most important tools---the real world. For a simple example, how do people find a license plate in a picture? Very simple, one glance, a suitable size rectangle, blue, or yellow, or other color in another black, or a large white in the shape of a car-like rectangle. This process is very intuitive, obvious, and can be ruled out blurred, color, unclear and many other effects. If you use grayscale images, you must rely on horizontal, vertical derivation and other methods.

In the future, if the platelocate process can be judged using color, it may be clearer and more accurate than the current positioning. But this requires research and experimentation, which may be implemented in future versions of EASYPR. But undoubtedly, the use of color judgment is a trend, because it not only conforms to the law of human eye recognition, but also closer to the nature of artificial intelligence, and it is more accurate, faster.

4. Practice

This is called grayscale in the platelocate process.

Cvtcolor (Src_blur, Src_gray, Cv_rgb2gray);

This gives the OPENCV's grayscale API (in English, 2.48 or more).

Three. Sobel operator

1. Target

Detects the vertical edges in the image to make it easier to differentiate between plates.

2. Effects

The following figure is the effect of the Sobel operator.

Figure 6 Sobel Effect

3. Theory

If you want to say which step is the core and soul in Platelocate, there is no doubt that the Sobel operator. There is no Sobel operator, there is no vertical edge detection, you can not get the possible location of the license plate, there is no subsequent series of license plate judgment, character recognition process. Through the Sobel operator, it is convenient to get a relatively accurate location of the license plate, for our follow-up treatment to lay a solid foundation. In the implementation of the above platelocate can be seen, it is through the Sobel operator, the license plate characters and the car's background is clearly differentiated, for the back of the two value and closed operation laid the foundation. So how does the Sobel operator work?

The principle of soble operator is to find the first-order horizontal and vertical derivative of the image, and determine whether it is the edge according to the size of the guide value. Please see csdn Xiao Wei's blog (Watch out for the GX and Gy in her blog).

In order to calculate conveniently, the soble operator does not really go to the derivation, but uses the weighted sum of the peripheral values, which is called "convolution" academically. The weight value is called the convolution template. For example, the left side of the figure is the Sobel GX convolution template (calculates the vertical edge), the middle is the original image, and the right is the new image after the convolution template.

Figure 7 Sobel operator GX schematic diagram

Here is demonstrated by the convolution template, the original image of the red pixel is originally a value of 5, after convolution calculation (-1 * 3-2 * 3-1 * 4 + 1 * 5 + 2 * 7 + 1 * 6 = 12) The value of the red pixel becomes 12.

4. Practice

Calling the soble operator in your code requires more steps.

    Generate grad_x and grad_y
    Mat grad_x, grad_y;
    Mat abs_grad_x, abs_grad_y;

    Gradient X
    //scharr (Src_gray, grad_x, ddepth, 1, 0, scale, Delta, Border_default);
    Sobel (Src_gray, grad_x, ddepth, 1, 0, 3, scale, Delta, Border_default);
    Convertscaleabs (grad_x, abs_grad_x);

    Gradient Y
    //scharr (Src_gray, grad_y, ddepth, 0, 1, scale, Delta, Border_default);
    Sobel (Src_gray, grad_y, ddepth, 0, 1, 3, scale, Delta, Border_default);
    Convertscaleabs (grad_y, abs_grad_y);

    Total Gradient (approximate)
    addweighted (abs_grad_x, Sobel_x_weight, abs_grad_y, sobel_y_weight, 0, grad);

This gives the API for OpenCV Sobel (in English, 2.48 or more)

There are two constants in the call parameter Sobel_x_weight and sobel_y_weight for the horizontal and vertical weights, the default is 1, the latter is 0, representing only the horizontal direction of the derivative, not the vertical direction of derivation. The implication is that if we take a vertical derivative, we will detect a lot of horizontal edges. The horizontal edges may help to generate more precise contours, but because some of the front end of the car is too many horizontal edges, such as the head vent, signs and so on, a lot of horizontal edge will mislead our connection results, resulting in we do not have a exact license plate location. For example, we do the following experiment on the test diagram, which sets both Sobel_x_weight and sobel_y_weight to 0.5 (which means that the weights of the two are equal), so the resulting graph of the last closed operation is

Since the Sobel operator is so important that it is possible to differentiate the license plate from other areas clearly, the problem is that there are no operators similar to the Sobel function that can achieve consistent results, or there are no better operators than the Sobel effect.

The first derivative of the image is obtained by the Sobel operator, and the Laplace operator is the second derivative of the image, and the edge can be detected under normal circumstances, but the detection of Laplace operator is not horizontal or vertical. The following figure is a comparison between the Laplace operator and the Sobel operator.

Fig. 8 schematic diagram of Sobel and Laplace
　　

As can be seen, the image through the Laplace operator contains both horizontal and vertical edges, according to the description we just made. Horizontal edge for the detection of license plate is generally harmful. After the test of nearly hundred images, the effect of Sobel operator is better than that of Laplace operator, so it is unsuitable to use Laplace operator instead of Sobel operator.

In addition to the Sobel operator, there is also an operator, Shcarr operator. But this operator is actually only a variant of the Sobel operator, because the Sobel operator is often less accurate in the 3*3 convolution template, so there is a special Sobel operator whose weights are expressed according to the following figure, called the ScHARR operator. The following figure is a comparison between the Sobel operator and the ScHARR operator.

Fig. 9 schematic diagram of Sobel and ScHARR

In general, the ScHARR operator is better than the Sobel operator to detect edges, as can be seen from the above figure. However, this "better" is a double-edged sword. Our goal is not to draw the edge of the image, but to determine a region of the license plate, the finer the edge will interfere with the subsequent closed operation. Therefore, the Sobel operator is generally superior to the ScHARR operator for a large number of image tests.

For more detailed explanations of sobel operators and ScHARR operators and Sobel operators, you can refer to the official website: Sobel and ScHARR