Easypr-detailed description of Chinese Open Source License Plate Recognition System Development (2) License Plate positioning

This is the third article in the series. The addresses of the first two articles are described as follows: introduction and explanation 1. I wrote this series of articles for the following purposes: 1. Popularize the technologies and knowledge points related to License Plate Recognition; 2. help developers understand the Implementation Details of easypr; 3. Improve communication.

The easypr Project address is here: GitHub. To run the easypr program, you must first configure opencv. For details, refer to this article.

In the first two articles, we have initially understood the rough content of easypr. In this article, we started to go deep into the program details of easyrp. Learn how easypr implements a License Plate Recognition process step by step. According to the easypr structure, we divide it into six parts. The first three parts are collectively referred to as the "Plate Detect" process. The main purpose is to find a picture that only contains a license plate in an image, so as to improve the overall recognition accuracy and speed. This process is very important. If this step fails, you can skip the subsequent character recognition process. The three parts in the "Plate Detect" process are also called "plate locate", "SVM train", and "plate judge ", the most important part is the "plate locate" process. This article mainly introduces the "plate locate" process and answers the following three questions:

1. What is the role of this process and why is it important?

2. How does one implement the license plate locating function in this process?

3. What are the details of this process and how to optimize it?

 

 1. role and importance of "plate locate"

Before explaining the role and importance of plate locate, see the following two images.

Figure 1 two pictures with different license plates

The picture on the left is the picture trained by the author (most of the training and testing by the author are based on the picture captured by such traffic ), the picture on the right is obtained from the "License Plate" in Baidu pictures (this picture can also be called a photo of life ). The question about the picture on the right was asked by a netizen during comments. He said easypr has a low recognition rate when processing Baidu images. Indeed, due to the different purposes of industrial and life applications, the size, angle, color, and clarity of the license plate are different. For image processing technology, some algorithms have certain requirements or assumptions for the form and structure of images. Therefore, the algorithms used in one scenario are not applicable to other scenarios. Currently, all easypr functions are made based on pictures in traffic capturing scenarios, which makes it unable to process these license plate photos in real life scenarios.

Can we use the same "plate locate" process to process it? The answer may be yes, but it is difficult. Even if the processing is successful, the efficiency may not be satisfactory. I recommend that you adapt to different scenarios. Although the plate locate process cannot process the positioning of living photos, the two are common in the subsequent character recognition process. We can transform the plate locate of easypr while still using the overall architecture. This may be feasible.

It is worth noting that in the production environment, the image format you are facing is fixed, for example, the picture on the left. You can optimize your license plate program based on the specific image format, so that your program is robust enough for such images, and the efficiency is high enough. After the release, it also has good results. However, when you adjust the image format, you must adjust your algorithm. Some parameters can be adjusted during plate locate. It would be better to adjust these parameters to make the program work well. When these parameters cannot meet your needs, you need to completely modify the implementation code of easypr. Therefore, developers need to understand how easypr implements platelocate.

In easypr, the "plate locate" process is encapsulated into a "cplatelocate" class, which is implemented in "plate_locate.cpp" through the "plate_locate.h" declaration.

Cplatelocate contains three methods and several variables. The method provides the main functions of plate number locating, while variables provide customizable parameters. Some parameters have a significant impact on the effect of plate number locating, for example, Gaussian Blur radius, horizontal and vertical weight of the Sobel operator, and closed rectangular width. The cplatelocate class declaration is as follows:

class CPlateLocate {public:    CPlateLocate();    //! 车牌定位    int plateLocate(Mat, vector<Mat>& );    //! 车牌的尺寸验证    bool verifySizes(RotatedRect mr);    //! 结果车牌显示    Mat showResultMat(Mat src, Size rect_size, Point2f center);    //! 设置与读取变量    //...protected:    //! 高斯模糊所用变量    int m_GaussianBlurSize;    //! 连接操作所用变量    int m_MorphSizeWidth;    int m_MorphSizeHeight;    //! verifySize所用变量    float m_error;    float m_aspect;    int m_verifyMin;    int m_verifyMax;    //! 角度判断所用变量    int m_angle;    //! 是否开启调试模式，0关闭，非0开启    int m_debug;};

Note that all classes in easypr are declared in the namespace easypr, which is not listed here. The most important method in cplatelocate is the platelocate method. Its statement is as follows:

    //! 车牌定位    int plateLocate(Mat, vector<Mat>& );

The method has two parameters: the first parameter represents the Input Source image, and the second parameter is the output array, representing all the retrieved license plates. If the returned value is int type, 0 indicates success, and other values indicate failure. How is platelocate implemented internally? Let's take a closer look.

 

2."Plate locate"Implementation Process

The platelocate process is based on the processing process of the taotao1233 blog, but slightly different.

The overall recognition concept of platelocate is: If the license plate is not rotated or deformed, it must include many vertical edges (these vertical edges are often due to characters in the license plate ), if you can find a rectangle that contains many vertical edges, it is likely that it is a license plate.

Based on this idea, we can design a license plate Locating Process. After the design is completed, the optimization is performed based on the actual results. The following process is obtained after several adjustments and attempts. It contains an optimal process for the author to test the image set for several months (this process does not necessarily apply to all situations ). The implementation code of platelocate is not pasted here. Git has all the source code. The main processing flowchart of platelocate is as follows:

Figure 2 platelocate Flowchart

Next, we will refer to the above flowchart step by step to provide a temporary image in the middle of each step. You can set the following code in cplatelocate of version 1.01 to enable the debugging mode.

    CPlateLocate plate;    plate.setDebug(1);

Temporary images are generated in the TMP folder. Only temporary images of the last license plate image are retained.

1. original image.

 

2. image after Gaussian blur. After this step, we can see that the image is blurred. This step is used to remove interference noise for the next Sobel operator.

 

3. grayscale the image. This step is a watershed, meaning that all subsequent operations cannot be based on color information. This step has advantages and disadvantages and will be analyzed later.

4. Perform Sobel operations on the image to obtain the first-order horizontal derivative of the image. After this step, the license plate is clearly differentiated.

5. binarization the image. Converts a grayscale image (each pixel has 256 values) to a binary image (each pixel has only 1 and 0 values ).

6. Use the closed operation. After the image is closed, you can see that the license plate area is connected into a rectangular area.

 

7. Obtain the contour. Find all the outlines in the graph. This algorithm will calculate all the outlines of the entire graph, so it is necessary to filter.

8. filter. Find the smallest external rectangle for the contour, and then verify that the condition is not met. In this step, only the rectangles with six yellow borders are filtered.

8. Angle judgment and rotation. Discard a rectangle whose skew angle is greater than the threshold (such as plus or minus 30 degrees. The first, second, and fourth rectangles on the left are discarded. The remaining rectangle is rotated to make it horizontal.

10. uniform size. The size of the image block obtained in the previous step is different. Uniform dimensions are required to enter the machine learning model. The standard width of the uniform size is 136 and the length is 36. This standard is the general value obtained after the average of thousands of test licenses. It is the final three candidate "license plate" block.

 

These "license plates" have two functions: 1. accumulated as a training set for the Support Vector Machine (SVM) model to train a license plate judgment model; 2. In the actual license plate detection process, these candidate license plates are handed over to the trained license plate judgment model for judgment. If the model determines this is a license plate, the next step is the character recognition process. If not, discard it.

 

3."Plate locate"In-depth discussion and Optimization Strategies

Well, after talking about this, the reader must have a complete understanding of the entire "plate locate" process. Let's review every step in the process step by step. Answer the following three questions: What is the role of this step? Can I omit or replace this step? Is there any parameter in this step that can be optimized? Through these questions, we can better understand the license plate locating function and make it easier to modify and customize it by ourselves.

Due to the length of the article, the following in-depth discussions will be discussed in the next phase.

 

Copyright description:

All texts, images, and codes in this article are copyrighted by the author and the blog. You are welcome to reprint it, but be sure to indicate the author and reference source. Any unauthorized plagiarism and crawler crawling are infringing, and the author and blog garden reserve all rights.

 

References:

1. http://my.phirobot.com/blog/2014-02-opencv_configuration_in_vs.html

2. http://blog.csdn.net/jinshengtao/article/details/17883075

3. http://www.ruanyifeng.com/blog/2012/11/gaussian_blur.html

4. http://blog.csdn.net/xiaowei_cqu/article/details/7829481

 

Easypr-detailed description of Chinese Open Source License Plate Recognition System Development (2) License Plate positioning