Sort out the License Plate Recognition Process Using SVM and neural networks in Chapter 5th mastering opencv with practical computer vision Projects

ANPR (automatic number plate recognition) is divided in two main steps: plate detection and plate recognition. Plate detection has the purpose of detecting the location of the plate in
Whole camera frame. When a plate is detected in an image, the plate segment is passed to the second step -- Plate Recognition -- which uses an OCR algorithm to determine the alphanumeric characters on the plate.

I. Plate Detection

1. Segmentation: it is the process of dividing an image into multiple segments.

One important feature of Plate segmentation is the high number of vertical edges in a license plate assuming that the image was taken frontally, and the plate is not rotated and is
Perspective distortion.

Before finding vertical edges, we need to convert the color image to a grayscale image, (because color can't help us in this task), and remove possible noise generated by the camera or other
Ambient noise. we will apply a Gaussian blur of 5x5 and remove noise. if we don't apply a noise-removal method, we can get a lot of vertical edges that produce a falied detection.

To find the vertical edges, we will use a Sobel filter and find the First horizontal derivative. The derivative is a mathematical function that allows us to find the vertical edgeson an image.

After a Sobel filter, we apply a threshold filter to obtain a binary image with a threshold value obtained through Otsu's method.

By applying a close morphological operation, we can remove blank spaces between each vertical edge line, and connect all regions that have a high number of edges. In this step we have
Possible regions that can contain in plates.

After applying these functions, we have regions in the image that coshould contain a plate; however, most of the regions will not contain license plates. These regions can be split with a connected-component
Analysis or by using the findcontours function. this last function retrieves the amount s of a binary image with different methods and results. we only need to get the external locale s with any hierarchical relationship and any polygonal approximation results.
For each contour detected, extract the bounding rectangle of minimal area. We make basic validations about the regions detected based on its area and aspect ratio.

We can make more improvements using the license plate's white background property. All plates have the same background color and we can use a flood fill algorithm to retrieve the rotated rectangle
For precise cropping. the first step to crop the license plate is to get several seeds near the last rotated rectangle center. then get the minimum size of plate between the width and height, and use it to generate random seeds near the patch center. we want
To select the white region and we need several seeds to touch at least one white pixel. then for each seed, we use a floodfill function to draw a new mask image to store the new closest cropping region. once we have a crop mask, we get a minimal area rectangle
From the image-mask points and check the valid size again.

2. Now that the segmentation process is finished and we have valid regions, we can crop each detected region, remove any possible rotation, crop the image region, resize the image, and equalize
The light of cropped image regions.

First, we need to generate the transform matrix with getrotationmatrix2d to remove possible rotations in the detected region.

After we rotate the image, we crop the image with getrectsubpix, which crops and copies an image portion of given width and height centered in a point. If the image was rotated, we need
Change the width and height sizes with the c ++ swap function.

Cropped images are not good for use in training and classification since they do not have the same size. Also, each image contains different light conditions, increasing their relative differences.
To resolve this, we resize all images to the same width and height and apply light histogram equalization.

3. Classification: After we preprocess and segment all possible parts of an image, we now need to decide if each segmentis (or is not) a license plate. to do this, we will use a Support Vector
Machine (SVM) algorithm.

We need to train the algorithm with an amount of data that is labeled; each data set needs to have a class. We trained our system with 75 license-plate images and 35 images without license
Plates of 144x33 pixels. In a real application, we wocould need to train with more data.

We need to set the SVM parameters that define the basic parameters to use in an SVM algorithm; we will use the cvsvmparams structure to define it. It is a mapping done to the training data
To improve its resesponance to a linearly separable set of data. this mapping consists of increasing the dimensionality of the data and is done efficiently using a kernel function. we choose here the cvsvm: Linear types which means that no mapping is done.
We then create and train our classifier. opencv defines the cvsvm class for the Support Vector Machine algorithm and we initialize it with the training data, classes, and parameter data.

Our classifier is ready to predict a possible cropped image using the predict function of our SVM class; this function returns the Class Identifier I. In our case, we label a plate class
1 and no plate class with 0. Then for each detected region that can be a plate, we use SVM to classify it as a plate or no plate, and save only the correct responses.

2. Plate Recognition

The second step in License Plate Recognition aims to retrieve the characters of the license plate with optical character recognition. For each detected plate, we proceed to segment the plate
For each character, and use an artificial neural network (ANN) machine-Learning Algorithm to recognize the character.

1. OCR Segmentation

First, we obtain a plate image patch as the input to the segmentation OCR function with an equalized histogram, we then need to apply a threshold filter and use this threshold image as
Input of a find algorithm s algorithm.

We use the cv_thresh_binary_inv parameter to invert the threshold output by turning the white input values black and black input values white. This is needed to get the resulting s of each character,
Because the parameter S algorithm looks for white pixels.

For each detected contour, we can make a size verification and remove all regions where the size is smaller or the aspect is not correct. In our case, the characters have a 45/77 aspect, and
We can accept a 35 percent error of aspect for rotated or distorted characters. if an area is higher than 80 percent, we consider that region to be a Black Block, and not a character. for counting the area, we can use the countnonzero function that counts
The number of pixels with a value higher than 0.

If a segmented character is verified, we have to preprocess it to set the same size and position for all characters and save it in a vector with the auxiliary charsegment class. This class
Saves the segmented character image and the position that we need to order the characters because the find contour algorithm does not return the seconds in the required order.

2. Feature Extraction

The next step for each segmented character is to extract the features for training and classifying the artificial neural network algorithm.

Unlike the plate detection feature-extraction step that is used in SVM, we don't use all of the image pixels; we will apply more common features used in optical character recognition containing
Horizontal and vertical accumulation histograms and a low-resolution image sample. Each image has a low-resolution 5x5 and the histogram accumulations.

For each character, we count the number of pixels in a row or column with a nonzero value using the countnonzero function and store it in a new data matrix called mhist. We normalize it
Looking for the maximum value in the data matrix using the minmaxloc function and divide all elements of mhist by the maximum value with the convertion function. we create the projectedhistogram function to create the accumulation histograms that have as input
A binary image and the type of histogram we need-horizontalor vertical.

Other features use a low-resolution sample image. instead of using the whole character image, we create a low-resolution character, for example 5x5. we train the system with 5x5, 10 x
10, 15x15, and 20x20 characters, and then evaluate which one returns the best result so that we can use it in our system. once we have all the features, we create a matrix
MColumns by one row where the columns are the features.

3. OCR Classification

In the classification step, we use an artificial neural network machine-learning algorithm. More specifically, a multi-layer perceptron (MLP), which is the most commonly used ANN algorithm.
MLP consists of a network of neurons with an input layer, output layer, and one or more hidden layers. Each layer has one or more neurons connected with the previous and next layer.

An Ann-trained network has a vector of input with features. It passes the values to the hidden layer and computes the results with the weights and activation function. It passes outputs further
Downstream until it gets the output layer that has the number of neuron classes.

The weight of each layer, synapses, and neuron is computed and learned by training the ANN algorithm. To train our classifier, we create two matrices of data as we did in
SVM training, but the training labels are a bit different. Instead ofN
X 1 matrix whereNStands for training data rows and 1 is the column, we use the label number identifier. We have to create
NXMMatrix whereNIs the training/samples data and
MIs the classes (10 digits + 20 letters in our case), and set 1 in a position (I,J) If the data row
IIs classified with ClassJ.

We create an OCR: Train function to create all the needed matrices and train our system, with the training data matrix, classes matrix, and the number of hidden neurons in the hidden layers.
The training data is loaded from an XML file just as we did for the SVM training. we have to define the number of neurons in each layer to initialize the ANN class. for our sample, we only use one hidden layer, then we define a matrix of 1 row and 3 columns.
The first column position is the number of features, the second column position is the number of hidden neurons in the hidden layer, and the third column position is the number of classes. opencv defines a cvann_mlp class for Ann. with the create function,
We can initiate the class by defining the number of layers and neurons, the activation function, and the alpha and beta parameters.

After training, we can classify any segmented plate feature using the OCR: classify function.

The cvann_mlp class uses the predict function for classifying a feature vector in a class. Unlike the SVM classify function, the Ann's predict function returns a row with the size equal
The number of classes with the probability of belonging to the input feature of each class.