Using OPENCV SVM and neural network to realize license plate recognition

First, preface

This article refers to the automatic license plate recognition project in the deep understanding of OPENCV practical Computer Vision Project analysis, and then carries on the practice after understanding the method. The difficulty of accurately identifying the sequence of the characters in the license plate area is to realize the exact location of the license plate area. So the final recognition effect is still to be further improved.

Second, the procedure flow

The program flow is as follows:

The corresponding main function is as follows

#include "carid_detection.h" int main () {Mat img_input = Imread ("testcarid.jpg");//If the read-in image fails if (Img_input.empty ()) { fprintf (stderr, "Can not load Image%s\n", "testcarid.jpg"); return-1;} Mat hsvimg; Cvtcolor (IMG_INPUT,HSVIMG,CV_BGR2HSV);vector<mat> planes;split (hsvimg,planes);  Mat simg;simg = planes[1];   Get red component Blur (Simg,simg,size (3,3));      3*3 Gaussian filter vector <RotatedRect> rects_simg;posdetect (sImg, rects_simg); Mat grayimg;   Rgbconvtogray (Img_input, grayimg); Medianblur (grayimg,grayimg,3);  3*3 median filter vector <RotatedRect> rects_grayimg;posdetect (grayimg, rects_grayimg); Vector <RotatedRect>  rects_closeimg; The license plate area is closer to Posdetect_closeimg (SIMG, rects_closeimg); vector <RotatedRect> Rects_optimal;optimposdetect (rects_ simg,rects_grayimg,rects_closeimg,rects_optimal); vector <Mat> Output_area;normalposarea (img_input, Rects_  Optimal,output_area); Obtain 144*33 's candidate license plate area OUTPUT_AREACVSVM Svmclassifier;  Svm_train (Svmclassifier); Training of positive and negative samples using SVM vector<mat> pLATES_SVM; The candidate license plate area needs to be output_area each pixel in the image as a line eigenvector, and then predicted for (int i=0;i< output_area.size () ++i) {Mat img = output_area[i]; Mat p = img.reshape (a); P.convertto (P,CV_32FC1); int response = (int) svmclassifier.predict (p);    if (response = = 1) plates_svm.push_back (Output_area[i]);     Save Prediction Results}if (plates_svm.size ()! = 0) {imshow ("Test", plates_svm[0]); If you predict correctly, there is only one result Plates_svm[0]waitkey (0);} else{std::cout<< "Location failed"; return-1;} Vector <Mat> Char_seg;char_segment (plates_svm[0],char_seg) in the area of the license plate obtained from SVM prediction; Get the corresponding feature matrix vector <Mat> char_feature;char_feature.resize (7) for 7 character matrices; for (int i =0;i<char_seg.size (); + + i)   Features (Char_seg[i], char_feature[i],5);//neural Network training CVANN_MLP Ann_classify;ann_train (Ann_classify, 34, 48); 34 is the number of classes for the sample, 48 is the number of neurons in the hidden layer//character prediction vector<int> char_result;classify (Ann_classify,char_feature,char_result); This function waits for the key and presses any key on the keyboard to return to Svmclassifier.clear (); returns 0;}

Three, the code introduction

This article does not use the concept of class, purely process-oriented programming, the following are the main functions used

Carid_detection.h#include "opencv2/core/core.hpp" #include "opencv2//highgui/highgui.hpp" #include "opencv2/ Imgproc/imgproc.hpp "#include" opencv2/ml/ml.hpp "#include <time.h> #include <stdlib.h> #include < Iostream>using namespace std;using namespace cv;void rgbconvtogray (const mat& Inputimage,mat & Outpuimage); RGB to Gray void Posdetect (Mat &, vector <RotatedRect> &);  Bold step selection of the candidate license plate region BOOL Verifysizes (const Rotatedrect &);  License plate area needs to meet the conditions void posdetect_closeimg (Mat &inputimage, vector <RotatedRect> & rects); Taking into account the situation when the license plate distance is very close, bool Verifysizes_closeimg (const Rotatedrect & candidate); Conditions required to meet the license plate area near the time void Optimposdetect (vector <RotatedRect> & rects_simg, Vector <RotatedRect> & RECTS_GRAYIMG,//license plate area further positioning vectors <RotatedRect> & rects_closeimg,vector <RotatedRect> & Rects_  optimal); float caloverlap (const rect& box1,const rect& box2); Calculates the overlap ratio of 2 matrices void Normalposarea (Mat &intpuTIMG, vector<rotatedrect> &rects_optimal, Vector <Mat>& Output_area);  Plate cutting, standardized for 144*33void Svm_train (CVSVM &); Remove the feature matrix and label matrix from the Svm.xml to train void Char_segment (const Mat & Inputimg,vector <Mat>&);   Split the characters in the license plate area BOOL Char_verifysizes (const Rotatedrect &); The character area needs to meet the conditions of void Char_sort (vector <rotatedrect > & In_char);  Sort the character area void features (const MAT & In, Mat & out, int sizedata);             Get a character matrix corresponding to the feature vector Mat projecthistogram (const mat& img, int t); Calculates the horizontal or cumulative histogram, depending on whether T is 0 or 1void ann_train (CVANN_MLP &ann, int numcharacters, int nlayers); Remove data from Ann_xml and perform neural network training void classify (cvann_mlp& ann, vector<mat> &char_feature, vector<int> & Char_result); Use neural network model to predict license plate characters and print to screen

Carid_detection.cpp#include "Carid_detection.h" void Rgbconvtogray (const mat& Inputimage,mat & OutpuImage)/  /g = 0.3r+0.59g+0.11b{outpuimage = Mat (Inputimage.rows, Inputimage.cols, CV_8UC1); for (int i = 0; i<inputimage.rows; + + i) {Uchar *ptrgray = outpuimage.ptr<uchar> (i); Const VEC3B * Ptrrgb = Inputi Mage.ptr<vec3b> (i); for (int j = 0; j<inputimage.cols; + + j) {Ptrgray[j] = 0.3*ptrrgb[j][2]+0.59*ptrrgb[j][1]+ 0.11*ptrrgb[j][0];}}} void Posdetect_closeimg (Mat &inputimage, vector <RotatedRect> & rects)//Preliminary Find candidate Area Rects{mat img_canny;c Anny (Inputimage, Img_canny, 150, 220); Mat Img_threshold;threshold (Img_canny, img_threshold,0,255, cv_thresh_otsu+cv_thresh_binary);  The Otsu algorithm automatically obtains a threshold of mat element = Getstructuringelement (Morph_rect, Size (15, 3));  Structural elements of closed morphology Morphologyex (Img_threshold, img_threshold,cv_mop_close,element); Morphological processing//looking for the contour of the license plate area vector< Vector <Point> > contours;findcontours (img_threshold, Contours,cv_retr_ EXTERNAL, Cv_chain_Approx_none);//detection of contour only//For further screening of candidate profiles vector< vector <Point> >:: Iterator itc = Contours.begin (); while (ITC!) = Contours.end ()) {Rotatedrect Mr = Minarearect (Mat (*ITC));//returns the smallest bounded rectangular region of each contour if (!verifysizes_closeimg (MR))//determines if the rectangle outline Compliance Requirements {ITC = Contours.erase (ITC);}      else {rects.push_back (MR); ++ITC;} }}bool verifysizes_closeimg (const Rotatedrect & candidate) {Float error = 0.4;const float aspect = 44/14;//aspect ratio int min = 100*aspect*100;  Minimum region int max = 180*aspect*180; Maximum zone float rmin = aspect-aspect*error; The minimum aspect ratio after considering the error is float Rmax = aspect + Aspect*error; Maximum aspect ratio after considering error int area = Candidate.size.height * Candidate.size.width;float r = (float) candidate.size.width/(float) Candidate.size.height;if (r <1) R = 1/r;if (Area < min | | area > max) | | (r< rmin | | r > Rmax)) return False;elsereturn true;} void Posdetect (Mat &inputimage, vector <RotatedRect> & rects)//Preliminary Find candidate area Rects{mat Img_sobel; Sobel (Inputimage, Img_sobel, cv_8u, 1,0,3,1,0); Mat IMg_threshold;threshold (Img_sobel, img_threshold,0,255, cv_thresh_otsu+cv_thresh_binary);  The Otsu algorithm automatically obtains a threshold of mat element = Getstructuringelement (Morph_rect, Size (15, 3));  Structural elements of closed morphology Morphologyex (Img_threshold, img_threshold,cv_mop_close,element); Look for the contour of the license plate area vector< Vector <Point> > contours;findcontours (img_threshold, contours,cv_retr_external, cv_ Chain_approx_none);//detection of contour only//For further screening of candidate profiles vector< vector <Point> >:: Iterator itc = Contours.begin (); while (ITC! = Contours.end ()) {Rotatedrect Mr = Minarearect (Mat (*ITC));//returns the smallest bounded rectangular region of each contour if (!verifysizes (MR))//The judging rectangle outline is No compliance requirements {ITC = Contours.erase (ITC);}      else {rects.push_back (MR); ++ITC;} }}bool verifysizes (const Rotatedrect & candidate) {Float error = 0.4;const float aspect = 44/14;//aspect ratio int min = 20*aspe ct*20;  Minimum region int max = 180*aspect*180; Maximum zone float rmin = aspect-2*aspect*error; The minimum aspect ratio after considering the error is float Rmax = aspect + 2*aspect*error; The maximum aspect ratio after considering the error int area = Candidate.size.height * Candidate.size.width;float r = (float) candidate.size.width/(float) candidate.size.height;if (r <1) R = 1/r;if (Area < min | | area > max) | | (r< rmin | | r > Rmax)) The condition is satisfied that the candidate is the license plate area return False;elsereturn true;} void Optimposdetect (vector <RotatedRect> & rects_simg, vector <RotatedRect> & Rects_grayimg,vector <RotatedRect> & Rects_closeimg,vector <RotatedRect> & Rects_optimal) {for (int i=0;i<rects_ Simg.size (); + + i) {for (int j=0;j<rects_grayimg.size (); ++j) {if (Caloverlap (Rects_simg[i].boundingrect (), Rects_ Grayimg[j].boundingrect ()) > 0.2) {if (Rects_simg[i].boundingrect (). Width * Rects_simg[i].boundingrect (). Height >= rects_grayimg[j].boundingrect (). Width * Rects_grayimg[j].boundingrect (). Height) Rects_optimal.push_back ( Rects_simg[i]); Elserects_optimal.push_back (Rects_grayimg[j]);}} if (Rects_closeimg.size () <2)//Consider only one, for speed {for (int i =0;i < rects_optimal.size (); + + i) for (Int J =0;j < Rects_ Closeimg.size (); + + j) {if (calOverlap (Rects_optimal[i].boundingrect (), Rects_closeimg[j].boundingrect ()) < 0.2 &&caloverlap (rects_ Optimal[i].boundingrect (), Rects_closeimg[j].boundingrect ()) > 0.05)) {Rects_optimal.push_back (rects_closeImg[j ]);}}}} Float Caloverlap (const rect& box1,const rect& box2) {if (box1.x > Box2.x+box2.width) {return 0.0;} if (Box1.y > Box2.y+box2.height) {return 0.0;} if (Box1.x+box1.width < box2.x) {return 0.0;} if (Box1.y+box1.height < BOX2.Y) {return 0.0;} float colint = min (box1.x+box1.width,box2.x+box2.width)-Max (box1.x, box2.x); float rowint = min (box1.y+box1.height,box 2.y+box2.height)-Max (BOX1.Y,BOX2.Y), float intersection = Colint * Rowint;float area1 = Box1.width*box1.height;float is A2 = Box2.width*box2.height;return intersection/(area1 + area2-intersection);} void Normalposarea (Mat &intputimg, vector<rotatedrect> &rects_optimal, vectors <Mat>& output_ Area) {float r,angle;for (int i = 0;i< rects_optimal.size (); ++i) {//rotation area angle = Rects_optimal[i].angle;r = (float) rects_optimal[i].size.width/(float) (float) rects_optimal[i]. Size.height;if (r<1) angle = + angle; Mat Rotmat = getrotationmatrix2d (Rects_optimal[i].center, angle,1);//Get Morph Matrix object Mat Img_rotated;warpaffine (INTPUTIMG, Img_rotated,rotmat, Intputimg.size (), cv_inter_cubic);//crop image size Rect_size = rects_optimal[i].size;if (r<1) Swap ( Rect_size.width, Rect_size.height); Mat Img_crop;getrectsubpix (img_rotated, Rect_size,rects_optimal[i].center, img_crop);//adjust all cropped images with a light histogram, Made with the same width and height, suitable for training and classification Mat resultresized;resultresized.create (33,144,CV_8UC3); resize (Img_crop, resultresized, Resultresized.size (), 0,0,inter_cubic); Mat Grayresult; Rgbconvtogray (resultresized, Grayresult);//blur (Grayresult, Grayresult,size (3,3)); Equalizehist (GrayResult, Grayresult); Output_area.push_back (Grayresult);}} void Svm_train (CVSVM & svmclassifier) {filestorage fs;fs.open ("Svm.xml", Filestorage::read); Mat Svm_trainningdata; Mat svm_classes;fs["Trainingdata"); >svm_trainningdata;fs["Classes" >>SVM_Classes; Cvsvmparams Svm_params; Svm_params.kernel_type = Cvsvm::linear;svmclassifier.train (Svm_trainningdata,svm_classes, Mat (), Mat (), SVM_params) ; SVM Training Model Fs.release ();} void Char_segment (const mat & inputimg,vector <Mat>& Dst_mat)//Get 20*20 of the standard character segmentation image {Mat img_threshold; Threshold (Inputimg, Img_threshold, 180,255, cv_thresh_binary); Mat Img_contours;img_threshold.copyto (img_contours), vector < vector <Point> > contours;findcontours (img_ Contours, contours,cv_retr_external,cv_chain_approx_none);vector< vector <Point> >:: Iterator itc = Contours.begin ();vector<rotatedrect> char_rects;while (itc! = Contours.end ()) {Rotatedrect MinArea = MinAreaRect (Mat (*ITC));  Returns the smallest bounded rectangular region of each contour point2f vertices[4];minarea.points (vertices); if (!char_verifysizes (Minarea))//Determine if the rectangular outline conforms to the requirements {ITC = Contours.erase (ITC);}  else {++itc; Char_rects.push_back (Minarea); }}char_sort (char_rects); Sort the characters vector <Mat> Char_mat;for (int i = 0; I<char_rects.size (); ++i) {Char_mat.push_back (Mat (Img_threshold,char_rects[i].boundingrect ( ))) ;} Mat Train_mat (2,3,CV_32FC1); int length;d st_mat.resize (7);  POINT2F Srctri[3];  POINT2F dsttri[3];for (int i = 0; i< char_mat.size (); ++i) {srctri[0] = point2f (0,0);  SRCTRI[1] = point2f (char_mat[i].cols-1, 0); SRCTRI[2] = point2f (0, char_mat[i].rows-1); length = char_mat[i].rows > char_mat[i].cols?char_mat[i].rows:char_mat[  I].cols;dsttri[0] = point2f (0.0, 0.0);  DSTTRI[1] = point2f (length, 0.0); DSTTRI[2] = point2f (0.0, length); Train_mat = Getaffinetransform (Srctri, Dsttri);d St_mat[i]=mat::zeros (Length,length,char_mat[i].type ()); warpAffine (Char_mat[i],dst_mat[i],train_mat,dst_mat[i].size (), Inter_linear,border_constant,scalar (0)); Resize (Dst_mat[i],  Dst_mat[i],size (20,20)); Size adjustment to 20*20}}bool char_verifysizes (const Rotatedrect & candidate) {float aspect = 33.0f/20.0f;float Charaspect = ( float) candidate.size.width/(float) Candidate.size.heiGht  Width-to-height ratio float error = 0.35;float minheight = 11;  Minimum height 11float maxheight = 33;  Maximum height 33float minaspect = 0.20; Considering the number 1, the minimum aspect ratio is 0.15float Maxaspect = aspect + aspect*error;if (Charaspect > Minaspect && charaspect < maxasp ect&& candidate.size.height >= minheight && candidate.size.width< maxheight)//Non 0 pixel points, aspect ratio, The height must meet the condition return True;elsereturn false;} void Char_sort (vector <rotatedrect > & In_char)//Sort the character area {vector <rotatedrect > out_char;const int len           Gth = 7; 7 characters int Index[length] = {0,1,2,3,4,5,6};float centerx[length];for (int i=0;i < length; + + i) {centerx[i] = In_char[i] . center.x;} for (int j=0;j <length;j++) {for (int i=length-2;i >= j;i--) if (Centerx[i] > centerx[i+1]) {float T=centerx[i];cen Terx[i]=centerx[i+1];centerx[i+1]=t;int TT = index[i];index[i] = index[i+1];index[i+1] = TT;}}     for (int i=0;i<length; i++) Out_char.push_back (in_char[(Index[i])); In_char.clear (); Empty In_charin_char = Out_char; //Re-assigns the sorted character area vector to In_char}void features (const Mat & In, Mat & out, int sizedata) {Mat vhist = Projecthistogram (In, 1);  Horizontal histogram Mat hhist = Projecthistogram (in, 0); Vertical Histogram mat lowdata;resize (in, Lowdata, Size (Sizedata, sizedata)); int numcols = Vhist.cols + hhist.cols + lowdata.cols * Lowdata.cols;out = Mat::zeros (1, Numcols, cv_32f); int j = 0;for (int i =0; i<vhist.cols; ++i) {out.at<float> (j) = Vhist.at<float> (i); j + +;} for (int i=0; i < hhist.cols; ++i) {out.at<float> (j) = hhist.at<float> (i);} for (int x =0; x<lowdata.rows; ++x) {for (int y =0; y < lowdata.cols; + + y) {out.at<float> (j) = (float) lowdata.at <unsigned char> (x, y); j + +;}}  Mat projecthistogram (const mat& img, int t)//horizontal or vertical histogram, 0 for column statistics {//1 for row statistics int SZ = (t)? Img.rows:img.cols; Mat mhist = Mat::zeros (1, SZ, cv_32f); for (int j = 0; J < Sz; J + +) {Mat data = (t)? Img.row (j): Img.col (j); Mhist.at<flo At> (j) = Countnonzero (data);}Double Min,max;minmaxloc (mhist, &min, &max), if (Max > 0) mhist.convertto (mhist, -1,1.0f/max, 0); return mhist;} void Ann_train (CVANN_MLP &ann, int numcharacters, int nlayers) {Mat traindata, classes; Filestorage Fs;fs.open ("Ann_xml.xml", Filestorage::read), fs["Trainingdata"] >>traindata;fs["classes"] > >classes; Mat layersizes (1,3,CV_32SC1);layersizes.at<int> (0) = traindata.cols;layersizes.at<int> (1) = nlayers; The number of hidden neurons can be set to 3layersizes.at<int> (2) = Numcharacters;  The number of sample classes is 34ann.create (layersizes, Cvann_mlp::sigmoid_sym, 1, 1); Initialize Annmat trainclasses;trainclasses.create (traindata.rows, Numcharacters, CV_32FC1); for (int i =0;i< Traindata.rows; i++) {for (int k=0; k< trainclasses.cols; k++) {if (k = = (int) classes.at<uchar> (i)) {TRAINCLASSES.AT&LT;FLOAT&G t; (i,k) = 1;} Elsetrainclasses.at<float> (i,k) = 0;}} Mat weights (1, traindata.rows, CV_32FC1, Scalar::all (1)); Ann.train (Traindata, trainclasses, weights);} void ClassiFY (cvann_mlp& ann, vector<mat> &char_feature, vector<int> & Char_result) {char_result.resize ( Char_feature.size ()); for (int i=0;i<char_feature.size (); ++i) {Mat output (1, cv_32fc1),//1*34 matrix Ann.predict ( Char_feature[i], output); Point maxloc;double maxval;minmaxloc (output, 0, &maxval, 0, &maxloc); char_result[i] = maxloc.x;} std::cout<< "the license plate after 6 bits:"; char s[] = {' 0 ', ' 1 ', ' 2 ', ' 3 ', ' 4 ', ' 5 ', ' 6 ', ' 7 ', ' 8 ', ' 9 ', ' A ', ' B ', ' C ', ' D ', ' E ', ' F ', ' G ', ' H ', ' J ', ' K ', ' L ', ' M ', ' N ', ' P ', ' Q ', ' R ', ' S ', ' T ', ' U ', ' V ', ' W ', ' X ', ' Y ', ' Z '};for (int i=1;i<char_result.size (); ++i)// The first is the kanji, there is no prediction of the Chinese characters {std::cout<<s[char_result[i]];} std::cout<< ' \ n ';}

Iv. about Svm.xml and Ann_xml.xml

Svm.xml is a training matrix and class matrix data for SVM training, labeled "Trainingdata" corresponding to the training matrix, for 195*4752 size, 195 for 195 training samples, and 4752 for each sample's eigenvector dimension, Because each image size is 144*33, turn it into a row, and each pixel value as a feature value, you can get 4,752 eigenvalues, saved as 1 rows, the sample image used has 75 positive samples and 120 negative samples, save as follows:

The label is "classes" corresponding to the category matrix, for the 195*1 matrix, the first 75 values correspond to a positive sample of 1.0, the last 120 values corresponding to a negative sample of-1.0.

Similarly, Ann_xml.xml is used for neural network training data, labeled "Trainingdata" corresponding to the training matrix, for 1700*65 size, 1700 is the number of training samples, because the license plate characters have 34 classes, each class has 50, so the total 1700, 65 the feature vectors extracted for each sample. The label is "classes" corresponding to the category matrix, for the 1700*1 size, the number of negative samples is labeled, the training sample is taken from

V. Results and Analysis

Reluctantly found 2 pictures, you can fully identify the license plate after the 6 character, the effect is as follows.

So the performance of this system is still to be improved, but I think there can be some improvement in the license plate location, the use of other better license plate location algorithm will be better, and can increase the neural network training samples.

Six, complete code file download

Download Link: http://download.csdn.net/detail/ap1005834/9513328

Using OPENCV SVM and neural network to realize license plate recognition