Using OPENCV SVM and neural network to realize license plate recognition

First, preface

This article refers to the automatic license plate recognition project in the deep understanding of OPENCV practical Computer Vision Project analysis, and then carries on the practice after understanding the method. The difficulty of accurately identifying the sequence of the characters in the license plate area is to realize the exact location of the license plate area. So the final recognition effect is still to be further improved.

second, the procedure flow

The program flow is as follows:

The corresponding main function is as follows

#include "carid_detection.h" int main () {Mat img_input = Imread ("testcarid.jpg");
		If the read-in image fails if (Img_input.empty ()) {fprintf (stderr, "Can not load Image%s\n", "testcarid.jpg");
	return-1;
	} Mat hsvimg;
	Cvtcolor (IMG_INPUT,HSVIMG,CV_BGR2HSV);
	Vector<mat> planes;
	Split (Hsvimg,planes);
	Mat sImg;  SIMG = planes[1];   Get red component Blur (Simg,simg,size (3,3));
	3*3 Gaussian filter vector <RotatedRect> rects_simg;      	


	Posdetect (SIMG, rects_simg);
	Mat grayimg;
	Rgbconvtogray (Img_input, grayimg);   Medianblur (grayimg,grayimg,3);
	3*3 median filter vector <RotatedRect> rects_grayimg;

	Posdetect (grayimg, rects_grayimg);  Vector <RotatedRect> rects_closeimg;

	The license plate area is closer to Posdetect_closeimg (SIMG, rects_closeimg);
	Vector <RotatedRect> Rects_optimal;

	Optimposdetect (Rects_simg,rects_grayimg,rects_closeimg,rects_optimal);
	Vector <Mat> Output_area;  Normalposarea (Img_input, Rects_optimal,output_area); Obtain 144*33 's candidate license plate area Output_area CVSVM svmclassIfier;  Svm_train (Svmclassifier);   Using SVM to train the positive and negative samples vector<mat> PLATES_SVM; 
 		The candidate license plate area needs to be output_area each pixel in the image as a line eigenvector, and then predicted for (int i=0;i< output_area.size () ++i) {Mat img = output_area[i];
 		Mat p = img.reshape (a);
 		P.convertto (P,CV_32FC1);
 		int response = (int) svmclassifier.predict (p);    if (response = = 1) plates_svm.push_back (Output_area[i]);     Save prediction Result} if (Plates_svm.size ()! = 0) {imshow ("Test", plates_svm[0]);
	If you predict correctly, there is only one result plates_svm[0] Waitkey (0);
		} else {std::cout<< "failed to locate";
	return-1;
	}//The character area vector <Mat> Char_seg obtained from the SVM prediction in the license plate area; 

	Char_segment (PLATES_SVM[0],CHAR_SEG);
	Get the corresponding characteristic matrix vector <Mat> Char_feature of 7 character matrices;
	Char_feature.resize (7);

	for (int i =0;i<char_seg.size (); + + i) features (Char_seg[i], char_feature[i],5);
	Neural network training CVANN_MLP ann_classify;   Ann_train (Ann_classify, 34, 48);
	34 is the class number of the sample, 48 is the number of neurons in the hidden layer//character prediction vector<int> Char_result; Classify (ann_classify, Char_feature,char_result);

	This function waits for the key, press any key on the keyboard to return svmclassifier.clear ();
return 0; }

Three, Code introduction This article does not use the concept of class, purely process-oriented programming, the following are the main functions used

Carid_detection.h #include "opencv2/core/core.hpp" #include "opencv2//highgui/highgui.hpp" #include "opencv2/ Imgproc/imgproc.hpp "#include" opencv2/ml/ml.hpp "#include <time.h> #include <stdlib.h> #include <

Iostream> using namespace std;


using namespace CV; void Rgbconvtogray (const mat& Inputimage,mat & Outpuimage); RGB to Gray void Posdetect (Mat &, vector <RotatedRect> &);  Bold step selection of the candidate license plate region BOOL Verifysizes (const Rotatedrect &);  License plate area needs to meet the conditions void posdetect_closeimg (Mat &inputimage, vector <RotatedRect> & rects); Taking into account the situation when the license plate distance is very close, bool Verifysizes_closeimg (const Rotatedrect & candidate); Conditions required to meet the license plate area near the time void Optimposdetect (vector <RotatedRect> & rects_simg, Vector <RotatedRect> &am P RECTS_GRAYIMG,//license plate area further positioning vectors <RotatedRect> & rects_closeimg,vector <RotatedRect> & Rects_
optimal);  Float Caloverlap (const rect& box1,const rect& box2); Calculates the overlap ratio void for 2 matrices Normalposarea (Mat &intputimg, vector<rotatedrect> &rects_optimal, vectors <Mat>& output_area)  ;  Plate cutting, standardized for 144*33 void Svm_train (CVSVM &); Remove the feature matrix and label matrix from the Svm.xml to train void Char_segment (const Mat & Inputimg,vector <Mat>&);   Split the characters in the license plate area BOOL Char_verifysizes (const Rotatedrect &); The character area needs to meet the conditions of void Char_sort (vector <rotatedrect > & In_char);  Sort the character area void features (const MAT & In, Mat & out, int sizedata);             Get a character matrix corresponding to the feature vector Mat projecthistogram (const mat& img, int t); Calculates the horizontal or cumulative histogram, depending on whether T is 0 or 1 void ann_train (CVANN_MLP &ann, int numcharacters, int nlayers); Remove data from Ann_xml and perform neural network training void classify (cvann_mlp& ann, vector<mat> &char_feature, vector<int> & Char_result); Use neural network model to predict license plate characters and print to screen

Carid_detection.cpp #include "carid_detection.h" void Rgbconvtogray (const mat& Inputimage,mat & Outpuimage)  

	g = 0.3r+0.59g+0.11b {outpuimage = Mat (Inputimage.rows, Inputimage.cols, CV_8UC1); 
		for (int i = 0; i<inputimage.rows; + + i) {Uchar *ptrgray = outpuimage.ptr<uchar> (i);
		Const VEC3B * Ptrrgb = inputimage.ptr<vec3b> (i);	
		for (int j = 0; j<inputimage.cols; + + j) {Ptrgray[j] = 0.3*ptrrgb[j][2]+0.59*ptrrgb[j][1]+0.11*ptrrgb[j][0]; }}} void Posdetect_closeimg (Mat &inputimage, vector <RotatedRect> & rects)//Preliminary Find candidate area Rects {Ma
	T Img_canny;
	Canny (Inputimage, Img_canny, 150, 220);
	Mat Img_threshold; Threshold (Img_canny, img_threshold,0,255, cv_thresh_otsu+cv_thresh_binary);  The Otsu algorithm automatically obtains a threshold of Mat element = Getstructuringelement (Morph_rect, Size (15, 3));  Structural elements of closed morphology Morphologyex (Img_threshold, img_threshold,cv_mop_close,element);
	Morphological processing//looking for the contour of the license plate area vector< vector <Point> > contours; FiNdcontours (Img_threshold, contours,cv_retr_external, cv_chain_approx_none);//Only the outer contour is detected//the candidate contour is further screened vector<

	Vector <Point> >:: Iterator itc = Contours.begin (); while (ITC! = Contours.end ()) {Rotatedrect Mr = Minarearect (Mat (*ITC));//returns the smallest bounded rectangular region of each contour if (!verifysizes_closei
		MG (MR))//Determine if the rectangular contour meets the requirements {ITC = Contours.erase (ITC);
			} else {rects.push_back (MR);
		++ITC;
	}}} bool Verifysizes_closeimg (const Rotatedrect & candidate) {Float error = 0.4; Const float aspect = 44/14; aspect ratio int min = 100*aspect*100;  Minimum region int max = 180*aspect*180; Maximum zone float rmin = aspect-aspect*error; The minimum aspect ratio after considering the error is float Rmax = aspect + Aspect*error;
	The maximum aspect ratio after considering the error int area = Candidate.size.height * candidate.size.width;
	float R = (float) candidate.size.width/(float) candidate.size.height;

	if (r <1) r = 1/r; if (Area < min | | area > max) | |
		(r< rmin | | r > Rmax))
	return false;
else return true; } void Posdetect(Mat &inputimage, Vector <RotatedRect> & rects)//Preliminary Find candidate area Rects {mat Img_sobel;

	Sobel (Inputimage, Img_sobel, cv_8u, 1,0,3,1,0);
	Mat Img_threshold; Threshold (Img_sobel, img_threshold,0,255, cv_thresh_otsu+cv_thresh_binary);  The Otsu algorithm automatically obtains a threshold of Mat element = Getstructuringelement (Morph_rect, Size (15, 3));  

	Structural elements of closed morphology Morphologyex (Img_threshold, img_threshold,cv_mop_close,element);
	Look for the contour of the license plate area vector< vector <Point> > contours; Findcontours (Img_threshold, contours,cv_retr_external, cv_chain_approx_none);//Only the outer contour is detected//the candidate contour is further screened vector<

	Vector <Point> >:: Iterator itc = Contours.begin (); while (ITC! = Contours.end ()) {Rotatedrect Mr = Minarearect (Mat (*ITC));//returns the smallest bounded rectangular region of each contour if (!verifysizes (MR))//
		Determine if the rectangular outline meets the requirements {ITC = Contours.erase (ITC);
			} else {rects.push_back (MR);
		++ITC;
	}}} bool Verifysizes (const Rotatedrect & candidate) {Float error = 0.4; Const float aspect = 44/14; Length widththan int min = 20*aspect*20;  Minimum region int max = 180*aspect*180; Maximum zone float rmin = aspect-2*aspect*error; The minimum aspect ratio after considering the error is float Rmax = aspect + 2*aspect*error;
	The maximum aspect ratio after considering the error int area = Candidate.size.height * candidate.size.width;
	float R = (float) candidate.size.width/(float) candidate.size.height;

	if (r <1) r = 1/r; if (Area < min | | area > max) | | (r< rmin | | r > Rmax))
	It is considered that the candidate is the license plate region to return false when the condition is satisfied;
else return true; } void Optimposdetect (vector <RotatedRect> & rects_simg, Vector <RotatedRect> & rects_grayimg, Vect or <RotatedRect> & rects_closeimg,vector <RotatedRect> & Rects_optimal) {for (int i=0;i<rects_si Mg.size (); + + i) {for (int j=0;j<rects_grayimg.size (); ++j) {if (Caloverlap (Rects_simg[i].boundingrect (), R Ects_grayimg[j].boundingrect ()) > 0.2) {if (Rects_simg[i].boundingrect (). Width * Rects_simg[i].boundingrect (). He ight >= Rects_grayimg[j].boundingrect ().Width * rects_grayimg[j].boundingrect (). Height) rects_optimal.push_back (rects_simg[i]);	
			else Rects_optimal.push_back (Rects_grayimg[j]); }}} if (Rects_closeimg.size () <2)//Only one is considered, for speed {for (int i =0;i < rects_optimal.size (); + + i) for (in T J =0;j < Rects_closeimg.size (); + + j) {if (Caloverlap (Rects_optimal[i].boundingrect (), rects_closeimg [J].boundingrect ()) < 0.2 && Caloverlap (Rects_optimal[i].boundingrect (), Rects_closeimg[j].boundingrect (
				)) (> 0.05)) {Rects_optimal.push_back (rects_closeimg[j]); }}}} float Caloverlap (const rect& box1,const rect& box2) {if (box1.x > Box2.x+box2.width) {return 0 .0;
	} if (Box1.y > Box2.y+box2.height) {return 0.0;}
	if (Box1.x+box1.width < box2.x) {return 0.0;}
	if (Box1.y+box1.height < BOX2.Y) {return 0.0;}
	float colint = min (box1.x+box1.width,box2.x+box2.width)-Max (box1.x, box2.x); float rowint = min (Box1.y+box1.height,Box2.y+box2.height)-Max (BOX1.Y,BOX2.Y);
	float intersection = Colint * ROWINT;
	float area1 = box1.width*box1.height;
	float area2 = box2.width*box2.height;
return intersection/(area1 + area2-intersection); } void Normalposarea (Mat &intputimg, vector<rotatedrect> &rects_optimal, vectors <Mat>& output_
	Area) {float r,angle;
		for (int i = 0;i< rects_optimal.size (); ++i) {//rotation area angle = rects_optimal[i].angle;
		r = (float) rects_optimal[i].size.width/(float) (float) rects_optimal[i].size.height;
		if (r<1) angle = + angle;
		Mat Rotmat = getrotationmatrix2d (Rects_optimal[i].center, angle,1);//Get Morph Matrix object Mat img_rotated;

		Warpaffine (intputimg, Img_rotated,rotmat, Intputimg.size (), cv_inter_cubic);
		Crop image Size rect_size = rects_optimal[i].size;
		if (r<1) swap (rect_size.width, rect_size.height);
		Mat Img_crop;

		Getrectsubpix (img_rotated, Rect_size,rects_optimal[i].center, Img_crop); Adjusts all cropped images with a light histogram, making them the same width and height for training andCategory Mat resultresized;
		Resultresized.create (33,144,CV_8UC3);
		Resize (Img_crop, resultresized,resultresized.size (), 0,0,inter_cubic);
		Mat Grayresult;
		Rgbconvtogray (resultresized, Grayresult);
		Blur (Grayresult, Grayresult,size (3,3));

		Equalizehist (Grayresult,grayresult);
	Output_area.push_back (Grayresult);

	}} void Svm_train (CVSVM & svmclassifier) {Filestorage fs;
	Fs.open ("Svm.xml", Filestorage::read);
	Mat Svm_trainningdata;	

	Mat svm_classes;
	fs["Trainingdata" >>SVM_TrainningData;
	fs["Classes" >>SVM_Classes;
	Cvsvmparams Svm_params;

	Svm_params.kernel_type = Cvsvm::linear; Svmclassifier.train (Svm_trainningdata,svm_classes, Mat (), Mat (), svm_params);
SVM Training Model Fs.release ();
	} void Char_segment (const mat & inputimg,vector <Mat>& Dst_mat)//Get 20*20 of the standard character segmentation image {Mat img_threshold;
	Threshold (Inputimg, Img_threshold, 180,255, cv_thresh_binary);
	Mat img_contours;
	
	Img_threshold.copyto (img_contours); Vector < vector <point&Gt
	> contours;

	Findcontours (img_contours, Contours,cv_retr_external,cv_chain_approx_none);
	vector< vector <Point> >:: Iterator itc = Contours.begin ();
	
	Vector<rotatedrect> char_rects; while (ITC! = Contours.end ()) {Rotatedrect Minarea = Minarearect (Mat (*ITC));//returns the smallest bounded rectangular area of each profile point2f vertices[4
		];

  		Minarea.points (vertices);
		if (!char_verifysizes (Minarea))//Determine if the rectangular outline conforms to the requirements {ITC = Contours.erase (ITC); 
			} else {++ITC;  

		Char_rects.push_back (Minarea); }} char_sort (Char_rects);
	Sort the characters vector <Mat> Char_mat;

	for (int i = 0; I<char_rects.size (); ++i) {Char_mat.push_back (Mat (Img_threshold,char_rects[i].boundingrect ()));
	} Mat Train_mat (2,3,CV_32FC1);
	int length;
	Dst_mat.resize (7);  
	POINT2F Srctri[3];

	POINT2F Dsttri[3];  
		for (int i = 0; i< char_mat.size (); ++i) {srctri[0] = point2f (0,0);  
		SRCTRI[1] = point2f (char_mat[i].cols-1, 0); SRCTRI[2] = point2f (0, Char_mat[i].ROWS-1);
		Length = char_mat[i].rows > char_mat[i].cols?char_mat[i].rows:char_mat[i].cols;  
		Dsttri[0] = point2f (0.0, 0.0);  
		DSTTRI[1] = point2f (length, 0.0); 
		DSTTRI[2] = point2f (0.0, length);
		Train_mat = Getaffinetransform (Srctri, Dsttri);		
		Dst_mat[i]=mat::zeros (Length,length,char_mat[i].type ());
		Warpaffine (Char_mat[i],dst_mat[i],train_mat,dst_mat[i].size (), Inter_linear,border_constant,scalar (0));  Resize (dst_mat[i],dst_mat[i],size (20,20));
	Resize to 20*20}} bool Char_verifysizes (const Rotatedrect & candidate) {float aspect = 33.0f/20.0f; float Charaspect = (float) candidate.size.width/(float) candidate.size.height;
	Width-to-height ratio float error = 0.35;  float minheight = 11;  Min. height one float maxheight = 33;  Maximum height: float minaspect = 0.20;

	Considering the number 1, the minimum aspect ratio is 0.15 float Maxaspect = aspect + Aspect*error; if (Charaspect > Minaspect && charaspect < maxaspect && candidate.size.height >= minheight &amp ;& Candidate.size.width< maxheight)//Non 0 pixel points, aspect ratio, height required to meet the conditions return true;
else return false;
	} void Char_sort (vector <rotatedrect > & In_char)//Sort the character area {vector <rotatedrect > Out_char;           const int length = 7;
	7 characters int Index[length] = {0,1,2,3,4,5,6};
	float Centerx[length];
	for (int i=0;i < length; + + i) {centerx[i] = in_char[i].center.x; } for (int j=0;j <length;j++) {for (int i=length-2;i >= j;i--) if (Centerx[i] > Centerx[i+1]) {FL
				Oat T=centerx[i];
				CENTERX[I]=CENTERX[I+1];

				centerx[i+1]=t;
				int tt = Index[i];
				Index[i] = index[i+1];
			INDEX[I+1] = TT;

	}} for (int i=0;i<length; i++) Out_char.push_back (in_char[(Index[i])]);     In_char.clear (); Empty In_char In_char = Out_char; Reassign the sorted character area vector to In_char} void features (const MAT & In, Mat & out, int sizedata) {Mat vhist = projecthistogr AM (in, 1);  Horizontal histogram Mat hhist = Projecthistogram (in, 0);
	Vertical histogram Mat lowdata; Resize (in, Lowdata, SizE (Sizedata, sizedata));
	int numcols = vhist.cols + hhist.cols + lowdata.cols * lowdata.cols;

	out = Mat::zeros (1, Numcols, cv_32f);
	int j = 0;
		for (int i =0; i<vhist.cols; ++i) {out.at<float> (j) = vhist.at<float> (i);
	j + +;
	} for (int i=0; i < hhist.cols; ++i) {out.at<float> (j) = hhist.at<float> (i); } for (int x =0; x<lowdata.rows; ++x) {for (int y =0; y < lowdata.cols; + + y) {out.at<float> (j) = (f
			loat) lowdata.at<unsigned char> (x, y);
		j + +; }}} Mat projecthistogram (const mat& img, int t)//horizontal or vertical histogram, 0 per column statistic {/ /1 for Row statistics int sz = (t)?
	Img.rows:img.cols;

	Mat mhist = Mat::zeros (1, SZ, cv_32f);
		for (int j = 0; J < Sz; J + +) {Mat data = (t)? Img.row (j): Img.col (j);
	Mhist.at<float> (j) = Countnonzero (data);
	} double Min,max;

	Minmaxloc (Mhist, &min, &max);

	if (Max > 0) mhist.convertto (mhist, -1,1.0f/max, 0);
return mhist; } void Ann_train (CVANN_MLP &ann, int numcharacters, int nlayers) {Mat traindata, classes;
	Filestorage FS;
	
	Fs.open ("Ann_xml.xml", Filestorage::read);
	fs["Trainingdata" >>trainData;

	fs["Classes" >>classes;
	Mat layersizes (1,3,CV_32SC1);
	layersizes.at<int> (0) = Traindata.cols; Layersizes.at<int> (1) = nlayers; The number of hidden neurons can be set to 3 layersizes.at<int> (2) = Numcharacters;  The number of sample classes was Ann.create (layersizes, Cvann_mlp::sigmoid_sym, 1, 1);
	Initialize Ann Mat trainclasses;
	Trainclasses.create (Traindata.rows, Numcharacters, CV_32FC1); for (int i =0;i< traindata.rows; i++) {for (int k=0; k< trainclasses.cols; k++) {if (k = = (int) classes
			.at<uchar> (i)) {trainclasses.at<float> (i,k) = 1;			
		} else trainclasses.at<float> (i,k) = 0;
	}} Mat weights (1, traindata.rows, CV_32FC1, Scalar::all (1));
Ann.train (Traindata, trainclasses, weights); } void classify (cvann_mlp& ann, Vector<mat> &Amp;char_feature, Vector<int> & Char_result) {char_result.resize (Char_feature.size ()); for (int i=0;i<char_feature.size (); ++i) {Mat output (1, cv_32fc1),//1*34 matrix Ann.predict (Char_feature[i], out
		Put);
		Point Maxloc;
		Double maxval;
		Minmaxloc (output, 0, &maxval, 0, &maxloc);
	Char_result[i] = maxloc.x;
	} std::cout<< "the license plate after 6 bits for:"; Char s[] = {' 0 ', ' 1 ', ' 2 ', ' 3 ', ' 4 ', ' 5 ', ' 6 ', ' 7 ', ' 8 ', ' 9 ', ' A ', ' B ', ' C ', ' D ', ' E ', ' F ', ' G ', ' H ', ' J ', ' K ', ' L ', ' M ', ' N ', ' P ', ' Q ', '
	R ', ' S ', ' T ', ' U ', ' V ', ' W ', ' X ', ' Y ', ' Z '};
	for (int i=1;i<char_result.size (); ++i)//First digit is Chinese character, here does not realize the prediction of Chinese characters {std::cout<<s[char_result[i]];
} std::cout<< ' \ n '; }

Iv. about Svm.xml and Ann_xml.xml

Svm.xml is a training matrix and class matrix data for SVM training, labeled "Trainingdata" corresponding to the training matrix, for 195*4752 size, 195 for 195 training samples, and 4752 for each sample's eigenvector dimension, Because each image size is 144*33, turn it into a row, and each pixel value as a feature value, you can get 4,752 eigenvalues, saved as 1 rows, the sample image used has 75 positive samples and 120 negative samples, save as follows:

The label is "classes" corresponding to the category matrix, for the 195*1 matrix, the first 75 values correspond to a positive sample of 1.0, the last 120 values corresponding to a negative sample of-1.0.

Similarly, Ann_xml.xml is used for neural network training data, labeled "Trainingdata" corresponding to the training matrix, for 1700*65 size, 1700 is the number of training samples, because the license plate characters have 34 classes, each class has 50, so the total 1700, 65 the feature vectors extracted for each sample. The label is "classes" corresponding to the category matrix, for the 1700*1 size, the number of negative samples is labeled, the training sample is taken from

v. Results and Analysis

Reluctantly found 2 pictures, you can fully identify the license plate after the 6 character, the effect is as follows.

So the performance of this system is still to be improved, but I think there can be some improvement in the license plate location, the use of other better license plate location algorithm will be better, and can increase the neural network training samples.

Six, complete code file download

Download Link: http://download.csdn.net/detail/ap1005834/9513328