OpenCV Java Implementation of notes, paper quadrilateral edge detection and extraction, pendulum

Reference Link: http://blog.csdn.net/zxw_xzr/article/details/77358815

The internship company has the need for OCR recognition of VAT invoices. The OCR section is not difficult to implement (there is a ready-made SDK to call), but in practice, the invoices in the user-supplied photos tend to have some skew, and the company-provided OCR SDK does not detect skewed characters, so the image preprocessing is required first, and the positive invoice (the effect is similar to Office Lens). The effects to be achieved are as follows:

The specific steps of the algorithm are as follows:

1. Turn grayscale, noise reduction
2. Edge detection
3. Contour Extraction
4. Find convex hull, fit polygon
5. Find the largest square
6. Perform step 3 again to improve accuracy
7. Find the rectangular four edges, which is the outer quadrilateral of the paper
8. Perspective transformation, extracting quadrilateral

Paper quadrilateral detection and extraction of the tutorial online less, but also not enough detail, this is my writing this blog power. Next I'll step through the algorithm in detail:

1, turn gray, noise reduction

The first step is to preprocess the image. In order to apply the canny algorithm, the image is first converted to grayscale. Because of the edge detection, we must pre-empt the noise, and try the Gaussian filtering and meanshift filtering in the noise reduction algorithm. The effect of Meanshift filter is better than Gaussian filter, can be the desktop texture, invoice the characters and other redundant information to smear out, but because the Meanshift clustering efficiency is low, so still use the Gaussian filter.

// MeanShift滤波，降噪（速度太慢！）//Imgproc.pyrMeanShiftFiltering(img, img, 30, 10);// 彩色转灰度Imgproc.cvtColor(img, img, Imgproc.COLOR_BGR2GRAY);// 高斯滤波，降噪Imgproc.GaussianBlurnewSize(3,322);

2. Edge Detection

Next, edge detection is performed. This is the whole algorithm is a very critical step, the threshold selection of good is directly related to the subsequent contour line is correct, and can detect the quadrilateral.
Using the canny algorithm to detect edges, the principle of canny algorithm is not mentioned here, there are many high-quality resources on the Internet to help you understand this great edge detection algorithm. Threshold selection, to try to choose the low threshold!!! Because if the threshold selection is too high, it causes the outer quadrilateral of the invoice to be unclosed, which prevents the contour line from being found correctly. Although the low threshold value produces a lot of noise, the noise will be ignored in subsequent steps because contour detection and polygon fitting are followed.
After the canny algorithm, you need to perform an expansion operation again to ensure that the invoice edge is connected.

// Canny边缘检测Imgproc.Canny20603false);// 膨胀，连接边缘Imgproc.dilatenewMatnew Point(-1,-131newScalar(1));

3. Contour Extraction

The edge detection of the results of the contour extraction, using the OPENCV built-in findcontours function, the principle of the function is described in OpenCV Reference Manual. In practice, the retr_external parameter is used to extract only the external contour.

newnewMat();Imgproc.findContours(img, contours, hierarchy, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE);

4, look for convex hull, fit polygon

What should I do if the detected contour still looks messy? First, for each contour, find its convex hull and use a polygon to fit the convex hull border. Next, the area is larger than a certain threshold, and four angles are approximately equal to 90 degrees of convex quadrilateral. The convex quadrilateral found is the candidate periphery quadrilateral.
There are many types of conversions in this code. There are many types of OpenCV in Java, such as matofint,matofpoint,matofpoint2f and so on, and the parameter types of functions in Imgproc are various, so you should pay extra attention when calling functions.
After the code, the call of their own implementation of the function will be affixed to the top of the code, copy the code should be careful not to cuff the wrong oh.

//Calculates the angle of the middle point according to three points pt1 Pt0 pt2Private Static Double Getangle(Point pt1, point pt2, point pt0) {DoubleDX1 = pt1.x-Pt0.x;DoubleDy1 = pt1.y-Pt0.y;DoubleDX2 = pt2.x-Pt0.x;DoubleDy2 = pt2.y-Pt0.y;return(dx1*dx2 + dy1*dy2)/math.sqrt((dx1*dx1 + dy1*dy1) * (dx2*dx2 + dy2*dy2) +1e-10);}//Find out the quadrilateral fitting of the contour corresponding convex hulllist<matofpoint> squares =NewArraylist<> (); List<matofpoint> hulls =NewArraylist<> (); Matofint Hull =New Matofint(); MATOFPOINT2F approx =New matofpoint2f(); approx.ConvertTo(Approx, cvtype.cv_32f); for(Matofpoint contour:contours) {//convex hull of the borderImgproc.Convexhull(Contour, Hull);//using convex hull to calculate new contour pointspoint[] contourpoints = contour.ToArray();int[] indices = hull.ToArray(); List<point> newpoints =NewArraylist<> (); for(intindex:indices) {newpoints.Add(Contourpoints[index]); } matofpoint2f Contourhull =New matofpoint2f(); Contourhull.FromList(newpoints);//Polygon fitting convex hull border (at this point the precision of fitting is lower)Imgproc.APPROXPOLYDP(Contourhull, approx, imgproc.arclength(Contourhull,true)*0.02,true);//filter out an area greater than a certain threshold, and each angle of the quadrilateral is close to the convex quadrilateral of right angleMatofpoint Approxf1 =New Matofpoint(); Approx.ConvertTo(Approxf1, Cvtype.cv_32s);if(approx.rows() ==4&& Math.ABS(Imgproc.Contourarea(approx)) >40000&& Imgproc.Iscontourconvex(APPROXF1)) {DoubleMaxcosine =0; for(intj =2; J <5; J + +) {Doublecosine = Math.ABS(Getangle(Approxf1.ToArray() [j%4], Approxf1.ToArray() [J-2], Approxf1.ToArray() [J-1])); Maxcosine = Math.Max(Maxcosine, cosine); }//angle approx. 72 degrees        if(Maxcosine <0.3) {Matofpoint TMP =New Matofpoint(); Contourhull.ConvertTo(TMP, Cvtype.cv_32s); Squares.Add(APPROXF1); Hulls.Add(TMP); }    }}

5. Find the largest square

As you can see, we've found two large quads (you can zoom in if you can't see them clearly). Comparing the original image, we can find that the periphery is the edge of the invoice we want, while the inner quadrilateral is the table border within the invoice. So we're going to find the biggest square as the edge of the invoice. The implementation is simple, find the largest width and height on the line.

//Find the largest square profilePrivate Static int Findlargestsquare(list<matofpoint> squares) {if(Squares.size() ==0)return-1;intMax_width =0;intMax_height =0;intMax_square_idx =0;intCurrentindex =0; for(Matofpoint square:squares) {Rect rectangle = imgproc.Boundingrect(square);if(Rectangle.width>= max_width && Rectangle.Height>= max_height) {max_width = rectangle.width; Max_height = rectangle.Height;        Max_square_idx = Currentindex;    } currentindex++; }returnMax_square_idx;}//Find the largest quadrilateral of the external rectangleintindex =Findlargestsquare(squares); Matofpoint largest_square = squares.Get(index);if(Largest_square.rows() ==0|| Largest_square.cols() ==0)returnResult

6, re-execute step 3, improve accuracy

Next, for the quadrilateral, the convex hull is re-fitted with the polygon to improve the accuracy.

// 找到这个最大的四边形对应的凸边框，再次进行多边形拟合，此次精度较高，拟合的结果可能是大于4条边的多边形MatOfPoint contourHull = hulls.getnewMatOfPoint2f();contourHull.convertTo(tmp, CvType.CV_32F);Imgproc.approxPolyDP3truenew ArrayList<>();double maxL = Imgproc.arcLengthtrue0.02;

7, find the rectangle four edges, that is, the outer quadrilateral of the paper

The next step is very simple, first to exclude very close to the point in the polygon, and then find the distance is greater than a threshold of four points, it is a rectangle of four vertices. Finally, four vertices are connected, and the steps to extract the quadrilateral border are complete.

//Distance to pointPrivate Static Double Getspacepointtopoint(Point P1, point p2) {DoubleA = P1.x-P2.x;Doubleb = P1.y-P2.y;returnMath.sqrt(A * a + b * b);}//Two intersection of linesPrivate StaticPointComputeintersect(Double[] A,Double[] b) {if(A.length!=4|| B.length!=4)Throw NewClassformaterror ();DoubleX1 = a[0], y1 = a[1], x2 = a[2], y2 = a[3], x3 = b[0], y3 = b[1], x4 = b[2], Y4 = b[3];DoubleD = ((x1-x2) * (Y3-Y4))-((y1-y2) * (x3-x4));if(d! =0) {Point pt =NewPoint (); Pt.x= ((x1 * y2-y1 * x2) * (x3-x4)-(X1-X2) * (x3 * y4-y3 * x4))/D; Pt.y= ((x1 * y2-y1 * x2) * (Y3-Y4)-(y1-y2) * (x3 * y4-y3 * x4))/D;returnPt }Else        return NewPoint (-1, -1);}///Find the vertices of the four vertices maxl with a high-precision fitting that are less than the low-precision fit, and exclude the interference from some vertices for(Point P:approx.ToArray()) {if(! (Getspacepointtopoint(p, Largest_square.toList().Get(0)) > MaxL &&Getspacepointtopoint(p, Largest_square.toList().Get(1)) > MaxL &&Getspacepointtopoint(p, Largest_square.toList().Get(2)) > MaxL &&Getspacepointtopoint(p, Largest_square.toList().Get(3) > MaxL)) {newpointlist.Add(p); }}///Find the remaining vertex lines, the edges grow at 2 * Four sides of the MAXL as four sides of the Quadrilateral objectlist<Double[]> lines =NewArraylist<> (); for(inti =0; I < Newpointlist.size(); i++) {Point P1 = newpointlist.Get(i); Point P2 = newpointlist.Get((i+1)% newpointlist.size());if(Getspacepointtopoint(P1, p2) >2* MaxL) {lines.Add(New Double[]{p1.x, p1.y, p2.x, p2.y}); }}//Calculates the intersection of two adjacent edges in these four edges, that is, the four vertices of an objectList<point> Corners =NewArraylist<> (); for(inti =0; I < lines.size(); i++) {Point corner =Computeintersect(lines.Get(i), lines.Get((i+1)% lines.size())); Corners.Add(corner);}

8, perspective transformation, extraction of quadrilateral

Finally to the last step, the key to the final step is perspective Transform. Create a new mat and make a perspective transformation of the four vertices with the four vertices of the rectangle just detected by the original image, and you will get the final result. The mathematical principle of perspective transformation can be seen here, the introduction of the more detailed.

//Clockwise sorting of multiple pointsPrivate Static void sortcorners(List<point> Corners) {if(Corners.size() ==0)return; Point P1 = Corners.Get(0);intindex =0; for(inti =1; I < corners.size(); i++) {Point point = corners.Get(i);if(P1.x> Point.x) {P1 = point;        index = i; }} corners.Set(Index, corners.Get(0)); Corners.Set(0, p1); Point LP = Corners.Get(0); for(inti =1; I < corners.size(); i++) { for(intj = i +1; J < Corners.size(); J + +) {Point point1 = corners.Get(i); Point Point2 = Corners.Get(j);if(Point1.y-lp.y*1.0)/(Point1.x-lp.x) > (Point2.y-lp.y*1.0)/(Point2.x-lp.x) {Point temp = point1.Clone(); Corners.Set(I, corners.Get(j)); Corners.Set(j, temp); }        }    }}//Clockwise ordering of verticessortcorners(corners);//Calculate the size of the target imagePoint p0 = Corners.Get(0); Point P1 = Corners.Get(1); Point P2 = corners.Get(2); Point P3 = corners.Get(3);DoubleSpace0 =Getspacepointtopoint(P0, p1);DoubleSpace1 =Getspacepointtopoint(P1, p2);DoubleSpace2 =Getspacepointtopoint(P2, p3);DoubleSpace3 =Getspacepointtopoint(P3, P0);DoubleImgWidth = Space1 > Space3? Space1:space3;DoubleImgHeight = space0 > Space2? Space0:space2;//If the extracted image width is less than high, then rotate 90 degreesif(ImgWidth < ImgHeight) {Doubletemp = ImgWidth;    ImgWidth = ImgHeight;    ImgHeight = temp; Point temppoint = P0.Clone(); P0 = P1.Clone(); P1 = p2.Clone(); P2 = p3.Clone(); P3 = Temppoint.Clone();} Mat Quad = Mat.Zeros((int) ImgHeight *2, (int) ImgWidth *2, Cvtype.CV_8UC3); MATOFPOINT2F Cornermat =New matofpoint2f(P0, p1, p2, p3); MATOFPOINT2F Quadmat =New matofpoint2f(NewPoint (imgwidth*0.4, imgheight*1.6),NewPoint (imgwidth*0.4, imgheight*0.4),NewPoint (imgwidth*1.6, imgheight*0.4),NewPoint (imgwidth*1.6, imgheight*1.6));//Extract ImagesMat transmtx = Imgproc.Getperspectivetransform(Cornermat, Quadmat); Imgproc.warpperspective(result, quad, TRANSMTX, Quad.size());returnQuad

These are all the steps of my algorithm. Implementation, tuning down feeling this algorithm is not strong, the reader may need to modify my code to meet the specific business needs, my code is on the Internet other bloggers implementation (C + +, reference link) based on the modification, right when the argument, there are any comments and suggestions can be in the comment area and I exchange.
In addition, some implementations on the Internet do not use polygonal fitting, but the use of Hough transform to achieve this function, but their own implementation of the effect is not good, if someone has experience in this area also hope to enlighten.

OpenCV Java Implementation of notes, paper quadrilateral edge detection and extraction, pendulum