Learning opencv -- orb & brief (feature points) & location

 

What is orb first:

(This part is transferred from http://www.cvchina.info/2011/07/04/whats-orb)

Orb is short for oriented brief. Orb is described in the following article:

Ethan rublee and Vincent rabaud and Kurt konolige and Gary bradski, orb: an efficient alternative
To sift or surf, iccv 2011

The thesis can be downloaded from Google. opencv2.3 already has implementation, and willowgarage has a talk which also mentions this algorithm. Therefore, I will not make it simple. Here I will summarize it.

Brief

Brief is short for Binary robust independent elementary features. This feature descriptor is proposed by calonder of EPFL on eccv2010. The main idea is to randomly select several points near the feature points, and combine the gray-scale values of these points into a binary string, the binary string is used as the feature descriptor of the feature. For detailed algorithm description, see the following thesis:

Calonder M., lepetit v., strecha C., fua p.: Brief: Binary robust independent elementary features. eccv 2010

Note: In the brief eccv2010 article, each bit in the brief description is obtained by comparing two randomly selected pixels in binary format. As mentioned in the article, we need to select a proper Gaussian Kernel to smooth the image. (Why do we need to emphasize this, because the orb below has improved this .)

 

The advantage of brief is speed, and its disadvantages are also quite obvious:

1: it does not have rotation immutability.

2: sensitive to noise

3: Scale immutability is not available.

Orb is trying to solve the shortcomings 1 and 2.

 

How to solve rotation immutability:

In the orb solution, fast is used as the feature point detection operator. There are a lot of fast applications and they are famous for being fast. In case someone doesn't know about them, please refer to the following:

In the sift scheme, the main direction of the feature point is determined by the maximum value of the gradient histogram and the corresponding direction of the bin where the secondary histogram is located. Too time-consuming.

In the orb solution, the main direction of the feature point is calculated by the moment (moment). The formula is as follows:

The zero moment calculation is as follows:

The first moment is calculated as follows:

With the primary direction, you can extract the brief description based on the primary direction. However, the resulting problem is that, as the main direction changes, the correlation between random points is relatively large, thus reducing the discriminant of the descriptive sub. The solution is also very direct. It uses greedy and exhaustive methods to find random points with low correlation.

 

How to solve noise-sensitive problems:

As mentioned above, in the earliest eccv2010 article, brief uses the size of pixel and pixel to construct each bit of the description child. The consequence is that it is sensitive to noise. Therefore, this improvement is made in the orb solution. pixel-pair is not used, but patch-pair of 9 × 9 is used, that is, the sum of the pixel values of the patch is compared. (You can use the integral graph for quick calculation ).

 

Scale immutability:

Orb did not try to solve the scale immutability (because fast itself does not have the scale immutability .) However, feature descriptors that only require speed are generally applied to real-time video processing. In this way, we can solve the problem of scale immutability through tracking and some heuristic strategies.

 

About the computing speed:

Orb is 100 times of sift and 10 times of surf.

 

Performance:

The following is a performance comparison, ORB is still very powerful. Click to view the chart.

Refer to slides

 

 

 

Code:

Orb detection + fluoroscopy transformation and positioning

To achieve real-time Positioning and tracking, you can reduce the number of keypoints to reduce the detection and match time (SRC. keypoint = 30, IMG. keypoints = 100 to meet real-time requirements );

However, due to the computed perspective transformation

// Look for two matrices of perspective transformation 3*3 matrices ☆☆☆☆☆☆☆☆☆☆☆☆☆☆;
Homo1 = findhomography (pt1, pt2, cv_ransac, (5.0 ));

This step can be improved because it takes too much time (by calculating the average coordinate of the keypoint or fitting the edge, you can try again next)

 

The following code is attached:

 

# Include "opencv2/objdetect. HPP "# include" opencv2/features2d/features2d. HPP "# include" opencv2/highgui. HPP "# include" opencv2/calib3d/calib3d. HPP "# include" opencv2/imgproc/imgproc_c.h "# include" opencv2/imgproc. HPP "# include <string> # include <vector> # include <iostream> using namespace CV; using namespace STD; char * image_filename2 =" D:/src.jpg "; char * image_filename1 = "d :/Demo1.jpg "; unsigned int hamdist (unsigned int X, unsigned int y) {unsigned int Dist = 0, val = x ^ y; // count the number of set bits while (VAL) {++ Dist; Val & = Val-1;} return Dist;} unsigned int hamdist2 (unsigned char *, unsigned char * B, size_t size) {hamminglut LUT; unsigned int result; Result = LUT (a), (B), size); return result ;} void naive_nn_search (vector <keypoint> & keys1, mat & D Escp1, vector <keypoint> & keys2, mat & descp2, vector <dmatch> & matches) {for (INT I = 0; I <(INT) keys2.size (); I ++) {unsigned int min_dist = int_max; int min_idx =-1; // note pointer usage: mat. PTR (I) ☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆unsigned char * query_feat = descp2.ptr (I ); for (Int J = 0; j <(INT) keys1.size (); j ++) {unsigned char * train_feat = descp1.ptr (j); unsigned int Dist = hamdist2 (query_feat, train_fea T, 32); If (Dist <min_dist) {min_dist = DIST; min_idx = J ;}// if (min_dist <= (unsigned INT) (second_dist * 0.8) {If (min_dist <= 50) {matches. push_back (dmatch (I, min_idx, 0, (float) min_dist) ;}} void naive_nn_search2 (vector <keypoint> & keys1, mat & descp1, vector <keypoint> & keys2, mat & descp2, vector <dmatch> & matches) {for (INT I = 0; I <(INT) keys2.size (); I ++) {unsigned int min_dist = int _ Max; unsigned int sec_dist = int_max; int min_idx =-1, sec_idx =-1; unsigned char * query_feat = descp2.ptr (I); For (Int J = 0; j <(INT) keys1.size (); j ++) {unsigned char * train_feat = descp1.ptr (j); unsigned int Dist = hamdist2 (query_feat, train_feat, 32 ); // The shortest distance if (Dist <min_dist) {sec_dist = min_dist; sec_idx = min_idx; min_dist = DIST; min_idx = J;} // time short distance else if (Dist <sec_dist) {SEC _ Dist = DIST; sec_idx = J ;}} if (min_dist <= (unsigned INT) (sec_dist * 0.8) & min_dist <= 50) {matches. push_back (dmatch (I, min_idx, 0, (float) min_dist) ;}} int main (INT argc, char * argv []) {mat img1 = imread (image_filename1, 0); MAT img2 = imread (image_filename2, 0); // gaussianblur (img1, img1, size (5, 5), 0); // gaussianblur (img2, img2, size (5, 5), 0); orb orb1 (100, orb: commonparams (1.2, 1 )); Orb orb2 (10, orb: commonparams (1.2, 1); vector <keypoint> keys1, keys2; MAT descriptors1, descriptors2; int64 St, et; // extract orb features; orb1 (img1, MAT (), keys1, descriptors1, false); printf ("TEM feat num: % d \ n", keys1.size (); ST = cvgettickcount (); orb2 (img2, MAT (), keys2, descriptors2, false); ET = cvgettickcount (); printf ("orb2 extraction time: % F \ n", (ET-St) /(double) cvgettickfrequency ()/1000 .); prin TF ("query feat num: % d \ n", keys2.size (); // find matches vector <dmatch> matches; ST = cvgettickcount (); for (INT I = 0; I <10; I ++) {naive_nn_search2 (keys1, descriptors1, keys2, descriptors2, matches);} ET = cvgettickcount (); printf ("Match Time: % F \ n", (ET-St)/(double) cvgettickfrequency ()/1000 .); printf ("matchs num: % d \ n", matches. size (); MAT showimg; drawmatches (img2, keys2, img1, keys1, matc Hes, showimg, cv_rgb (0,255, 0), cv_rgb (0, 0,255); string winname = "matches"; namedwindow (winname, window_autosize); imshow (winname, showimg ); waitkey (0); vector <point2f> pt1; vector <point2f> pt2; For (INT I = 0; I <(INT) matches. size (); I ++) {pt1.push _ back (point2f (keys2 [matches [I]. queryidx]. PT. x, keys2 [matches [I]. queryidx]. PT. y); pt2.push _ back (point2f (keys1 [matches [I]. trainidx]. PT. x, keys 1 [matches [I]. trainidx]. PT. y);} mat homo, homo1; ST = cvgettickcount (); // look for two matrices of perspective transformation 3*3 matrices ☆☆☆☆☆☆☆☆☆☆☆☆; homo1 = findhomography (pt1, pt2, cv_ransac, (5.0 )); homo1.converize (Homo, cv_32f); // normalize (Homo, homo, 1, 0 ); printf ("homo \ n" "% F \ n", homo. at <float> (0, 0), homo. at <float> (0, 1), homo. at <float> (0, 2), homo. at <float> (1, 0), homo. at <float> (1, 1), homo. at <float> (1, 2), homo. At <float> (2, 0), homo. at <float> (2, 1), homo. at <float> (); // coordinate of the Target Image vertex ☆☆☆☆☆☆☆☆☆☆☆cvpoint src_corners [4] = {0, 0 }, {img2.cols, 0}, {img2.cols, img2.rows}, {0, img2.rows}; cvpoint dst_corners [4]; double H [9]; H [0] = homo. at <float> (0, 0); H [1] = homo. at <float> (0, 1); H [2] = homo. at <float> (0, 2); H [3] = homo. at <float> (1, 0); H [4] = homo. at <float> (1, 1); H [5] = homo. at <float> (1, 2); H [6] = homo. at <float> (2, 0); H [7] = homo. At <float> (2, 1); H [8] = homo. at <float> (); size_t I; // use the extracted 3*3 perspective transform matrix, perform pivoting transformation on the coordinates of the four vertices of the target image for (I = 0; I <4; I ++) {Double X = src_corners [I]. x, Y = src_corners [I]. y; Double Z = 1. /(h [6] * x + H [7] * Y + H [8]); double X = (H [0] * x + H [1] * Y + H [2]) * z; double Y = (H [3] * x + H [4] * Y + H [5]) * z; dst_corners [I] = cvpoint (cvround (x ), cvround (y);} mat IMG = imread (image_filename1, 1); // connect the transformed coordinates with a straight line to locate them! For (I = 0; I <4; I ++) {cvpoint R1 = dst_corners [I % 4]; cvpoint r2 = dst_corners [(I + 1) % 4]; line (IMG, cvpoint (r1.x, r1.y), cvpoint (r2.x, r2.y), scalar (255,), 2);} imshow ("location", IMG ); ET = cvgettickcount (); printf ("ransac time: % FMS \ n", (ET-St)/(double) cvgettickfrequency ()/1000 .); waitkey (0); vector <point2f> reproj; reproj. resize (pt1.size (); // perspective transform of Vector Array; ☆☆☆☆☆☆☆☆☆☆☆☆perspectivetransform (pt1, reproj, homo); MAT diff; diff = MAT (reproj)-mat (pt2); int inlier = 0; double err_sum = 0; For (INT I = 0; I <diff. rows; I ++) {uchar * PTR = diff. PTR (I); float err = PTR [0] * PTR [0] + PTR [1] * PTR [1]; If (ERR <25.f) {inlier ++; err_sum + = SQRT (ERR) ;}} printf ("inlier num: % d \ n", inlier); printf ("ratio % F \ n", inlier/(float) (diff. rows); printf ("mean Reprojection error: % F \ n", err_sum/inlier); Return 0 ;}