CMT Algorithm of motion tracking

CMT (clustering of static-adaptive correspondences for deformable Object Tracking) is a relatively new set of tracking algorithms, born in 2014 , originally called consensus-based Tracking and Matching of keypoints for Object Tracking, was in the Computer Vision application (application of computer vision) Won the best Paper award at the Winter Conference, which was then published in CVPR in 2015: The address of the paper.

CMT is a very useful and super-effective tracking algorithm that can track any object in any scene; This is a feature-based tracking method and uses the classical optical flow method as part of the algorithm. This article does not intend to analyze the CMT code, because he is easy to find (Github), there are many articles on the Internet that analyze it, the purpose of this blog is to combine the original thesis analysis of the main idea of CMT.

--Initialization

Step1: (interactive) Select the object frame PR to be traced, calculate the center of rectangle frame;

Step2: Gray frame, extracting fast feature points, and separating the feature points of foreground and background parts;

Step3: Creates the foreground class, which is the index of the feature point;

Step4: Generating a brisk feature descriptor for fast feature points;

STEP5: Normalization of the foreground feature points;

STEP6: Using normalized foreground point initialization match (bruteforce-hamming), Generate foreground class label, match Library;

STEP7: Using normalized foreground point initializer to generate the distance and angle matrix between any point pairs in a rectangle;

STEP8: Create an initial valid class and a valid point, i.e. the coordinates of the former attraction and the class label generated in the STEP6;

--optical flow method to obtain the first part of Track point

Step1: Using Forward-backward tracking to improve accuracy;

Step2: Reject the point where the trace failed;

--Global match get second part track point

That is, the full image matches the rectangular frame, and this part of the work corresponds to the ' 3.1 static-adaptive correspondences '; where the thr_dist in code corresponds to the paper, Thr_ratio corresponds to the paper;

--Data fusion for two-part tracking points

That is to find the two parts are traced to the point, retained;

--Estimating scale and rotation for condensation clustering

This part of the work corresponds to the paper in the ' 3.2 Correspondence Clustering ';

Step1: Estimating scale changes;

Step2: Estimating rotation changes;

Step3: Using scale and rotation to generate the transformation matrix H, that is, the transformation matrix h in the thesis;

Step4: To find the consistency between each feature point, the method is to let each point as the center vote, this part of the purpose is to make the match more accurate;

In this section, the thesis is based on the assumption that the target-related points are definitely included in the largest cluster, so our goal is to vote by calculating the non-similarity d (corresponding to D in the paper) between the vote points, and finally get the largest clustering; where the thr_ used in code Cutoff corresponds to the paper, this parameter controls the degree of deformation of the tolerated object, when its value is small, it will often lead to all points are zoned outside the point outliers, when its value is larger, often can identify more points for the inner point inliers, the duty is 0 o'clock, it means that the object is completely rigid;

Note: This part of the cluster, the author is called a cluster library fastcluster to complete the implementation;

--local matching using the points obtained by fusion;

That is, the rectangular box matches the rectangle, and this part of the work is intended to disambiguate, that is, in order to solve the exact same point or a point matching the same descriptor is difficult problem, it corresponds to the paper in the ' 3.3 disambiguation of correspondences ';

--Data fusion again

--Update the output of the Tracking rectangle box

--The rectangle box and the inner point inside the rectangle as the input of the next frame, loop

It can be found that the core of CMT lies in the following formulas:

Finally, in order to be clearer, I will re-annotate the good code attached, should be good to understand;

 Initialize void cmt::initialize (const Mat im_gray, const cv::rect Rect) {//remember Initial size stores the initial sizes of the tracking area        Size_initial = Rect.size ();        Remember Initial image stores the initial grayscale image im_prev = Im_gray;                Compute Center of Rect calculates the central location of the tracking area POINT2F Center = point2f (rect.x + rect.width/2.0, Rect.y + rect.height/2.0); Initialize Detector and descriptor initialization Detector fast and description sub-Extractor Brisk detector = Cv::fastfeaturedetector::crea    Te ();        Descriptor = CV::D escriptorextractor::create (Str_descriptor);        Descriptor = Cv::brisk::create (); Get initial keypoints in whole image and compute their descriptors//////////////////////////////////////////////        Optimizing vector<keypoint> keypoints; Detector->detect (Im_gray, keypoints);  Detects all key points of the initial full image//divide keypoints into foreground and background keypoints according to selection Isolate the key points of the foreground and background, beforeThe scene is the tracking box inside the vector<keypoint> KEYPOINTS_FG;        Vector<keypoint> KEYPOINTS_BG;            for (size_t i = 0; i < keypoints.size (); i++) {KeyPoint k = keypoints[i];            POINT2F pt = k.pt;            if (Pt.x > Rect.x && pt.y > Rect.y && pt.x < rect.br (). x && Pt.y < rect.br (). Y)            {Keypoints_fg.push_back (k);            } else {keypoints_bg.push_back (k);        }}//create foreground classes create the foreground class, which is the index of the feature point in the rectangle vector<int> classes_fg;        Classes_fg.reserve (Keypoints_fg.size ());        for (size_t i = 0; i < keypoints_fg.size (); i++) {classes_fg.push_back (int) i);        }//compute Foreground/background features calculates the foreground and background of the feature descriptor Mat DESCS_FG;        Mat DESCS_BG;        Descriptor->compute (Im_gray, KEYPOINTS_FG, DESCS_FG); Descriptor->compute (Im_gray, KEYPOINTS_BG, DESCS_BG); Keypoints to points, as compute () might remove some keypoints convert key points to point storage ve        Ctor<point2f> POINTS_FG;        Vector<point2f> POINTS_BG;        for (size_t i = 0; i < keypoints_fg.size (); i++) {points_fg.push_back (keypoints_fg[i].pt);        } for (size_t i = 0; i < keypoints_bg.size (); i++) {points_bg.push_back (keypoints_bg[i].pt); }//create normalized points creates normalized points, which are the coordinates of the computed foreground key to the relative position of the center of the foreground rectangle as a normalized point vector<point        2f> points_normalized;        for (size_t i = 0; i < points_fg.size (); i++) {Points_normalized.push_back (Points_fg[i]-center); }//initialize Matcher initializes the match, generates the class label, matches the library matcher.initialize (points_normalized, DESCS_FG        , CLASSES_FG, DESCS_BG, center);   Initialize consensus initializes the consensus, resulting in the distance and angle consensus.initialize (points_normalized) between any point pairs within the rectangle box;     Create initial set of active keypoints creates an initial valid point and a valid class, that is, the coordinates for the foreground key for (size_t i = 0; i < keypoints_fg.size ();            i++) {points_active.push_back (keypoints_fg[i].pt);        Classes_active = CLASSES_FG; }}//frame processing void CMT::p rocessframe (Cv::mat im_gray) {//track keypoints vector<point2f> poin        ts_tracked;                vector<unsigned char> status;        The first part of Feature point//uses optical flow method to get some tracking points.        Tracker.track (Im_prev, Im_gray, points_active, points_tracked, status);        File_log (Logdebug) << points_tracked.size () << "tracked points.";        Keep only successful classes culling trace failed points vector<int> classes_tracked; for (size_t i = 0; i < classes_active.size (); i++) {if (Status[i]) {class            Es_tracked.push_back (Classes_active[i]); }}//Second part feature point//detect keypoints, compute descriptors calculates the key point of the current image VectoR<keypoint> keypoints;        Detector->detect (Im_gray, keypoints);        Calculates the description of the current image feature point Mat descriptors;        Descriptor->compute (Im_gray, keypoints, descriptors); Match keypoints globally uses the database global matching feature points to calculate matching feature points and class//********** global matches (3.1 static-adaptive Correspo        ndences) ***************** vector<point2f> Points_matched_global;        Vector<int> Classes_matched_global;        Matcher.matchglobal (keypoints, descriptors, Points_matched_global, Classes_matched_global);                        File_log (Logdebug) << points_matched_global.size () << "points matched globally."; Fuse tracked and globally matched points//fusion tracking and globally matched point//********************* first Data fusion *********************        Vector<point2f> points_fused;        Vector<int> classes_fused; Fusion.preferfirst (points_tracked, classes_tracked, poiNts_matched_global, Classes_matched_global, points_fused, classes_fused);                        File_log (Logdebug) << points_fused.size () << "points fused.";                                Estimation of rotation and scale,//***************** clustering (3.2 correspondence clustering) *************************        Estimate scale and rotation from the fused points//the scale and rotation here is the transformation matrix H float scale in the thesis;        float rotation;        Consensus.estimatescalerotation (points_fused, classes_fused, scale, rotation);        File_log (logdebug) << ' scale ' << scale << ', ' << ' rotation ' << rotation;        Find Inliers and the center of their votes//COMPUTE consistency, get inliers and center POINT2F centre;        Vector<point2f> Points_inlier;        Vector<int> Classes_inlier; Consensus.findconsensus (points_fused, classes_fused, scale, rotation, center, Points_inlier, Classes_inlier        ); File_log (lOgdebug) << points_inlier.size () << "inlier points.";                                File_log (logdebug) << "center" << Center; Match keypoints locally local match//************************* disambiguation (3.3 disambiguation of correspondences) **************        Vector<point2f> points_matched_local;        Vector<int> classes_matched_local;        Matcher.matchlocal (keypoints, descriptors, center, scale, rotation, points_matched_local, classes_matched_local);        File_log (Logdebug) << points_matched_local.size () << "points matched locally.";        Clear Active points points_active.clear ();                        Classes_active.clear (); Fuse locally matched points and inliers//fusion locally matched points and inliers//********************* second data fusion **************** Fusion.preferfirst (points_matched_local, Classes_matched_locAl, Points_inlier, Classes_inlier, points_active, classes_active);        Points_active = points_fused;        Classes_active = classes_fused;        File_log (Logdebug) << points_active.size () << "final fused points."; Todo:use Theta to suppress result//Calculate new trace window//update output Rectangle Bb_rot = rotatedrect (center, Siz        E_initial * scale, ROTATION/CV_PI * 180);        Remember Current image updates the previous frame image im_prev = Im_gray;    File_log (logdebug) << "CMT::p rocessframe () return"; }

CMT Algorithm of motion tracking