The realization of OpenCV Zhang Zhengyu calibration method

on the principle of Zhang Zhengyu calibration method, the online information is many, although I saw some, but I do not feel to be able to speak very clear degree, so do not do too much of the principle described here. If you are interested in knowing the details, you can see the original text of Zhang, or this article. it is necessary to know the concept of the internal and external parameters and the distortion parameter in the camera calibration. There are five internal reference, namely: The mapping between the camera and the actual object on the x, Y axis (two parameters). The offset relationship between the center of the camera and the center of the image (two parameters). camera and lens mount are not completely vertical, there is an angle deviation. (one parameter) There are six external parameters, which are translation and rotation in the x, Y, z direction respectively. with the above two parameters, we basically know the camera's image and the actual object of the corresponding relationship, but "distortion" can not be ignored. It is the offset of the 2D point due to the lens quality and other causes. A simple example is to take a square with a camera, and the image will become a bucket or other shape. In the Zhang's calibration method, Zhang uses "maximum likelihood method" to calculate the parameters of the distortion (if you want to deepen understanding, you can also refer to a related article I wrote earlier). This introduces the three most important concepts of camera calibration. In general, we have to deal with the distortion of the camera, only require internal and distorted parameters can be, and to do double-target rule needs to be out of the external parameters. This paper mainly introduces the function of the Zhang Zhengyu calibration method of OpenCV to calibrate the camera (to get the parameters of the camera inside and outside and the distortion) and to correct the single image. The main reference is this article (as similar, I copied). need to prepare the calibration board, the appearance of the following (because the courier is slow, I did a first, but because the accuracy of this directly affect the accuracy of the subsequent calibration, suggest or buy a piece to force a bit):

calibration Process: Angular point extraction, camera calibration first, corner extraction　　 will use the more important functions are:

Used to extract the inner corner point of the calibration plate, that is, to extract the corners of each of the four black and white lattice in the above figure
bool Findchessboardcorners (inputarray image, Size patternsize, Outputarray corners, int flags=calib_cb_adaptive_thresh+calib_cb_normalize_image);

There are four parameters: The first "image" is the captured checkerboard image, which is the image shown above; The second "patternsize", that is, each board chart on the number of points on the inner corner, in general, the number of rows is not the same, so that the subsequent calibration program to identify the direction of the calibration plate, like the above board is a size (7, 5), that is, 7 corners per row, 5 corner points per column; The third "corners", which is used to store the detected coordinates of the interior point image, is usually represented by a vector of point2f elements; Fourth "Flage": used to define the different processing methods for the inner corner point lookups on the board chart, with default values. In addition, the return value is important, and it tells you if you actually found the corner point from the diagram. This is useful if you want to make an automatic calibration procedure later. For example, if the input image is shown in the figure above, and our second parameter is size (10, 5), an error is returned;

It is used to further extract subpixel information on the initial extracted corner information and reduce the camera calibration deviation, which is specially used to obtain the precise position of the inner angle point on the board chart. The more common method of extracting subpixel corners is Cornersubpix, which does not repeat the
bool Find4quadcornersubpix (inputarray img, Inputoutputarray Corners, Size Region_size);

There are five parameters: The first "mage", the input mat matrix, preferably 8-bit grayscale image, the detection efficiency is higher; The second "corners", the initial angular coordinate vector, is the output of the sub-pixel coordinate position, so it needs to be floating point data, which is usually represented by the vector of the pointf2f/point2d element. Enter the third parameter of the Findchessboardcorners function above. The third "winsize", which is half the size of the search window; The Fourth "Zerozone", half of the dead zone, is the area where the center of the search area does not have a summation operation. It is used to avoid some possible singularity of the autocorrelation matrix. When the value is ( -1,-1), no dead zone is indicated; The fifth "criteria", which defines the termination condition of the iterative process of the angular point, can be a combination of the number of iterations and the precision of the angular point;

Used to draw a calculated corner point to see if the correct
void drawchessboardcorners is calibrated (inputoutputarray image, Size patternsize, Inputarray Corners, BOOL patternwasfound);  

There are four parameters: the first "image", 8-bit grayscale or color image; The second "patternsize", the number of columns of the inner corner points of each calibration board, i.e. the second parameter of Findchessboardcorners; The third "corners", the corner coordinate vector, can be input by the output of the second parameter of the FIND4QUADCORNERSUBPIX function; The Fourth "Patternwasfound", the flag bit, is used to indicate whether the defined checkerboard inner corner point is fully detected, true indicates that it is fully detected, and the function connects all of the inner corner points in a straight line, as a whole, false means that there are no detected inner angles. This time the function will be detected at the (red) circle mark at the inner corner point; the sample code for the total lookup corner is as follows:

Mat imageinput = Imread ("1.bmp");
Size board_size = Size (7, 5);//number of corner points per row, column on the calibration board
vector<point2f> image_points_buf;//cache the detected corner of each image/
* Extract Corner points */
if (!findchessboardcorners (Imageinput, Board_size, image_points_buf))
{
    cout << "can not find chessboard corners!\n";//Cannot find corner point  
    return;
}
else
{
    Mat view_gray;
    Cvtcolor (Imageinput, View_gray, Cv_rgb2gray);
    /* Sub-pixel precision */
    Find4quadcornersubpix (View_gray, Image_points_buf, Size (5, 5));//precise  
    extraction of corner points Drawchessboardcorners (View_gray, Board_size, Image_points_buf, true); Used to mark corner points in the picture  
    imshow ("Camera calibration", view_gray);//Display Picture  
    waitkey (0);     
}

two. Camera Calibration　　 using the image obtained above the corner point (theoretically need three images, that is, three sets of data, in fact, with 10~20 Zhang is appropriate, because this error will be relatively small), you can use the Calibratecamera function to do camera calibration, calculate the camera's internal, external parameters and distortion parameter. Of course, the preceding code in I only made an image of the corner extraction, can be changed to find more than one, the code is as follows:

Size board_size = Size (7, 5);//number of corner points per row, column on the calibration board vector<point2f> image_points_buf;//cache the detected corner points on each image vector<vector <Point2f>> Image_points_seq;
Save all detected corners/* Extract Corner points */char filename[10];
    for (size_t image_num = 1; image_num <=; image_num++) {sprintf_s (filename, "%d.bmp", image_num);
    Mat imageinput = imread (filename);  if (!findchessboardcorners (Imageinput, Board_size, Image_points_buf)) {cout << "can not find chessboard Corners!\n ";
    Could not find the corner point return;
        } else {Mat view_gray;
        Cvtcolor (Imageinput, View_gray, Cv_rgb2gray); /* subpixel Precision */Find4quadcornersubpix (View_gray, Image_points_buf, Size (5, 5)); The angular point of coarse extraction is drawchessboardcorners (View_gray, Board_size, Image_points_buf, true); Used to mark the corner point Image_points_seq.push_back (IMAGE_POINTS_BUF) in the picture,//Save Subpixel Corner imshow ("Camera calibration", View_
Gray);//Display Picture Waitkey (500);//Stop Half Second} imageinput.release (); }

Double Calibratecamera (inputarrayofarrays objectpoints,  
                       inputarrayofarrays imagepoints,  
                       Size imageSize,  
                       cv_out Inputoutputarray Cameramatrix,  
                       Cv_out Inputoutputarray distcoeffs,  
                       outputarrayofarrays rvecs, outputarrayofarrays tvecs,  
                       int flags=0, Termcriteria criteria = 
                       Termcriteria (termcriteria::count+termcriteria::eps, Dbl_epsilon));  

the parameters are many, there are more than nine ... The first "Objectpoints" is a three-dimensional point in the world coordinate system. When used, you should enter a vector set of three-dimensional coordinate points. In general, we assume that the calibration plate is placed on the z=0 plane, and then based on the size of a single black-and-white square on the board (and can take 10 directly), you can calculate the world coordinates of each inner corner point if you do not need to map it exactly to the real thing. The second "imagepoints" is the coordinate point of the image corresponding to each internal angle point. It is the corner set of each image obtained above; The third "imageSize", for the pixel size of the image, needs to be used when calculating the camera's internal reference and distortion matrix; The fourth "Cameramatrix" is the camera's internal reference matrix. Enter a mat Cameramatrix, such as Mat Cameramatrix=mat (3,3,cv_32fc1,scalar::all (0)); the fifth "distcoeffs" is a distortion matrix. Enter a Mat distcoeffs=mat (1,5,cv_32fc1,scalar::all (0)); The Sixth "Rvecs" is a rotational vector; a vector of Mat type should be entered, i.e. vector<mat>rvecs; the seventh "Tvecs" is a displacement vector, like rvecs, which should be vector<mat> tvecs; The eighth "flags" is the algorithm used to mark the timing. There are several parameters (directly not written according to the following parameter description does not set parameters in the case):

Cv_calib_use_intrinsic_guess: When using this parameter, there should be an estimate of fx,fy,u0,v0 in the Cameramatrix matrix. Otherwise, the center point of the (u0,v0) image is initialized, and the fx,fy is estimated using the least squares. 
cv_calib_fix_principal_point: The optical axis points are fixed when optimizing. When the cv_calib_use_intrinsic_guess parameter is set, the optical axis point remains in the center or an input value. 
Cv_calib_fix_aspect_ratio: The ratio of fixed fx/fy, only the FY as a variable, to optimize the calculation. When Cv_calib_use_intrinsic_guess is not set, FX and FY will be ignored. Only the fx/fy ratio will be used in the calculation. 
cv_calib_zero_tangent_dist: Sets the tangential distortion parameter (P1,P2) to zero. 
cv_calib_fix_k1,..., cv_calib_fix_k6: The corresponding radial distortion remains unchanged in the optimization. 
Cv_calib_rational_model: Calculates the k4,k5,k6 three distortion parameters. If not set, only the other 5 distortion parameters are computed.

The Nineth "criteria" is the optimal iteration termination condition setting. before using this function for calibration, it is necessary to initialize the position coordinates of the spatial coordinate system of each interior point of the board, and the result of the calibration is to generate the camera's reference matrix Cameramatrix, the camera's 5 distortion coefficient distcoeffs, In addition, each image will generate its own translation vector and rotation vector. the specific implementation code is as follows:

Size image_size;//The dimensions of the image size Board_size = size (7, 5);  The angle points of each row and column on the calibration board vector<point2f> Image_points_buf; Cache the detected corner points on each image vector<vector<point2f>> Image_points_seq;
Save all detected corners/* Extract Corner points */char filename[10];
    for (size_t image_num = 1; image_num <= imgcount; image_num++) {sprintf_s (filename, "%d.bmp", image_num);
    Mat imageinput = imread (filename);  if (!findchessboardcorners (Imageinput, Board_size, Image_points_buf)) {cout << "can not find chessboard
    Corners!\n ";//Can not find the corner point return;
        } else {Mat view_gray;
        Cvtcolor (Imageinput, View_gray, Cv_rgb2gray); /* subpixel Precision */Find4quadcornersubpix (View_gray, Image_points_buf, Size (5, 5));//precise extraction of corner points DRAWCHESSBOARDC Orners (View_gray, Board_size, Image_points_buf, true);//Used to mark corner Image_points_seq.push_back (image_points_buf) in a picture;
   Save Subpixel Corner Point Imshow ("Camera calibration", view_gray);//Display Picture//waitkey (500);//Stop Half Second} Image_size.width = Imageinput.cols;
    Image_size.height = imageinput.rows;
Imageinput.release (); }/* Camera calibration */vector<vector<point3f>> object_points; Save the three-dimensional coordinates of the corner point on the calibration board, which is the first parameter of the calibration function size square_size = size (10, 10);//the size of each checkerboard on the calibration board actually measured, here is actually not measured, assume a value, the sense of influence is not too big, back again for
(int t = 0; t temppointset;
            for (int i = 0; i<board_size.height; i++) {for (int j = 0; j<board_size.width; j + +) {
            point3f Realpoint;
            Assuming that the calibration plate is placed in the world coordinate system in the z=0 plane realpoint.x = i*square_size.width;
            Realpoint.y = J*square_size.height;
            realpoint.z = 0;
        Temppointset.push_back (Realpoint);
}} object_points.push_back (Temppointset);   
}//External Parameter object Mat Cameramatrix = Mat (3, 3, CV_32FC1, Scalar::all (0));//Camera internal parameter matrix vector<int> point_counts;//number of corner points per image Mat distcoeffs = Mat (1, 5, CV_32FC1, Scalar::all (0));//5 distortion factor of the camera: K1,k2,p1,p2,k3 vector<mat> Tvecsmat;//the rotation vector of each image vector<mat> rvecsmat;//the translation vector of each image Calibratecamera (object_points, Image_points_seq, Image_size,
 Cameramatrix, Distcoeffs, Rvecsmat, Tvecsmat, 0);//Camera Calibration

by this we have completed the calibration process, get the camera parameters, after which we can use these parameters to do the camera correction. corrections can be used with the following functions:

void Undistort (Inputarray src, outputarray dst,inputarray cameramatrix,inputarray Distcoeffs,inputarray Newcameramatrix=noarray ());  

There are five parameters: The first "src", the input parameter, represents the original image of the distortion; The second "DST", the corrected output image, has the same type and size as the input image; The third "Cameramatrix" is the internal reference matrix of the previously obtained camera; The fourth "distcoeffs" is the camera distortion matrix previously obtained; Fifth "Newcameramatrix", the default and Cameramatrix consistent; The specific code is as follows:

/* Correct the image with calibrated results *
/for (size_t image_num = 1; image_num <= imgcount; image_num++)
{
    sprintf_s (filename, "% D.bmp ", image_num);
    Mat ImageSource = imread (filename);
    Mat newimage = Imagesource.clone ();
    Undistort (ImageSource, NewImage, Cameramatrix, distcoeffs);
    Imshow ("source", imagesource);//Display Picture 
    imshow ("DRC", newimage);//Display Picture  
    Waitkey (500);//Stop Half second
    Imagesource.release ();
    Newimage.release ();
}

to complete the camera calibration and image correction of the entire process, if you want to know how to evaluate the calibration effect, you can refer to the reference article mentioned above, many of the functions of this paper is basically copied, just do some code on the split. if you want to save the result of the calibration and download it directly, you can save the following code:

/* Save the Internal reference and distortion coefficients for direct correction/
ofstream fout ("Caliberation_result.txt");//file to save the calibration result
fout << "in-Camera parameter matrix:" << Endl;
Fout << cameramatrix << endl << Endl;
Fout << "distortion factor: \ n";
Fout << distcoeffs << endl << endl << Endl;
Fout.close ();

read the calibration file file code as follows:

Char read[100];
Double GetData;
Mat Cameramatrix = Mat (3, 3, CV_32FC1, Scalar::all (0));//Camera internal parameter matrix 
Mat distcoeffs = Mat (1, 5, CV_32FC1, Scalar::all (0)) ///Camera 5 distortion factor: K1,k2,p1,p2,k3
ifstream fin ("caliberation_result.txt");//Read the file to save the calibration results for correction
fin >> Read ;
FIN.SEEKG (3, ios::cur);
for (size_t j = 0, J < 3, J + +) for
    (size_t i = 0; i < 3; i++)  
    {
        fin >> getdata;
        Cameramatrix.at<float> (j, i) = GetData;
        Fin >> read;
    }
Fin >> read;
FIN.SEEKG (3, ios::cur);
for (size_t i = 0; i < 5; i++)
{
    fin >> getdata;
    Distcoeffs.at<float> (i) = GetData;
    Fin >> read;
}   
Fin.close ();

the entire code for this article is as follows:

#include "opencv2/core/core.hpp" #include "opencv2/imgproc/imgproc.hpp" #include "opencv2/calib3d/calib3d.hpp" #incl Ude "opencv2/highgui/highgui.hpp" #include <iostream> #include <fstream> #define Imgcount using Nam
Espace CV;

using namespace Std;     void Main () {size image_size;//image is sized board_size = size (9, 6);  The angle points of each row and column on the calibration board vector<point2f> Image_points_buf; Cache the detected corner points on each image vector<vector<point2f>> Image_points_seq;
    Save all detected corners/* Extract Corner points */char filename[10];
        for (size_t image_num = 1; image_num <= imgcount; image_num++) {sprintf_s (filename, "%d.bmp", image_num);
        Mat imageinput = imread (filename); if (!findchessboardcorners (Imageinput, Board_size, Image_points_buf)) {cout << ' can not ' find CH
        Essboard corners!\n ";//Cannot find the corner point return;
            } else {Mat view_gray; Cvtcolor (Imageinput, View_gray, Cv_rgb2GRAY); /* subpixel Precision */Find4quadcornersubpix (View_gray, Image_points_buf, Size (5, 5));//precise extraction of corner points Drawche Ssboardcorners (View_gray, Board_size, Image_points_buf, true);//used to mark corners in the picture Image_points_seq.push_back (Image_
        POINTS_BUF);//Save Subpixel Corner point Imshow ("Camera calibration", view_gray);//Display Picture Waitkey (500);//Stop Half second
        } image_size.width = Imageinput.cols;
        Image_size.height = imageinput.rows;
    Imageinput.release (); }/* Camera calibration */vector<vector<point3f>> object_points; Save the three-dimensional coordinates of the corner point on the calibration board, which is the first parameter of the calibration function size square_size = size (10, 10);//the size of each checkerboard on the calibration board actually measured, here is actually not measured, assume a value, the feeling influence is not too big, back to study F
        or (int t = 0; t temppointset;
            for (int i = 0; i<board_size.height; i++) {for (int j = 0; j<board_size.width; j + +)
                {point3f realpoint;
   Assuming that the calibration plate is placed on a z=0 plane in the world coordinate system             Realpoint.x = I*square_size.width;
                Realpoint.y = J*square_size.height;
                realpoint.z = 0;
            Temppointset.push_back (Realpoint);
    }} object_points.push_back (Temppointset);  }//External Parameter object Mat Cameramatrix = Mat (3, 3, CV_32FC1, Scalar::all (0));//Camera internal parameter matrix vector<int> point_counts;// The number of corner points in each image mat Distcoeffs = Mat (1, 5, CV_32FC1, Scalar::all (0));//5 distortion factor of the camera: K1,k2,p1,p2,k3 vector<mat> t vecsmat;//the rotation vector of each image vector<mat> rvecsmat;//the translation vector of each image Calibratecamera (object_points, Image_points_seq, Image_ Size, Cameramatrix, distcoeffs, Rvecsmat, Tvecsmat, 0);//Camera calibration//correction of images with calibrated results */for (size_t image_num = 1; image_num <= Imgcount;
        image_num++) {sprintf_s (filename, "%d.bmp", image_num);
        Mat ImageSource = imread (filename);
        Mat newimage = Imagesource.clone ();
        Undistort (ImageSource, NewImage, Cameramatrix, distcoeffs); Imshow ("source", imagesource);//Display Picture imshow ("DRC", newimage);//Display picture sprintf_s (filename, "%d_d.bmp", Image_n
        UM);
        Imwrite (filename, newimage);//Display Picture Waitkey (500);//Stop Half second imagesource.release ();
    Newimage.release (); }/* Save the Internal reference and distortion coefficients for direct correction of the back/Ofstream fout ("Caliberation_result.txt");//Save the calibration result file Fout << "in-Camera parameter matrix:" <&lt ;
    Endl
    Fout << cameramatrix << endl << Endl;
    Fout << "distortion factor: \ n";
    Fout << distcoeffs << endl << endl << Endl;

    Fout.close ();
    * Read pre-calibrated data directly corrected *///char read[100];
    Double GetData; Mat Cameramatrix = Mat (3, 3, CV_32FC1, Scalar::all (0));//Camera internal parameter matrix//mat distcoeffs = Mat (1, 5, CV_32FC1, Scalar::al L (0));//5 distortion factor of the camera: K1,k2,p1,p2,k3//ifstream fin ("caliberation_result.txt");//Read the file to save the calibration result for correction//fin >> read
    ;
    FIN.SEEKG (3, ios::cur); for (size_t j = 0; J < 3; J + +)//for (size_t i = 0; i < 3; i++)///{//fin >> GetData;
    Cameramatrix.at<float> (j, i) = GetData;
    Fin >> Read;
    }//fin >> read;
    FIN.SEEKG (3, ios::cur);
    for (size_t i = 0; i < 5; i++)//{//Fin >> GetData;
    Distcoeffs.at<float> (i) = GetData;
    Fin >> Read;

    }//fin.close ();
    Char filename[10]; for (size_t image_num = 1; image_num <= imgcount; image_num++)//{//sprintf_s (filename, "%d.bmp", Image_nu
    m);
    Mat ImageSource = imread (filename);
    Mat newimage = Imagesource.clone ();
    Undistort (ImageSource, NewImage, Cameramatrix, distcoeffs);  Imshow ("source", imagesource);//Display picture//Imshow ("DRC", newimage);//Display picture//sprintf_s (filename, "%d_d.bmp",
    Image_num);
    Imwrite (filename, newimage);//Display picture//Waitkey (500);//Stop half Second//imagesource.release ();
    Newimage.release (); //}
}