Opencv for iOS Study Notes (7)-gesture marking Simulation

Original address: opencv for iOS Study Notes (7)-gesture marking Simulation

Previously, we have obtained a precise marking corner point. We can simulate the transformation between the mark in the camera and the 3D space. In this process, we will find a European transformation between the camera and the object-only contains rotation and conversion.

Where C is represented as the center of the camera, the P1-P4 is the three-dimensional point of the world axis, and the p1-p4 is their projection on the camera image plane. Our goal is to use an internal matrix and a known point in the image plane to identify the conversion between the positions of known tags in a 3D space and camera C.

But how do we obtain the coordinates of the positions marked in the Three-dimensional space? Because our markup is always square and all vertices are in the same plane, we can define their corners as below:

Now we place our mark in the XY coordinate system (that is, Z = 0), and place the center of the mark at (, 0. This is a good reminder, because the starting point of the system on the left is the center point we mark.

To locate the camera location through a 2-3-dimensional point set, we use solvepnp.

An Implementation of solvepnp:

void cv::solvePnP( InputArray _opoints, InputArray _ipoints,                  InputArray _cameraMatrix, InputArray _distCoeffs,                  OutputArray _rvec, OutputArray _tvec, bool useExtrinsicGuess ){    Mat opoints = _opoints.getMat(), ipoints = _ipoints.getMat();    int npoints = std::max(opoints.checkVector(3, CV_32F), opoints.checkVector(3, CV_64F));    CV_Assert( npoints >= 0 && npoints == std::max(ipoints.checkVector(2, CV_32F), ipoints.checkVector(2, CV_64F)) );        _rvec.create(3, 1, CV_64F);    _tvec.create(3, 1, CV_64F);    Mat cameraMatrix = _cameraMatrix.getMat(), distCoeffs = _distCoeffs.getMat();    CvMat c_objectPoints = opoints, c_imagePoints = ipoints;    CvMat c_cameraMatrix = cameraMatrix, c_distCoeffs = distCoeffs;    CvMat c_rvec = _rvec.getMat(), c_tvec = _tvec.getMat();    cvFindExtrinsicCameraParams2(&c_objectPoints, &c_imagePoints, &c_cameraMatrix,                                 c_distCoeffs.rows*c_distCoeffs.cols ? &c_distCoeffs : 0,                                 &c_rvec, &c_tvec, useExtrinsicGuess );}

_ Opoints is an array of object point sets in the object coordinate system. It should be a STD: vector <CV: point3f> object. Here we pass the labeled 3D coordinate system (a set of four points ).

_ Ipoints is the image point (projection) array corresponding to the object. The parameter should be STD: vector <CV: point2f> or CV: mat-2 x n/n x 2, where N is the number of points, here we pass the points of the detected mark.

_ Cameramatrix: the internal reference matrix of the camera.

_ Distcoeffs: This is the distortion Coefficient of input 4x4, 5x1 or 1X5 vectors (K1, K2, P1, P2, [K3]). If it is null, all distortion coefficients are set to 0.

_ Rvec is the rotation vector that converts a point from the model coordinate system to the camera coordinate system.

_ Tvec is the same as above. Here is the output translation vector.

If useextrinsicguess is true, this function uses _ rvec and _ tvec as the initial approximate rotation and moving vector respectively, and further optimizes the function.

We use this function to calculate the square sum of distance between the observed projection and the expected projection, which will minimize the projection error.

The estimated conversion is composed of the rotation (_ rvec) and the translation component (_ tvec. This is also called Euclidean transformation or rigid transformation.

A rigid transformation is defined as the form of generating a conversion vector T (v) when an assembly is applied to any vector V:

T (v) = R v + T

RT = R-1 (that is, R is an orthogonal transformation), T is the original conversion vector, a rigid conversion satisfies:

Det (R) = 1

This means that R does not produce reflection, so it represents a rotation (one preserving Al orthogonal transformation ).

To obtain a rotation vector of a 3x3 rotation matrix, we will use CV: Rodrigues. This function converts a Rotation Parameter through a rotation vector and returns its equivalent rotation vector rotation matrix.

Note: Because the solvepnp function above finds the position of the camera in the 3-dimensional space, we must convert our results. Therefore, the transformation we will get will describe the tag transformation in the camera coordinate system, which is obviously more friendly to the rendering engine.

The following is estimateposition, which is used to locate the position of the tag detected above.

Void estimateposition (STD: vector <marker> & detectedmarkers) {for (size_t I = 0; I <detectedmarkers. size (); I ++) {marker & M = detectedmarkers [I]; CV: mat revc; CV: mat _ <float> tvec; CV: mat raux, taux; CV: solvepnp (m_markercorners3d, M. points, cammatrix, distcoeff, revc, tvec); raux. convertize (revc, cv_32f); Taux. convertize (tvec, cv_32f); CV: mat _ <float> rotmat (3, 3); // convert the rotation matrix and the rotation vector CV: Rodrigues (revc, rotmat ); m. transformation = Transformation (); For (INT Col = 0; Col <3; Col ++) {for (int row = 0; row <3; row ++) {// copy the rotating element m. transformation. R (). mat [row] [col] = rotmat (row, col);} // copy the transferred element m. transformation. T (). data [col] = tvec (COL);} M. transformation = m. transformation. getinverted ();}}

Rendering 3D virtual objects

At this point, you already know how to discover the tags on the image and calculate their precise positions in space relative to the camera. It's time to draw something. As we mentioned earlier, we will use OpenGL functions to render the scenario. 3D virtualization is the core part of augmented reality. OpenGL provides all the basic conditions to create

Build high-quality rendering.

For details, refer to: CV camera calibration and 3D reconstruction.

Continue Learning in the next section