Camera calibration: Application of Levenberg-marquardt algorithm in camera calibration

The LM algorithm has a total of three applications in camera calibration.

(1) Single-target or double-target, in the case of internal-fixation, the best external parameters are calculated. The corresponding function in OpenCV is FINDEXTRINSICCAMERAPARAMS2.

(2) The best internal and external parameters are calculated when the internal and external parameters are not fixed in a single target setting. The corresponding function in OpenCV is CALIBRATECAMERA2.

(3) In the case of the double target, the internal and external parameters of the camera and left and right camera pose are not fixed, and the best left-and outside parameters of the camera are calculated and the best position and attitude matrix of the camera. The corresponding function in OpenCV is stereocalibrate.

This article reads the premise that you have enough knowledge of the LM (Levenberg-marquardt) algorithm. This paper is mainly about the application of LM algorithm in camera calibration.

The analysis of this paper is based on the source code of OPENCV, so see the OPENCV source to read this article.

0 Variable Settings

Set the number of points on the calibration board is m, the calibration process of shooting n view (for the double target, left and right camera each grab n views).

With regard to camera imaging models and distortion models, I do not occupy space here, see OPENCV official documents or related papers. I use the following function to represent:

where (U,V) is the pixel coordinate, (x, Y, z) is the world coordinates, r= (R1, R2, R3) T is the rotation external parameter, t= (T1, T2, T3) T is the translation external parameter, a= (FX, FY, CX, CY) T is the projection internal reference, d= (K1, K2, P1,P2, K3, K4 , K6, S1, S2, S3, S4, A, B) T is the distortion internal.

As is known to all, (U,V) is the existence of distortion (later referred to as the distortion of coordinates), we use (UU,VV) to represent (U,V) corresponding non-distorted coordinates (referred to as the standard coordinates). The angular coordinates we extracted with findchessboardcorners are treated as standard coordinates.

1 calculate the best external parameters

We need to calculate the best external parameters in the case of a fixed internal reference. The imaging model is then simplified to:

For M corner points there are the following equations:

Where (Xi, Yi, Zi) is known. The Jacobian matrix is calculated as follows:

J and Jtj are available from J.

If the given initial value (R0, T0) can be distorted corner sequence uv= (u1,v1, U2,v2,..., um,vm) T. If findchessboardcorners obtains a standard corner sequence of uuvv= (UU1, VV1, UU2, Vv2, ..., um, VM) T, then the error sequence is E=UV-UUVV. So the final equation is:

After solving the equation, it is more accurate (R1, T1) = (R0, T0)-σ.

Note: There may be some differences with other LM algorithms described above. The introduction document by most LM algorithms should be E=uuvv-uv, (R1, T1) = (R0, T0) +σ. But careful observation will find that the final result is actually the same. I just wrote it in order to correspond with the source code in OPENCV.

2 calculate the best internal and external parameters

In cases where both internal and external parameters are not fixed, the imaging model should be:

is different from calculating the best foreign parameter. Calculating the best external parameters is for a single view, and the optimization goal is to minimize the re-projection error of this view. The best internal and external parameters are calculated for multiple views, and the optimization goal is to minimize the sum of the projection errors of all views. Each view has its own external parameters, but the same internal reference is shared. Therefore, the corner points on each view image are biased only to the internal and external parameters, and the external parameters of the other views are biased to 0. So the Jacobian matrix is:

Among them, Jini and Jexi are the Jacobian matrices of the I-view pairs and their appearance. The Jini is the nx18 matrix (because of 18 internal reference so 18 columns), Jexi is the nx6 matrix (because 6 external parameters so 6 columns), so J is the (n*n) x (18+6*n) matrix.

Can calculate:

3 calculate the best left and right camera's internal and external parameters and the best left and right camera position and pose matrix

The optimization parameters in this step are the pose matrix (R, T) between the two cameras, the internal reference of the two cameras (A1, D1, A2, D2) and the left two-machine external parameter sequence (r1i, t1i). The external parameters of the left camera (r2i, t2i) = (rr1i, rt1i+t). So the corner points on each view of the left camera are biased only to (A1, D1, Ri, Ti), and the partial derivative of the other parameters is 0. The corner points on each view of the right camera are biased only to (R, T, A2, D2, r1i, t1i), and the partial derivative of the other parameters is 0. The biases of (R, T, r1i, t1i) are derived from the bias of the (r2i, t2i) derivative.

Not finished, to be continued.

Camera calibration: Application of Levenberg-marquardt algorithm in camera calibration