Share some OPENCV experience in realizing stereoscopic vision

Try to use OPENCV to achieve stereoscopic vision also has a period of time, the main reference is learning OpenCV 11, 12 chapters and OpenCV forum some senior discussion. The process of bumps, took a lot of detours, and finally solved the biggest problem, the entire calibration, calibration, matching the process of debugging success. (although there are still some questions that have not been clarified yet)

Write this article here, the first aspect is to give oneself a summary, the second aspect is the feeling OpenCV stereoscopic visual aspect of the material or is quite fragmented and incomplete, the novice entry takes a long time to grope out, the third aspect, also is oneself in the process some problems still vaguely, hope can offer.

1. Camera

I use the camera is Taobao buy three-dimensional camera, two USB camera plus an adjustable bracket. Physical photos are as follows

1.1 Three-dimensional camera real-picture

OPENCV drivers for dual USB cameras can refer to the following links

http://www.opencv.org.cn/index.php/capturing images using DirectShow

After copying the above code to your project, you need to set up the project or the compilation environment.

Detailed settings under VC6 can be seen in the code's comments (modify the properties of the project)

The settings in VS2008 can also be referenced in the code comments vc++2005 settings (Modify the compilation environment)

2. Calibration

Because OpenCV in cvstereocalibrate always get very exaggerated results (see below 5.1 problem description), so finally decided to use Bouguet MATLAB Calibration Toolbox Stereoscopic Calibration, and then read the results of the calibration into OpenCV, For subsequent image calibration and matching.

The reference links of Matlab calibration toolbox are as follows:

http://www.vision.caltech.edu/bouguetj/calib_doc/

The above has the detailed use steps, the use is quite convenient.

The following is my personal use of MATLAB Toolbox for stereo calibration steps, for reference, if more detailed steps need to refer to the above link

Copy the MATLAB toolbox files to the corresponding directory, and put the checkerboard map you want to calibrate in the same directory as the. m file, then enter Calib_gui in the MATLAB command Line window and the following window will appear after selecting standard

2.1. Calilb_gui panel

We first calibrate the right camera, so we first copy the board chart from the right camera to the Toolbox directory.

Click on image names, the command line window will prompt you to enter the picture's basename and image format (such as your picture filename is right1, right2, ..., Right10,basename is right), Then MATLAB will automatically help you to read these pictures, as shown in the figure below, you can see, read into the 10 Right camera board chart.

When collecting the chessboard, be careful to make the board occupy as many pictures as possible so that you can get more information about the distortion of the camera.

2.2. Image BaseName Read-in

2.3. Board diagram read-in

Then go back to the main control screen and click Extract Grid corners to extract the corner points of each image.

2.4. Calib_gui panel

After clicking, the command line prompts you to enter the size of the board Corner Search window. The size of the window will be more convenient to extract the corner points (even if the point is deviated from the can be found), but also should be careful not to larger than a square. The remaining two options, as long as the return to choose the default settings can be

2.5. Select Window Size

Then the corner of the extraction work, in a certain order to extract the top corner of the board, the program will automatically help you find all the corresponding corner points

2.6. Extracting Corner points

2.7. Extracting Corner points 2

When extracting the first picture, the command-line window may prompt you to enter the size of the grid, where you can enter the actual size of your squares, for example, if my square is 27mm, enter 27. This step is actually quite critical, it defines the scale of the space, and if the object is to be measured, this step is necessary.

After extracting 10 images in the same way, click Calibration to start the camera calibration

2.8. Calib_gui panel

After many iterations, the program will eventually get the internal and external parameters of the camera, as shown in the following figure (the symbol in the picture is not fully displayed because of the font relationship, the middle question mark is the plus minus sign of the error)

2.9. Calibration iterative process and results

The calibration results can be viewed through the show extrinsic on the panel to verify the results of the calibrated external parameters.

2.10. External parameter diagram

After verifying that the calibration result is correct, click on the Save button on the panel, the program will put the calibration results in a call Calib_result.mat, in order to facilitate the subsequent stereoscopic calibration, the file name is changed to Calib_result_right.mat.

The left camera is calibrated in the same way as the right camera, and after the generated Calib_result.mat it is renamed to Calib_result_left.mat.

After the camera is calibrated, the stereo calibration can begin.

In the MATLAB command line, type Stereo_gui to start the stereo calibration panel, as shown in the following illustration

2.11. Stereo_gui panel

Click Load left and right calibration files and select the default file name (Calib_result_left.mat and Calib_result_right.mat) on the command line to start the run stereo Calibration, the result after run is as follows, the parameters of the left and right camera are corrected, and the rotation and migration relationship vectors between the two cameras (OM and t) are also calculated.

2.12. Stereo Calibration Results

By clicking on the show Extrinsics of Stereo rig on the panel, you can see the dual camera diagram shown in the figure below, and you can see that two cameras are basically forward parallel

2.13. The position relationship between the dual camera and the calibration board

After the stereo calibration parameters are obtained, the parameters can be put into the XML file and then read into the OpenCV with Cvload. The specific method can refer to the example of learning OpenCV 11th, the above is to save the calibration results with Cvsave, and then use the Cvload to read the previous calibration results into the matrix

2.14. xml File Example

Note here is the MATLAB calibration results in the OM vector, this vector is the rotation matrix through the Rodrigues transformation after the results, if you want to use in cvstereorectify, it is necessary to first convert the vector to a rotation matrix cvrodrigues. As for the Rodrigues transformation, the 11th chapter of Learning OpenCV is also explained.

2.15. Rodrigues form representation of the rotation matrix

3. Stereo calibration and matching

With the calibration parameters, the calibration process is simple.

I used the cvstereorectify in OpenCV to calibrate the left and right images of the input using Cvremap after the calibration parameters were obtained. This part of the code refers to the example of Learning OpenCV 12 chapter.

Once calibrated, it is possible to match the stereo. Stereo matching OPENCV There are two methods, one is block Matching, and the other is graph Cut. Block matching uses the sad method, the speed is relatively fast, but the effect is general. Graph cut can refer to Kolmogrov03 's doctoral dissertation, The effect is good, but the speed is really slow to endure. So choose BM.

Here is the parameter setting for stereo matching with BM

Bmstate = Cvcreatestereobmstate (cv_stereo_bm_basic,0); ASSERT (Bmstate! = 0); bmstate->prefiltersize=13; bmstate->prefiltercap=13; bmstate->sadwindowsize=19; bmstate->mindisparity=0; bmstate->numberofdisparities=unitdisparity*16; bmstate->texturethreshold=10; bmstate->uniquenessratio=20; bmstate->specklewindowsize=13;

Where mindisparity this parameter I set to 0 is due to my two camera is forward parallel placement, the same object in the left image must be in the right side of the image, as shown in Figure 3.1 below. Therefore, it is not necessary to set the back-search parameters.

This parameter needs to be considered if the two cameras are deflected inward in order to pursue a larger binocular overlap area.

3.1. Left and right view after correction

Another parameter that needs to be mentioned is uniquenessratio, and I feel that this parameter has a great effect on the final matching result. Uniquenessratio can prevent mis-match, its main effect from the following three pictures of the disparity effect than the pair can be seen. In stereo matching, we would rather not match the region and not be mistaken. If there is a mismatch, it can be cumbersome to detect the application with an obstacle.

3.2. Uniquenessratio is a 0 o'clock match chart, you can see a large area of the wrong match

3.3. Uniquenessratio is a 10 o'clock disparity map, you can see that the mismatch is much reduced, but there is still noise.

3.4. Uniquenessratio for the 20 o'clock disparity map, you can see the error is basically removed, point cloud clean a lot

About CVFINDSTEREOCORRESPONDENCEBM This function of the source code, has done a more detailed study, over a period of time will be written before the code comments collated, send a blog post.

4. Measurement of actual distance

After using CVFINDSTEREOCORRESPONDENCEBM to derive disparity map, it is also necessary to convert the single channel disparity map to the actual coordinate matrix of three channels through the CVREPROJECTIMAGETO3D function.

The specific mathematical principle can refer to the following formula (from Chenyusiyuan http://blog.csdn.net/chenyusiyuan/archive/2009/12/25/5072597.aspx, Some questions about the actual depth are also mentioned in this blog post)

4.1 Distance Conversion formula

However, in the actual operation process, the data obtained by CVREPROJECTIMAGETO3D is not as actually thought, and the world coordinate system defined by the depth matrix is generated. I haven't figured it out yet. This is explained in detail in the following example, and I hope experts in this area can help you:

Figure 4.2 is the actual scene of the measurement, the scene of the main measurement of the three objects is the front of the Tetra Pak box, the middle of the paper cup, and the farthest plastic bottle.

4.2. Position of three objects to be measured in the actual scene

Figure 4.3 is the calibration of the left and right figure and matching out the disparity map,disparity window is the actual point cloud, the object window is to disparity map added a threshold value after the two value graph, mainly to split the foreground and background. You can see that the three objects being measured are basically properly segmented.

4.3. Disparity map obtained by binocular camera

Figure 4.4 is the corresponding three-dimensional information obtained in the disparity window after selecting a point and then in the actual coordinate matrix, where I select one point on the point cloud of three objects to represent the actual coordinate information of an object. (Here's a way to get a little bit of coordinate information from the mouse by referring to the watershed.cpp in OpenCV sample)

4.4. Three-dimensional coordinates of the corresponding point

As you can see here, (265, 156) The coordinates of the Tetra Pak box are (13, 12,-157), (137, 142) The coordinates of the cups are (77, 30,-312), (95, 115) The coordinates of the plastic bottle are (144, 63,-482).

Add: In order to facilitate the display, so the disparity map came out after a 0-255 normalize, so the value of the first one is normalize after the point of gray value, the latter is the actual disparity map before the point of normalize.

Source code by the CVFINDSTEREOCORRESPONDENCEBM algorithm:

Dptr[y*dstep] = (short) (((Ndisp-mind-1 + mindisp) *256 + (d! = 0?) (p-n) *128/d: 0) >> 4); where ndisp is Ndisp = state->numberofdisparities; Mindisp is Mindisp = state->mindisparity; Mind is sad to get the disparity between the actual parallax is about (64-mind-1) *256=1163, the basic is right, the next correction value can be ignored under normal circumstances

I am still not very clear about the stereo coordinate system origin and scale, but from the z-coordinates of these three points can be roughly seen that the distance difference of these three objects is probably 1:2:3, basically with the actual scene of the object in the same position. Therefore, the approximate distance information of the object can be determined by this method.

However, if you measure the distance from the camera parameter itself, it is not very clear, but also to ask this aspect of the Daniel Solution.

5. Some issues

5.1 About Stereocalibrate

OpenCV self-cvstereocalibrate feel not very good, use this function to find the external parameters and rotation translation matrix calibration, often can not achieve line alignment, and sometimes even a more terrible distortion. After looking at Piao's http://www.opencv.org.cn/forum/viewtopic.php?f=1&t=4603 post, also tried to use cvCalibrateCamera2 separate calibration (left and right 20 pictures ), the results are basically the same as the MATLAB alone calibration, and then in the Cvstereocalibrate parameters are set to cv_calib_use_intrinsic_guess, used to refine the internal parameters and distortion parameters, the results of the calibration results are also distorted.

I do not know anyone who has had successful experience in this area, can come out to share. After all, using the MATLAB toolbox is still a bit of a hassle.

5.2 Translation vector and the world coordinate system obtained by stereo matching

Learning OpenCV for translation and rotation.

5.1. Illustration in learning OpenCV

However, it was found in the experiment that if the translation vector is scaled by scale, the left and right views after stereorectify will not change, such as T = [ -226.73817-0.62302 8.93984], to t = [-22.673817 -0.062302 0.893984], the results shown in OPENCV will not change. And if I modify one of these parameters, the change in the left and right view is not the same as the one shown in Figure 5.1 (for example, by narrowing the X, the change in the view is not the shift in the x-axis direction).

So back to the old question, what is the scale of these coordinates? The three-dimensional coordinates obtained by reprojectto3d that function are also the origin point, which is the x, Y, Z axis of the three directions.

Add: The answer to this question comes from a discussion with Maxwellsdemon

His explanation is as follows: rotation is the angle of rotation between the two, but you have to straighten it parallel, also need to translation matrix. You can imagine that two seemingly already parallel cameras, but there is a gap in depth, then in the correction of the meeting translation matrix of the corresponding angle or straight line as the benchmark, the two rotate a small angle, making it completely parallel.

PPS: Webcam URL http://www.hytekautomation.com.cn/BNE001.aspx?productId=20