Bumblebee2-Technical description of Binocular Vision System

U can read from: http://www.uptech-robot.com/

Also u can get the documents from me. Send me the private message

Content of this article:

AboutBumblebeePrinciple of stereoscopic vision

AboutBumblebee demoThe meanings of various parameters in the program and how to adjust them explain why the Deep Image and Reconstruction3DInvalid pixels in the image

The reading method of this article: The red font is the key

 

 

Stereoscopic Vision

 

 

 

 

This article will try to describe the stereoscopic vision technology. After reading this article, you will have a deeper understanding of how data flows in the system and all the adjustable parameters. This allows you to customize your system to complete special tasks.

 

 

 

 

Basic Principles of stereoscopic vision

 

The task of stereoscopic vision is to complete the measurement of space, which is based on the images collected by cameras with spatial offsets. The processing process of stereoscopic vision can be divided into the following three steps: establish the correlation between image features of the same scene obtained from different observations. Calculates the relative displacement of the same feature in each image.

Based on the geometric structure of the camera, the feature is determined relative to the camera's 3D
Location

 

Consider the following two images. These two images are taken from a pair of triclops camera models with horizontal offset. We can define two vertices A and B in the two images. PointAleft
Corresponding to PointAright, Same, PointBleftCorresponding to PointBright.

Using a ruler, if you measure the horizontal distance between a point and the image edge, you will find that the distance on the left is greater than the horizontal distance between the corresponding point and the image edge in the right image. For example, the distance between the edge and the phone receiver in the left image is greater than that between the edge and the phone receiver in the right image. We can determine the distance from the phone receiver to the camera model based on this distance (also called Parallax.

 

We define the difference between the values of the same features in the left and right axes in their respective coordinate systems as Parallax. You will find that in the two images, the distance from the top of the image to the matched features is exactly the same, because the camera is horizontally arranged, so there is only horizontal bit shifting.

 

The parallax of feature a is defined as D (A) = x (Aleft)-X (aright), and B is D (B) = x (bleft) -X (bright), in which X (Aleft) is the X axis coordinate of the Aleft point.

 

If you calculate the values of D (A) and
D (B), you will find D ()
<D (B), which indicates a in this scenario.
Point B

To be far away.

 

 

 

Establish related

The triclops library uses the absolute correlation deviation and method to establish the correlation between images. The principle of this method is as follows:

In the reference image, select a neighborhood based on the given square size and compare the neighborhood along the same line with a series of neighborhoods in the other image to find the best match.

End

 

The following formula compares the neighborhood values:

 

DMax

M m

2 2

Min Sigma

I right[X+I] [Y+J] −Ileft[X
+I+D] [Y+J]

D=DMin

I= −J= −

2 2

M m

Where:

 

DMin andDMax is the minimum and maximum parallax (disparity)

 

MIs the template size (mask size)

 

I rightAndIleftIs the right image and the left image

 

 

 

 

Calculating the distance from an object to the camera can be determined by the parallax and the camera's geometric structure. The distance between the matched features is the parallax, the length of the lens focal length, the CCD Resolution, and the baseline length (the distance between the two cameras.

 

The triclops Library provides a function for converting depth ing into a distance image.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The following equation shows the relationship between the parallax and the distance and the relationship between the world coordinates and the image coordinates of a single camera:

 

Z=BF;X=U;
Y=V

D Z F

 

Where:

(X, y, z) is 3D
Position in the world coordinate system

F indicates the focal length of the camera.

(U, v) is like Coordinate

D is the difference value.

B is the baseline length between cameras.

 

 

 

TriclopsData Flow in the database

 

 

Shows the data flow in the triclops database. The system first obtains raw from the camera model.
Format Image, and finally it is processed into a deep image. There are two main processing modules in the system. The first processing module is a preprocessing module that applies low-pass filtering, Image Correction, and edge detection. The second processing module is used for stereo matching, result validation, and sub-pixel interpolation. The final result is a deep image.

 

 

 

Preprocessing (Pre-processingTo correct an image, it is very important to smooth it in advance. Therefore, if you want to correct an image, it is good to open the low-pass filter in advance. Of course, the image can be corrected without a low-pass filter, but the corrected image may be confused. If you want to increase the processing speed, you can turn off the low-pass filter.

Calibration (Rectification)

Correction is used to correct the distortion produced by the lens. The distortion caused by the lens can be seen in the original image. For example,

A straight line in the scene will change to a curve in the original image, which is particularly evident at the corner of the image. Correction aims to correct such type of distortion. Furthermore, if the correction is not correct, the feature search along the row or column may produce incorrect results.

 

Edge Detection (Edge DetectionEdge detection is an optional feature that uses brightness changes to match features. This feature is useful when the camera in the system has an automatic gain function. If the automatic gain of each camera is inconsistent
While the absolute brightness is inconsistent, the brightness change is a constant. Therefore, edge detection is suitable for environments with great illumination changes. Although edge detection can improve the detection results, it is equivalent to introducing another processing step. Therefore, we need to weigh the relationship between the improvement status and speed of the results to use this function.
Note that the validation function is only valid in edge detection mode.

 

Stereo processing (Stereo Processing)

Stereo processing uses the absolute correlation deviation and algorithm described earlier.

 

Parallax range (Disparity rangeThe parallax range is the range of pixels that the Stereo Algorithm searches for the best match in two images. In triclops
In general, the parallax of 0 pixels represents an object in an infinite distance. The maximum parallax defines the nearest object that can be detected. We need to set the appropriate parallax range based on the actual task. Reducing the parallax range can speed up the system operation and reduce the chance of mismatching.

 

Related templates (Correlation mask) Related templates are a square neighborhood of a pixel. This pixel is the image element that the system wants to search. We can set the size of this template. Large templates generate more intensive and smoother depth images. However, there is a lack of accuracy in recognizing discontinuous depth locations. On the other hand, the images produced by small templates are sparse and
There will be more noise, but it will be superior in locating the discontinuous depth. To generate the same results, the size of the template must be proportional to the resolution of the image to be processed. To obtain comparable results
5x5 template,
When the image is 320x240, the template is increased to 9x9. The size of the template must be odd. 3x3, 5x5, and 7x7 are valid template sizes, while
4x4, 6x6, 8x8
Is invalid template size. Triclops provides the maximum
15x15 and minimum
1x1 template. In addition, triclops
A new experiment function is provided: troclopssetanystereomask, which allows you to set templates of any size.

 

Confirm (ValidationIn some cases, for example, the closed texture does not have a changed texture, so it is impossible to establish the correlation between images. If the related establishment is incorrect, the measurement result is incorrect. To avoid incorrect measurements, two methods are introduced:

Texture validation is based on the texture level of the template, and determines whether the difference value is valid. If the texture level is insufficient to produce a correct match, This pixel will be declared invalid.
Uniqueness validation indicates whether the optimal match for a specific pixel is good enough to compare with other matches in this template. Even if the quality of the template meets the requirements

The matching is not correct. If the related results are not strong enough, this pixel will be declared invalid.

You can set two thresholds to control the degree of rigor of validation-one is texture and the other is uniqueness.