An Introduction to stereo matching

Reprint Please specify source: http://blog.csdn.net/wangyaninglm/article/details/51531333,
From:
Shiter The art of writing programs

2.1 Parallax Theory

Computer stereoscopic vision system by imitating the human visual system, according to the same scene from the different positions of the two-view or multi-view images, the use of geometric method can calculate the depth of information. This paper mainly studies the binocular stereo vision system as shown in

Dual camera system
In similar triangles and proportional relationships based on corresponding edges:

Where z is the depth of the scene, B is the distance between the camera baselines, and F is the camera focal length. And because bf/z
As a positive number, the matching algorithm is greatly simplified according to the XL>XR of the above formula. In the general algorithm, the parallax in the vertical direction uses the standard stereoscopic camera system, which is the image that has been corrected in the horizontal direction. The parallax in the vertical direction is 0.

2.2 Stereo matching constraints

Because stereo matching is the restoration of three-dimensional information from a two-dimensional image, it has the characteristics of uncertainty, so in order to obtain the correct matching result, it is necessary to reduce the matching search difficulty and improve the matching accuracy by means of various constraint information. There are several types of constraint information commonly used.

2.2.1 Polar Line constraints

A plane consisting of a three-dimensional point and a point in the imaging plane of the two cameras contains a baseline, a plane that connects two camera centers and connects two polar lines, a plane called a polar plane. The P-point and its projection point or and ol constitute the polar plane, the polar plane and the camera imaging plane intersection line is polar line. The projection point PL of the space point P on the imaging plane and the PR are located on the corresponding polar line.
The problem of searching image points is reduced from two-dimensional search problem to one-dimensional search problem in stereo matching field, which greatly simplifies the problem complexity. In the standard stereoscopic vision system, the pole line is collinear with the sweep surface line of the image.

2.2.2 Compatibility constraints

The compatibility constraint, also known as the consistency constraint, requires that in the two images to be matched, the eigenvalues of the corresponding region should have the same attributes within a given threshold or corresponding feature points. When judging whether two primitives in a matching image have compatibility constraints, there are three types of features that you can choose:
1. Pixel features, such as grayscale values
2. Illumination features
3. Geometric features (centroid, shape, boundary, contour)

2.2.3 Uniqueness Constraints

The uniqueness constraint requires that for the image to be matched, there is at most one point in the original image. This constraint simplifies the matching process. Each primitive on an image corresponds to only one primitive on another image, so that each matching primitive in the image can have at most one parallax value

2.2.4 Continuity constraints

2.3 Stereo Matching method

In stereo matching, the matching problem can be considered as the process of finding the correlation between two sets of data [3]. There are many classifications for stereo matching, and the classical partitioning method for matching algorithms in this field is two sets of hierarchies: local matching algorithm and global matching algorithm. The partitioning is based on the scope of the algorithm runtime constraints. Another division is based on the generated parallax map. For all pixels, a dense disparity map is generated to determine the parallax value, and the method becomes a dense match. The method is widely used, such as the synthesis of examples. On the other hand, the sparse disparity map corresponding to the dense disparity map, called sparse matching, has the Parallax value only for the selected pixel (usually corner or edge point), such method is fast, but it needs to process the missing parallax value by interpolation algorithm later, so the application scenario has a great limitation. In this paper, we mainly introduce stereo matching algorithm for classical partition.

2.3.1 Local matching algorithm

The matching algorithm based on local area constraint is used to calculate the local information around a given location, which involves less amount of data and lower computational complexity, and most real-time platform uses the idea of this algorithm. But its effect on non-texture, parallax discontinuity and occlusion area matching is not satisfactory. Local algorithms can be divided into two categories: feature matching algorithm and region matching algorithm.
* (1) Feature matching
The method first extracts the features from the image to be matched, uses the similarity measure and some constraints to determine the geometric transformation, and finally acts on the image to be matched. These methods mainly include feature extraction, feature matching, model transformation, interpolation and disparity refinement and several other steps.
The feature matching algorithm is based on the characteristics of the image to be matched: gray-scale change, edge, illumination, etc., the corresponding relationship is established, and the process of disparity map is obtained according to the interpolation algorithm. Feature matching is insensitive to image noise, insensitive to occlusion, small in computation and short in computational time, but can only obtain sparse parallax maps, which is prone to loss of precision in interpolation operation and poorly matched to low texture regions.
* (2) region matching
This class method uses matching windows to measure the similarity of all sub-region matching windows to determine the corresponding region. There are two problems in region matching, one is the selection of similarity criterion and the other is window selection. Some common criteria for regional matching are:


The greater the value of the cross-correlation metric and normalized cross-correlation measure, the higher the similarity, the smaller the remaining value indicates the higher the similarity. In the window selection aspect, the key factor that affects the matching effect is the size of the matching window, the window is too small to contain enough brightness information, so that the ratio of brightness change and image noise is very small, so that the error rate increases; window is too large, the edge of the view is not very good embodiment, and the calculation is increased, while the matching effect

2.3.2 Global optimization Matching algorithm

The solution of image problem can be considered as the maximum posterior probability solution under Markov random field frame, and further converted to the solution of energy minimization problem. The global matching method first constructs an energy function, in the form of which the data item describes the matching degree, the smoothing term embodies the constraint of defining the scene, the dynamic Programming (DP), the confidence Extension (BP), the graph Cut (GC), the simulated annealing (SA), the Scan line optimization (so), the cooperative algorithm (CA) and other optimization algorithms can be used as a method to solve energy minimization. Dynamic programming, confidence expansion and graph cutting are the most common methods.
* (1) Dynamic planning
The method uses the sequential constraints on each scan line to treat the matching energy function as the minimum cost distance from the beginning to the end of the scan line. The cost of the optimal path is the sum of the cost of all sub-paths, and the matching cost of the points passed by these sub-paths can be determined by the area correlation metric operator.
The dynamic programming algorithm decomposes the problem into multiple-stage decisions. Multiple phases are interconnected and make decisions that make the process energy optimal. Optimal matching is obtained by searching the optimal path on the planning plane. Dynamic programming can achieve global optimization in one-dimensional optimization, but because it is matched on the scan line, it has a serious trailing phenomenon between scanning lines.
Because the dynamic programming obtains the best match of each polar line and does not consider the constraint relation between the polar line and the polar line, people join the pole-line constraint to get the minimum value of the energy function between the polar lines. Compared with other optimization methods, the advantage of dynamic programming is that it provides global constraints for those regions which lack texture and is prone to mis-matching, and solves the problem that these regions are difficult to match due to the low local energy values under different parallax. For the occlusion problem, the energy of the occlusion part is replaced by a fixed value in the dynamic programming, then the occlusion is detected by the consistency constraint. The disadvantage of the dynamic programming method is that the error match may be extended along the kernel line to cause other correct matching failures, so there are often stripes appearing on the parallax map obtained by using the dynamic programming method.
* (2) Confidence expansion
The confidence extension algorithm was first proposed by Pearl in 1988, and after 1999 it was widely used in various fields of computer vision to solve the optimization problem of graph structure with ring and got good results. The algorithm can converge to the optimal solution for the graph structure without ring, but it is not guaranteed to converge to the optimal solution for the graph structure with ring. At present, the research focus of this algorithm is how to improve the efficiency of the algorithm. Sun et [25] In 2003, the confidence extension algorithm was applied to stereo matching and achieved good results, in 2005, Sun and so on in the algorithm added visibility constraints to detect occlusion phenomenon. Felzenszwalb and other [26] proposed a hierarchical confidence extension algorithm, which improves the speed of the confidence expansion algorithm from many aspects. Yang et [38] uses the hierarchical confidence extension algorithm to realize the occlusion detection. Tappen and freemanl[28] respectively with the graph cut and the confidence extension to the same parameters of the Potts Model Markov random field optimization, the conclusion is that the confidence expansion than the results of the graph cut smoother, faster than the graph cut, but the energy is higher than the graph cut, the effect is quite.
* (3) Figure cut [8][9][23][24]
In recent years, with the application of Graph optimization algorithm in computer vision, the minimization problem of energy function based on graph cutting has been paid much attention. ROY[18] The first application of graph cutting algorithm to stereo matching, and through experiments show that the graph cutting algorithm can effectively overcome the shortcomings of other global optimization algorithms (such as dynamic programming algorithm, such as the generation of disparity map generated horizontal stripe defects), to avoid the disparity in the adjacent polar line discontinuity problem. But the contour edge of the disparity map is blurred, and the disparity layer is low in sensitivity. Boykov and KOLMOGOROV[15] use specific constraints to construct energy functions, and to minimize the energy function by the improved maximum flow method, the graph cutting algorithm is applied to stereo matching problem, and the compact disparity map with good effect is obtained. (And it is proved that the minimum and global minimum values obtained by the graph cutting method in the energy minimization time difference is a known constant) but the method constructs the network graph to generate a large number of nodes, resulting in a high degree of space complexity, at the same time, the algorithm operation process needs multiple iterations, high complexity, can not achieve real-time computing requirements. In order to improve the matching speed li[19] A stereo matching algorithm based on non-overlapping parallax region segmentation is proposed, and the energy minimization of the segmented block is used to minimize the pixel-level energy minimization of the algorithm, and the time complexity is reduced, but there is a glitch at the edge of the generated parallax image. BLEYER[20] such as the use of images in each segment of the disparity has a smooth character, the image segmentation based on the stereo matching algorithm of the general algorithm. However, this method can not get the optimal allocation of pixel-level, and the complexity is high and the computational amount is large. Bleyer and Rother[21] based on the existing low-scale segmentation, the image is segmented into a super-pixel form to reduce the graph cut algorithm generation node stereo matching method. A new stereo matching method based on object segmentation is proposed, assuming that the same object is compact, connected and the surface parallax changes smoothly. Although the method has good effect on object segmentation and Parallax acquisition, it lacks the depth information of texture in the inner region of object and background, and the area between objects is not accurate parallax annotation.
In the above literature, the stereo matching method based on image segmentation, because the use of automatic non-interactive color image segmentation method will be the same parallax region to separate or hide part of the image details, resulting in segmentation error, and eliminate the error need to introduce other methods, such as by introducing the initial parallax estimation [20][21] and other methods, But these methods increase the overall complexity of the stereo matching algorithm, and the segmentation information is not used effectively. In order to obtain the fine disparity map of the area of interest in the practical application, a stereo matching method based on interactive image segmentation is proposed, aiming at the shortcomings of the previous stereo matching algorithm based on image segmentation, which has a large computational capacity, and does not make full use of the information of segmentation results. This method obtains the interest target by the interactive graph cut method in the image segmentation, and only the stereo matching for the object of interest, so the computation amount is greatly reduced, and the global optimal characteristic of the original graph cutting algorithm is preserved.

2.4 occlusion

The invisible problem in stereo matching is due to the geometric structure of the scene and the occlusion of the object in the scene. Occlusion refers to the fact that some points in a scene are visible in one camera because of the relationship between the scene and the camera, but not in the other camera. As the depth of the scene is different from the camera farther away from the scene may be close to the scene of the camera to block, and thus can not form an image and, due to changes in the viewpoint, the scene of the occlusion area may change, a point of view visible under another point of view may become occlusion area, this phenomenon is called semi-occlusion phenomenon. Almost all of the occlusion phenomena in computer vision belong to semi-occlusion. The non-binocular visibility of occlusion problem and the accompanying surface discontinuity make it not satisfy the requirement of the three-dimensional vision, and it is a difficult and important point in the research of stereoscopic vision. In recent years, people have made a lot of researches on the detection and measurement of occlusion area, and even the correct depth estimation of the restoration of occlusion area. Some solutions for occlusion problems are proposed.

In the actual matching system, in order to overcome the occlusion problem, the depth interpolation operation is needed for the occlusion point.

2.5 Occlusion Detection method

Cross check = left and right check for LRC (poor results in scenes with low spatial frequency structure)

DOT order constraint (ORD)
If the order of the matching points is different in two images, then the matching point in the scene is the occlusion point. (The whole has the lowest false rate and the lowest trigger rate)

Occlusion constraint (OCC)
Assuming that the discontinuities in the Parallax map are occlusion areas, to find the occlusion area, only the discontinuous regions in the Parallax map need to be found. This process is performed two times: once with the left image as a reference, and another with the right image as a reference.

Evaluation method of 2.6 stereo matching

The error match rate is defined as follows [3]:

Among them, for the whole image of the number of pixels, for the calculation of the disparity map, for the real disparity map, in the comparison, the standard real parallax map only takes the same part as the partition template, the rest is set as the background, for the tolerance value of the error, generally for the range of positive integers in the Parallax callout 1.

[1] Bai Ming, solemn, Wang Wei. Research and development of binocular stereo matching algorithm [J]. Control and decision, 2008, 23 (7): 721-729. doi:doi:10.3321/j.issn:1001-0920.2008.07.001.
[2] The race stem inside the gram. Introduction to three-dimensional computer vision technology and algorithms [m]//Defense Industry Press, 2014.
[3] Cyganek B, Siebert J. An Introduction to 3D computer Vision techniques and Algorithms[j]., 2009.
[4] Yin, Liu, Song Jianzhong. An improved stereo matching algorithm based on image segmentation [J]. Journal of Computer Aided design and graphic Science, 2008, 6th (6): 808-812.
[5] Wang Baofeng, Zhou Jianliang, Tangaroa, et. The visual location method of the Chang ' e third Tour device [J]. Science China: Information Science, 2014, 04 (04): 452-460.
[6] Yin, Liu, Song Jianzhong. An improved stereo matching algorithm based on image segmentation [J]. Chinese Journal of Computer Aided design and graphic science, 2008, 20 (6): 808-812.
[7] Zhu Daixian. Research on workpiece positioning and grasping based on binocular vision [J]. Computer Measurement and control, 2015, 19 (1): 92-94.
[8] Gu Yu, Chan Weixian, Chen Qian, et. Rapid head detection method based on binocular stereo vision [J]. China Laser, 2014, 01 (01): 150-155.
[9] Zhu Sujie, Zhou, Liu Zhongyan. A phase-based stereo matching algorithm [J]. Industrial instrumentation and automation devices, 2013, 2nd (02): 101-104.
[10] Yang Q. A non-local Cost aggregation method for stereo matching[c]//PROCEEDINGS/CVPR, IEEE Computer Society Conference on Compu ter Vision and Pattern recognition. IEEE Computer Society Conference on Computer Vision and Pattern recognition. 2012:1402-1409.
[11] Yang Q, Ji P, Li D, et al. Fast Stereo matching using adaptive guided Filtering[j]. Image and Vision Computing, 2014, 32 (3): 202-211.
[12] Yang Q. Hardware-efficient Bilateral filtering for stereo matching[j]. Pattern analysis and Machine Intelligence, IEEE transactions on, 2014, 36 (5): 1026-1032.
[13] Yang Q. Stereo Matching Using Tree filtering[j]. Pattern analysis & Machine Intelligence IEEE transactions on, 2015, 37 (4): 834-846.
[14] Geiger A, Roser M, Urtasun R. Efficient Large-scale stereo Matching[m]//computer VISION–ACCV 2010. Springer Berlin Heidelberg, 2011:25-38.
[15] Boykov Y, Kolmogorov v. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision[j]. IEEE Transactions on Pattern analysis and Machine Intelligence, 2004, 26 (9): 1124-1137.
[16] Bleyer m, Gelautz m. graph-cut-based stereo matching using image segmentation with symmetrical treatment of occlusions[j]. Signal processing:image Communication, 2007, 22 (2): 127-143.
[17] Bleyer M, Rother C, Kohli P, et al. Object Stereo-joint stereo Matching and object segmentation[a]//IEEE Conference on Co Mputer Vision and Pattern recognition[c], June 21-23, Colorado, usa:3081-3088.
[18] Roy S, Cox I J. A Maximum-flow Formulation of the N-camera stereo correspondence problem[a]//IEEE International Conference on computer Vi Sion[a], 1998 January 4-7, Bombay india:492-499.
[19] Hong L, Chen G. segment-based stereo matching using graph cuts[a]//IEEE Conference on computer Vision and Pattern Recogni tion[c],2004 June 27-july 2,washington DC usa:74-81.
[20] Bleyer m, Gelautz m. graph-cut-based stereo matching using image segmentation with symmetrical treatment of occlusions[j]. Signal processing:image Communication, 2007, 22 (2): 127-143.
[21] Bleyer M, Rother C, Kohli P, et al. Object Stereo-joint stereo Matching and object segmentation[a]//IEEE Conference on Co Mputer Vision and Pattern recognition[c], June 21-23, Colorado, usa:3081-3088.
[22] Tang M, Gorelick L, Veksler O, et al grabcut in one cut[a]//IEEE international Conference on Computer vision[c], De C 01-08, Sydney, Australia 1769-1776.
[23] Wang years, Fan Yi, Bowen and so on. Image matching algorithm based on graph cutting [J]. Journal of Electronics, 2006, 34 (2): 232-236.

Reprint Please specify source: http://blog.csdn.net/wangyaninglm/article/details/51531333,
From:
Shiter The art of writing programs

An Introduction to stereo matching