Three-dimensional Reconstruction Foundation

The three-dimensional reconstruction technique describes the real scene as a mathematical model which conforms to the computer logic expression through the process of deep , preprocessing, point cloud registration and fusion, and surface generation. This model can play an auxiliary role in such studies as cultural relics protection, game development, architectural design and clinical medicine.

1.1 Research background and significance

Man explores and discovers the world through his eyes. Less than 30% of the way humans receive external information comes from receptors such as hearing, touch, and smell, while more than 70%, the richest and most complex information is perceived through visual [1]. Computer vision is an exploration of the computer equipment Eye (camera) and the Brain (algorithm) technology, so that the computer can independently control behavior, solve problems, while perceiving, understanding, analysis of the external environment. In the the 1960s, computer vision was first developed. The focus of this phase is mainly on how to recover the three-dimensional shape of cubic and cylinder from two-dimensional image, and explain the spatial position relation of each object. 1982, David Marr[2] from the perspective of information processing, mathematics, neurophysiology, computer graphics and other disciplines of the research results are summarized, and on this basis a series of computer vision theory. Thanks to this complete and clear theoretical system, computer vision has developed vigorously. Its core idea is to recover three-dimensional structure from two-dimensional image. Figure 1-1 shows the classic Marr visual information processing process.

Figure 1-1 Marr visual information processing process
Fig.1-1 Process of Marr visual information
With the rapid development of science and technology, the application of computer vision has been paid more and more attention in various industries, such as equipment detection and monitoring, medical image processing, cultural relic protection [3], Robot vision, automatic navigation, industrial product design and production and other fields. The computer vision technology brings the opportunity to the people, also brings the challenge. As one of the most popular research directions in computer vision Technology, three-dimensional reconstruction involves several disciplines, including image processing, stereoscopic vision and pattern recognition. Using computer to establish a three-dimensional model to express realistic objective scenery, and to meet the needs of production and life. With the development of industrialization process, the realization of multiple technologies depends on the acquisition of three-dimensional information of target object. Three-dimensional reconstruction has now been used in life and scientific research work, especially in medical treatment, cultural relics protection, game development, industrial design, space navigation and other aspects, showing a strong vitality and influence.

1.2 Three-dimensional reconstruction technology introduction

The focus of three-dimensional reconstruction technique is how to obtain the depth information of the target scene or object. The three-dimensional reconstruction of the scene can be realized only by the registration and fusion of the point cloud data [4] under the condition that the scene depth information is known. Deep application research based on three-dimensional reconstruction model can also be expanded immediately. According to the passive measurement and active measurement [5], the method of acquiring the depth information of the target object is classified, and the two methods are introduced accordingly.

1.2.1 Passive three-dimensional reconstruction technology

Passive general use of the surrounding environment such as natural light reflection, using the camera to obtain images, and then through a specific algorithm to calculate the object's stereoscopic space information. There are three main ways of doing this:

1. Texture Recovery Shape method

The surface of various objects has different texture information, which is composed of texture elements, and the surface direction can be determined according to the texture element, thereby recovering the corresponding three-dimensional surface. This method is called texture recovery shape method [6] (shape from texture,sft).
The basic theory of texture method is as follows: As a visual primitive which is repeated in the image field, the texture element is covered in each position and direction. When an object with a textured element is projected onto a plane, its corresponding texture element also bends and changes. For example, the longer a texture element with a smaller angle to the image plane, the larger the texture element that is closer to the image plane than the perspective shrinkage deformation. Through the image measurement to obtain the deformation, and then according to the deformation of the texture element, the inverse calculation of depth data. SFT is demanding on the texture information of the object surface, it needs to understand the distortion information of the texture element in the imaging projection, the application scope is narrow, only suitable for some special cases such as texture characteristic determination. All are rarely used in practice.

2. Shadow Recovery Shape method

SFS[7] (Shape from shading, restoring shapes from shadows) is also a common method. Considering that the shadow boundary of the image contains the Contour feature information of the image, the depth information of the object surface can be calculated by using the intensity and shading of the image under different illumination conditions, and the three-dimensional reconstruction is carried out with the reflected illumination model. It is important to note that the luminance of the pixel is restricted by including the light source, camera parameters, the target surface material and so on.
Shadow Recovery shape method has a wide range of applications and can restore three-dimensional models of various objects except mirrors. The shortcomings are embodied in the process of mathematical calculation, reconstruction results are not fine, in addition, the SFS method needs accurate light source parameters, including location and direction information. This leads to the inability to apply to situations such as open-air scenes with complex light.

3. Stereo Vision Method

Stereo Vision method [8] (Multi-View Stereo,mvs) is another common three-dimensional reconstruction method. It includes three ways to obtain the distance information directly using the rangefinder, to infer three-dimensional information through an image and to recover three-dimensional information by using two or more images on different viewpoints. By simulating the human visual system, the position deviation between the corresponding points of the image is obtained based on the Parallax principle, and the three-dimensional information is recovered. S.T.BARNARD[9] and other people on the the 1970s to 80 between the three-dimensional reconstruction of the algorithm and evaluation system is outlined. By the middle and late 80, more and deeper visual principles, including stereo measurement methods and depth sensors, have greatly promoted the development of related disciplines. The new method can get the three-dimensional information of the scene directly, and greatly save the material and manpower cost. U.R.DHOND[10] and others proposed a three-mesh three-dimensional constraint method based on hierarchical processing. In the the late 1990s, the frontier algorithms, occlusion processing algorithms and so on were emerged. M.Z.BROWN[11] and others summed up the general situation of three-dimensional visual development from 2000 to 2010, including the correlation analysis of occlusion, registration and efficiency.
Binocular Stereoscopic vision reconstruction, in practical application is superior to other three-dimensional reconstruction method based on vision, also gradually appeared in some commercial products; The deficiency is that the computation is still large, and the reconstruction effect is significantly lower in the case of a large baseline distance.
Representative article: Akimoio T Automatic creation of 3D facial models 1993
CHEN C L Visual Binocular Vison Systems to solid model reconstruction2007
As one of the key technologies of computer vision, stereo Vision method is also its disadvantage. For example, stereo vision needs to assume that the plane of space is a positive plane, but the actual situation is far from this. In addition, there is ambiguity in the match: For some feature points on an image, another image may have a number of similar feature points. So how to choose the most suitable match point, it seems more difficult. 1-2, shows the middlebury[16] data set Teddy and cones scenes of the base color image, standard parallax, and through the graph cuts[17] algorithm obtained by stereo matching parallax estimation results. Although parallax results reflect the three-dimensional location of the scene, there is still a slight gap between the parallax and the standard value of some pixels. In addition, the problems such as the determination of camera motion parameters and the need to obtain multi-frame images for large-scale scene reconstruction greatly affect the deep application of stereoscopic vision.

Figure 1-2 (a) benchmark color image

Figure 1-2 (b) Standard Parallax
Reference: Introduction to Stereo matching

1.2.2 Active three-dimensional reconstruction technology

Active refers to the use of light sources such as lasers, sound waves, electromagnetic waves and other sources of energy emitted to the target object, by receiving the returned waves to obtain depth information of the object. There are four methods, such as Moire fringe method, flight time method, structural light method and triangulation method, which are active ranging.

1. Moire Fringe method

Moire stripes are more common in life, such as two layers of thin silk overlapping together, that is, you can see irregular moire (Morie) stripes; When the breeze blows the window, the stripes also move with it. The Moire stripe method [18], which originated in France in the 18th century, is an ancient and modern method of measurement. The basic principle is to overlap two lines of equal spacing, such as straight or curved clusters, to form moire stripes at a very small angle relative to motion. 1-3, in the main grating and the intersection of the indicator grating, due to the transmission and occlusion of light to produce different shades, namely moire stripes. The moire fringes have vertical displacements with the left and right translation of the grating, and the fringe phase information produced shows the depth information of the object to be measured, and then the reverse demodulation function is adopted to realize the recovery of the depth information. This method has the advantages of high precision and strong real-time, but it is sensitive to illumination and weak in anti-jamming ability.

Fig. 1-3 Double Grating Moire fringe method

Proposed: Wiktin recovering surface shape and orientation from Texture (1987) (Referenced 454 times).
Development: Warren 2010 The Wiktin method is improved using perspective projection;
Liboy 2006 gives a reconstruction method in the case of texture element structure change.
Advantages: High precision, insensitive to light and noise.
Disadvantage: Apply only to objects that have regular textures.

2. Flight Time method

The flight time method [Flight,tof] refers to the method by which the distance is obtained by measuring the flight time interval between the transmitting signal and the receiving signal at the speed of light and sound velocity. This signal can be an ultrasound, or it can be infrared. The flight time method is not limited by the length of the baseline, is independent of the texture, and the imaging speed is faster than that of stereo vision. But it also has some shortcomings. First, the resolution of the TOF camera is very low. As shown in example 1-4, today's highest resolution PMD Camcube 2.0 cameras are only 204x204 pixels, and second, the TOF camera is susceptible to environmental factors, such as mixed pixels, external light sources, and so on, resulting in inaccurate scene depths; Finally, the system error and random error have great influence on the measurement result, The need for post-processing, mainly reflected in the scene pixel location coincident. It is worth noting that the TOF camera is priced at $ tens of thousands of and has a narrower audience.

Figure 1-4 SR4000 tof Camera
Fig.1-4 SR4000 ToF Camera

3. Structured Light method

The structured light method [structured light] emits a ray of characteristic points to an object with smooth surface, and extracts the depth information of the object according to the stereo information in the light source. The specific process consists of two steps, first using a laser projector to project a coded beam to a target object, generating a feature point, and then calculating the distance between the camera's centroid and the feature point based on the projection mode and the geometric pattern of the projected light, thus obtaining the depth information of the generating feature points, and realizing the model reconstruction. This kind of encoded beam is structured light, including a variety of specific styles of points, lines, polygons and other patterns. The structure light method solves the problems of flat surface, single texture and slow gray change. Because the implementation is simple and high precision, so the application of structural light method is very extensive, there are many companies have produced a structured light technology-based hardware equipment, such as PrimeSense Company's prime Sensor, Microsoft's Kinect and Asus Company's Xtion PRO live and other products [21]. Figure 1-5 shows the use of structured light technology to collect three-dimensional information of the scene.
Proposed: Woodham to improve SFS (1980): Photometric method for determining surface orientation from multiple images (this article was cited 891 times)
Development: Noakes: Non-linearity and noise deduction for 2003 years;
Horocitz: Gradient Occasion Control Point 2004;
Tang: Credibility transfer with Markov Random Airport 2005;
Basri: 2007 years of three-dimensional reconstruction under unknown condition of light source;
Sun: Non-Lambert van Meerten 2007;
Hernandez: 2007 Years of color light reconstruction method;
Shi: Self-calibrating photometric stereo Vision method for 2010 years.

Fig. 1-5 schematic diagram of structure light method

4. Triangle Ranging method

The triangulation method [22] is a non-contact distance measurement method based on the triangulation principle. The infrared device casts infrared light at a certain angle to the object, which is reflected after the object is encountered and detected by the CCD (charge-coupled device, charge-coupled Element) image sensor. As the target object moves, the reflected light obtained at this time also produces the corresponding offset value. The distance between the emitter and the object can be calculated based on the emission angle, the offset distance, the center moment value and the position relationship. The triangulation method is widely used in military surveying and topographic prospecting.

Reference documents

[1] Szeliski R. Computer vision:algorithms and Applications[m]. Berlin:springer, 2010.
[2] D. Marr, et al. A computational theory of Human Stereo Vision. Proc.r.soc.lond. 1979, b.204:301-328.
[3] Levoy, M. Pulli, et al. The Digital Michelangelo project:3d scanning of Large statues. proc.siggraph,2000.
[4] Anand A, Koppula H S, Joachims T, et al contextually guided semantic labeling and search for three-dimensional point CLOUDS[J]. The international Journal of Robotics, 2013, 32 (1): 19-34.
[5] Mada S K, Smith M L, Smith L N, et al Overview of passive and active vision techniques for hand-held 3D data acquisit Ion [C]//opto Ireland. International Society for Optics and Photonics, 2003:16-27.
[6] D. Forsyth, J. Ponce, computer vision:a modern approach. Prentice Hall 2001
[7] Horn B. P. Shape from Shading:a method for obtaining the shape of a smooth opaque object from one view. PhD thesis, Department of electrical Engineering, MIT, Cambridge. 1970.
[8] Ikeuchi K. Determining surface orientations of specular surfaces by using the photometric stereo method [J]. Pattern analysis and Machine Intelligence, IEEE transactions, 1981, (6): 661-669.
[9] S. T. Barnard, M. A. Fisehler. Computational Stereo[j]. ACM Computing surveys. 1982, vol.14:553-572.
[J]. R. Dhond, J. K. Aggarval. Struct from Stereo-a Review [J]. IEEE Trans Systems, man, and cybemeties.1989, vol.19:1489-1510.

Reprint Please specify source: http://blog.csdn.net/wangyaninglm/article/details/51558310
From: Shiter The art of writing programs

Three-dimensional Reconstruction Foundation