Image processing Image splicing---panorama video stitching

First, the principle of introduction

Image stitching is a technology that makes use of real image to compose Panorama space, it will mosaic multiple images into a large scale image or 360 degree panorama, and the image splicing technology involves computer vision, graphics, digital image processing and some mathematical tools. The basic steps of image stitching mainly include the following aspects: Camera calibration, sensor image distortion correction, image projection transformation, matching point selection, panorama image stitching (fusion), and brightness and color equalization processing, the following steps are analyzed.

Camera Calibration

Due to the installation design, as well as the difference between cameras, there will be a zoom between video images (lens focal length inconsistency caused), tilt (vertical rotation), azimuth difference (horizontal rotation), so the physical differences need to be pre-calibrated, to obtain a consistent image, easy to follow the image stitching.

The relationship between the camera's motion and the imaging results is shown.

Figure 1: The relationship between the camera's motion and the imaging results

Image coordinate transformation

In practical applications, the acquisition of panoramic images often requires cameras to be arranged in different positions and with different inclination angles. For example, due to airborne or in-vehicle characteristics, the camera arrangement is not the same, it can not guarantee that the camera on the same surface, such as the cylindrical projection is not necessarily on the same cylinder, the plane projection is not necessarily on the same plane; in order to avoid blind spots, the camera will tend to tilt down a certain angle. These conditions are more common, and easy to be neglected, the direct projection and splicing effect is poor. It is therefore necessary to obtain a coordinate transformed image based on the position information and angle information of the camera before all images are projected onto a cylinder (or plane).

In theory, if any of the two conditions of a stationary three-dimensional image or a planar scene are satisfied, the corresponding relationship of the two images can be represented by a projection transformation matrix, in other words, if any of these conditions are satisfied, the image taken by one camera can be represented by a coordinate transformation as an image taken by another virtual camera.

In general, the perspective projection transformation of the 8 parameter is best suited to describe the coordinate relationship between the images, where the matrix of 8 parameters is [M0,M1,M2;M3,M4,M5; m6,m7,1]; the camera motion corresponding to each parameter is shown as follows:

2 shows the camera downward tilt a certain angle to shoot the image, this angle and M6 and M7 have a corresponding relationship, if you want to get the correction image, only 8 parameter matrix to reverse the coordinate transformation.

(a) Original image

Figure 2: (a) original image; (b) x-directional deformation effect; (c) Tilt correction effect

Image Distortion correction

Due to manufacturing, installation, technology and other reasons, the lens has a variety of distortions. In order to improve the accuracy of camera stitching, the distortion of imaging lens must be considered when image stitching is performed. The general distortion is divided into internal distortion and external distortion, and the internal distortion is caused by the distortion of the structure of the photographic itself, and the distortion of the geometric factor of the external deformity into the projection mode. Lens distortion belongs to internal distortion, and the distortion caused by lens can be divided into two types, radial distortion and tangential distortion. The radial distortion is the aberration of the set optics, which is caused by the different radial curvature of the lens, there are two kinds of barrel distortion and pillow type distortion. Tangential distortion is usually caused by the non-collinear optical center of the lens lens group, including various generation errors and assembly errors. In the imaging process of optical system, the radial distortion is the main factor that causes image distortion. Radial distortion causes the image to be curved in a straight line, and the closer The edge is, the more obvious the effect is. According to the mechanism of radial distortion, the video image is corrected. As shown in effect 3 (b), the corrected image, whose effective pixel area is reduced, can generally be corrected by electronic amplification, as shown in 3 (c).

Figure 3: (a) The original acquisition of the image, (b) an image corrected by a radial distortion, (c) a magnified image

Image projection Transformation

Since each image is photographed at different angles, they are not on the same projection plane, and if the overlapping images are seamlessly stitched together, the visual consistency of the actual scene will be destroyed. So we need to make a projection transformation of the image before stitching. There are usually planar projections, cylindrical projections, cube projections, and spherical projections.

Plane projection is a picture in the sequence image of the coordinate system as the benchmark, the image is projected into the datum coordinate system, the adjacent image overlapping area alignment, said the resulting stitching is a plane projection splicing, the cylindrical projection refers to the collected image data is re-projected to a camera focus radius of the cylinder face, The projection stitching of panorama is performed on the cylinder, the spherical projection is the characteristic of human eye observation, and the image information is projected into the eyeball part by perspective transformation, and the projection is constructed into a spherical surface, and the cube is designed to solve the disadvantage that the data in spherical innuendo is not suitable for storage, and a projection stitching method is developed. It is suitable for computer-generated images, but it is more difficult to take pictures of real-scene images. As shown in 4, the image stitching process.

Figure 4: Image stitching Process

Selection and calibration of matching points

Because the method of feature points is easier to deal with the transformation relation between image rotation, affine, perspective and so on, it is often used, which includes the corner point of the image and the point of interest which shows some singularity in relation to its domain. Harris has proposed a corner detection algorithm, which is generally accepted as a good corner detection algorithm with rigid transformation invariance, and has affine transformation invariance to some extent, but the algorithm does not have the invariant of scaling transform. In view of such shortcomings, Lowe proposes a SIFT feature point with scale invariance.

Figure 52 The matching feature points in the selected image

As shown in 5, image stitching needs to find an effective feature matching point in the image sequence. The feature point of the image is to find the accuracy and efficiency directly affecting the image stitching. For the image sequence, if the number of feature points ≥ 4, it is easy to automatically calibrate the image matching point, if the feature point is very small, image stitching often can not achieve a more ideal effect.

Image Mosaic Fusion

The key two steps of image stitching are: registration (registration) and fusion (blending). The purpose of registration is to register the image in the same coordinate system according to the geometric motion model, and the fusion is to combine the registration image into a large mosaic image.

In the process of multi-image registration, the geometric motion model mainly includes: translation model, similarity model, affine model and perspective model.

The translation model of an image is the displacement of the direction and direction of the image only in the two-dimensional space, and the translational model can be used if the camera only has a translational motion. Image similarity model refers to the camera itself in addition to translational motion may also occur in rotational motion, at the same time, in the presence of the scale of the scene can also be used to describe the zoom factor multi-scale motion, so when the image can be translated, rotated, scaled motion, you can use the similarity model. Image affine model is a 6-parameter transformation model, that is, with parallel lines transformed into parallel lines, the finite point mapping to the general characteristics of the finite point, the specific performance can be a uniform scale transformation coefficients of all directions of uniformity or transformation coefficients inconsistent non-homogeneous and scale transformation and shear transformation, can describe the translational motion, Rotational motion and small scale and distortion. The perspective model of image is a transformation model with 8 parameters, which can express all kinds of table exchange perfectly and is the most accurate transformation model.

Image fusion technology can be divided into two categories: non-multiresolution technology and multi-resolution technology. In non-multi-resolution technology, there are mean value method, Hat function method, weighted average method and median filter method. The multi-resolution technology includes Gaussian pyramid, Laplace Pyramid, contrast pyramid, gradient pyramid and wavelet.

Figure 6 Cylindrical panorama image stitching

(a)-(d) an image of four different viewing angles, (e) A cylindrical panorama image for final splicing

Equalization of brightness and color processing

Because the camera and the light intensity difference, will cause an image inside, as well as the brightness between the image is uneven, after stitching the image will appear the alternating light and shade, so as to the observation caused great inconvenience.

Brightness and color equalization processing, the usual way is through the camera's light model, correction of an image of the interior of the illumination Inhomogeneity, and then through the adjacent two images overlap the relationship between the two images of the histogram mapping table, through the mapping table of two images to do the whole mapping transformation, Finally achieve the overall brightness and color consistency.

Ii. current situation at home and abroad

Panorama stitching Reconnaissance system has earlier research abroad, as early as 1992, Cambridge University's L.g.brown on the core technology of image stitching, 1996, Richard Szeliski of Microsoft Research, proposed a motion-based panorama stitching model. Szeliski later published a number of this paper, 2000 Shmuel Peleg proposed an improved method, according to the camera's motion mode Adaptive selection splicing model, 2003 M.Brown published the SIFT feature for image stitching method, but the calculation is very large, 2007 Seong Jong ha proposed a mobile camera system panorama stitching method, not only to ensure the effect, but also good operation speed.

In the domestic aspect, there are many universities research institutions on video splicing technology and application research, among them, Shanghai Kai Vision into Information Technology Co., Ltd. Research and development of "panoramic visual situational Awareness System" the most representative, the system has perfect function, advanced technology, reliable performance, and has been successfully applied to a variety of models.

Shanghai Kai Vision Information Technology Co., Ltd. Panoramic Vision situational Awareness System--pvs9112 is a real-time, all-solid-state non-mechanical motion of HD 360-degree gaze video system, the system provides real-time continuous coverage of the entire battlefield of full-motion video, human-computer interface intuitive and fast. The system adapts to harsh environments, supports color and infrared sensors, dark and daytime work, real-time image processing and high-definition video display, the display interface simultaneously provides 360-degree panoramic windows and an area of interest HD images. The system provides a development-based structure that facilitates integration with other systems such as radars to obtain a complete situational understanding.

Sensor Head:

Figure 8 Several sensor heads

Graphical interface:

Figure 9 PVS9112 graphical interface

Characteristics:

• Real-time 360-degree video panorama display for situational awareness, security monitoring, target detection. So as to improve the platform's attack ability and security protection.
• Supports HD color and infrared sensors. Can work day and night.
• Display the global stitching screen, local interest.
• Graphical interactive interface. and support a variety of human-computer interface, support touch screen, mouse, keyboard, custom keys, joystick, etc., can seamlessly access the existing system.
• No moving parts, high reliability.
• Adapt to the bad working environment such as vehicle and vessel.

Optional Features:

• Target Detection and alarm
• Automatic multi-target tracking
• Video recording and playback
• The optional support PTZ long distance photoelectric detection system, in the Panorama video can touch the way to control the rapid rotation of PTZ to the specified position, overcome the traditional control PTZ mode defects, make PTZ camera efficiency greatly improved.
• Image fog Enhancement algorithm
• Electronic image Stabilization algorithm

Open architecture:

Second, the application

From the actual application of image splicing, there are mainly large aerial photos, satellite image splicing, vehicle system monitoring, virtual scene realization, video compression; many of the materials mentioned in-car system splicing, such a splicing system can be used for different vehicles, such as anti-terrorism, security surveillance, reconnaissance, patrol and police cars, etc. , the system provides the operator with real-time panoramic images around the vehicle, so that it can perceive a comprehensive and rich situation, while manipulating the vehicle can effectively protect itself in the car, without the windshield of the vehicle can be controlled in real time. Panorama images greatly enhance the user's visual perception system, which has a broad market prospect in special vehicles, military and civil. Compared with the traditional multi-screen monitor, the Panorama stitching screen is more consistent with human eye observation, which greatly improves the accuracy of reconnaissance. However, the airborne system is seldom mentioned, so the application foreground is huge as long as the design is reasonable.

Application Examples:

Way One: Basic mode

Camera Group + one or two terminal, support record or not record, each terminal display content can be different.

Way two: Enhanced mode

Camera Group + 2 terminal, can be extended through the GigE network to connect other devices, such as video recorder. The acquisition and preprocessing module realizes the capture of the video signal and manages the camera group, such as PTZ control. In this layer, the implementation of different ways, different interface forms of the camera support. and make the necessary preprocessing functions, such as scaling translation projection transformation, data compression, etc., to prepare the data for subsequent processors. The module also distributes video data to a number of different processors or other devices, such as video recorders, through multiple gige networks. This composition structure can be adapted to different application requirements: such as different camera types and quantities, terminal processing function requirements and so on.

About the video splicing product introduction

http://blog.csdn.net/shanghaiqianlun/article/details/12090595

Image processing Image splicing---panorama video stitching