[Translation] Application of robust scale-unchanged feature matching in Remote Sensing Image Registration (robust scale-invariant feature matching for Remote Sensing Image Registration)

Li qiaoliang, Wang Guo, Liu Jianguo, Member, IEEE, and Chen shaobo

Received on July 15, and modified on July 15. The first version in February 2, 2009; the current version was published in April 17, 2009. This project is funded by China National Basic Research Project 60672060.

State Key Laboratory of pattern recognition and AI, Huazhong University of Science and Technology, Wuhan, China, 430074 (Email: [email protected]; [email protected]; [email protected])

Digital Object Recognition number 10.1109/lgrs.2008.2011751

Summary-- When Scale Invariant Feature Transform (SIFT) is applied to remote sensing images, if there is a significant difference between remote sensing images and visible images, there will be many false matching feature points. In order to achieve robust feature matching in remote sensing images, a scale-direction joint restriction standard is proposed. The feature description operator is redefined to overcome the difference between the gradient intensity and the direction of the remote sensing image. The results of multi-time phase, multi-spectrum and multi-sensor remote sensing images show that the proposed method has a higher matching rate and accuracy than the method based on Brightness and sift.

Index words-feature matching, image registration, scale-unchanged Feature Transform (SIFT), and joint restriction criteria of Scale-direction.

 

1. Introduction

Image registration [1] is an important step in remote sensing image analysis, such as remote sensing image fusion, environmental monitoring, change monitoring, and map update. Many methods have been kicked out to automate image registration. These methods can be summarized into the following two categories.

1) pixel brightness-based methods: Among these methods, the most representative similarity measurement is cross correlation (CC) [1] and mutual information (mutual information, mi) [2]. However, when the image center has a large rotation or scaling displacement, the method of Interconnectivity is insufficient. At the same time, due to the high complexity of global optimization, the method based on mutual information is not suitable for real-time applications.

2) image feature-based methods: these technologies extract features from the image, such as edge, corner points, outlines, and regional centers, and then use the correlation between these features to achieve optimal arrangement between images. However, registration robustness cannot be guaranteed only when sparse features are used. Manual assistance is often required when these algorithms are used. Otherwise, the correct matching rate (CMT) will become very low.

Figure 1. multi-spectral, multi-temporal, and multi-sensor remote sensing image pairs. (a) Daedalus band2. (B) Daedalus band4. (c) Landsat thematic mapper (TM) in 1986. (d) Landsat TM in 1988. (e) Landsat TM. (f) spot.

So far, automatic registration of remote sensing images with large-scale translation, rotation, and scale transformation remains a challenge. Recently, scale-unchanged Feature Transform (SIFT) has been successfully used for registration and recognition of visible images, this is because it has good immutability in image scale transformation and Rotation Transformation, as well as partial immutability in brightness and camera angle of view changes. In addition, PCA-SIFT [8], csift [9] And gloh [10] on the basis of Sift improvement to make it more efficient. However, when we use the sift-based method to align remote sensing images, many feature points may be incorrectly matched, and thus the accuracy rate decreases sharply. The root cause of the problem is that the brightness of the same region in the remote sensing image may become significantly different for some reason, such as the exposure time, spectrum, and sensor differences, and the brightness ing relationships between image pairs can be linear, non-linear, or uncertain (figure. 1 ). In order to solve this problem, Yi et al. [11] proposed the method of SR-SIFT, the criterion of scale restriction is used in the process of feature matching. They claim that this method improves performance when matching visible and infrared images. However, because there is a significant difference between the image acquisition equipment in the spectrum range and the hardware equipment, the correct rate of matching using SR-SIFT is also significantly reduced.

In order to realize the robustness of feature point matching in scale-unchanged feature transformation of remote sensing images, we propose a joint criterion of Scale-direction to propose a large number of key points for incorrect matching. In addition, the feature description of the key points of remote sensing images is redefined. Compared with the brightness-based registration algorithm and SR-SIFT, the proposed algorithm fully improves the matching accuracy while maintaining the high alignment precision of remote sensing images. In many application scenarios, the geometric distortion of remote sensing images can be modeled using "shape distortion" [1] (only translation, rotation, and scale transformation) without generating large errors, we also used this simple model to estimate the transformation parameters between two key point sets, and at the same time, it has better computing performance. If necessary, more complex models such as perspective transformations are also used for image registration algorithms.

 

2. Problems in applying sift matching to remote sensing images

The sift-based registration algorithm has three main models: Key Point Extraction, feature description, and feature matching algorithms. The key Extraction Tool uses Laplace to approach Gaussian difference filter, and detects local extreme points in the scale space as candidate key points. In each candidate location, more complex and detailed models are used to detect accurate positions and scales. Low-contrast or edge key points will be rejected. Finally, each vertex is represented by a four-dimensional vector (XI, Yi, σ I, θ I) T, where (XI, Yi) represents the position, σ I indicates the scale and θ I indicates the main direction. The descriptive operator allocates a 128-dimension feature vector for each key point based on a 4x4 subarea (2. The feature matching algorithm uses the least Euclidean distance between feature descriptors of key points to determine the nearest neighbor to find the corresponding feature points of another image. For more information about sift, see [7.

 

Figure 2. Sift description operator. (a) Fig 1 (B) Fig 2

(C) Cartesian grid (d) Polar Grid

 

 

 

Table 1 Comparison of accuracy rates

To ensure correct matching, Lowe [7] suggests that the ratio between the nearest neighbor and the next nearest neighbor should be less than a threshold value. This metric performs well in visible light, because in order to achieve trusted matching, correct matching needs to be closer than the latest false matching. However, when used for remote sensing images, this significance is greatly weakened due to the brightness relationship of remote sensing images.

When there is no significant difference between the image spectrum or sensor, sift is often unable to align the image.

 

3. Robust Sift

A. Redefine the feature description Operator

In a remote sensing image, the brightness and gradient of the pixel are significantly different, and the gradient orientation pits of the same region in the image are completely opposite. These differences may lead to different main directions of the two key points, and the main direction is used for correct matching measurements. To make the feature description more robust in these differences, we redefine the main direction of each feature point, which is the primary factor for feature matching. First, we use the following formula to change the gradient direction of each pixel,

,

Where, R (?) Indicates the original direction? . Then, we use adaptive technology to give each key point more directions than sift. This technology can be described as follows:

,

HK indicates the value of K in the gradient histogram. In the gradient histogram, the direction greater than th is considered as the main direction. As shown in 2 (d), we use the gloh [10] circular graphic block to create a feature vector. The gradient value only contributes to the histogram around it, to achieve a large local offset. After the purified descriptive operator is used, the precision of P-A, P-B and P-C is increased from 61%, 95%, 56% to 72%, 98% and 62%, respectively.

To test the feature descriptive operators used in formulas (1) and (2), we calculate the relative main orientation, rmo ). Compared with figure 4 (a), when the absolute value of the Offset is smaller than 5, the curve of Figure 4 (c) is essentially smooth and maintains near 0 relative to the main direction.

B. robust feature matching

Considering that there is usually no local distortion in the remote sensing image center, the registration algorithm in this paper introduces the "shape preserving ing" model [1]. Under a model, the rotation angle and scale factor of all key points are the same. the proposed scale-direction joint restriction criterion makes full use of this feature to extract the key points of correct matching.

Assume there are two key point sets c = {C1, C2,..., extracted from the image center ,..., cm} and c = {C' 1, C' 2 ,..., c'n}, where each key point Ci and the corresponding point c' I, while the I and I represent their respective main directions, Si and S' I represent their respective scales, r? * I indicates the relative direction. At the same time, the appearance (scale) error of the Key Point pair (CI, c' I) is defined [14] And the RMO error is

,

In formula, S * and R * are the current estimated values of the Scale Factor and rotation angle. To consider the influence of appearance error and RMO error, we define the Joint Distance (Joint Distance, JD) of the Key Point pair (CI, c' I)

,

In the formula, Ed (I) indicates the Euclidean distance of the feature vector of the Input key point. When the apparent and RMO errors of key points are both zero, the joint distance is equal to the Euclidean distance. When the key-point pairs of remote sensing images are matched, when different brightness ing between image pairs leads to finding the key point of incorrect matching with the smallest Euclidean distance, a false match will occur. However, in many cases, incorrect matching points do not have the smallest Euclidean distance, the same scale, and the same primary direction at the same time; the correct matching point usually has the same scale, the same main direction, and a relatively small Euclidean distance. This means that the Union distance between the pairs is the minimum when the point is correctly matched.

The key idea of the scale-direction joint restriction criterion is to use the joint distance as the distance measurement rather than the Euclidean distance. The matching algorithm can be summarized into three steps.

1) matching preprocessing: For each key point, use the minimum Euclidean distance of its feature vector to find its corresponding key points, in addition, the ratio of the nearest neighbor to the next nearest neighbor is used to exclude incorrect matching (as used in SIFT ). We call this processing a Euclidean Distance Ratio (Euclidean distance ratio, ED-R) filter, using td to represent the ratio of the threshold. In this way, the key point pair set CC is obtained, and then the CC scale ratio histogram and RMO Histogram can be created. Then, use the Adaptive Threshold Value in formula (2) to find the peak values in the two histograms, and then register all the values of the two peaks {S *, R *} with the serial number k = ,... K sorting (5 ).

2) Matching: The detailed processing process is shown in Figure 6. JD-R Filters use JD as the ED-R filter for distance measurement, and DK is the JD mean of the UDF point set.

Find the smallest DK to obtain the optimized set number *, where k is 1, 2 ,..., K.

3) matching post-processing:Unless the two images are absolutely the same, no detected feature points in an image will match correctly. Therefore, after the JD-R filter processing, the * will still have a wrong match.

The first step is to obtain all candidate Clustering Centers for the scale factor and rotation angle. The matching process in step 2 uses these candidate values as the input for JD-R filtering, and then looks for a trusted clustering center by minimizing DK. When there are many false matching points, the highest peak of the histogram is directly used as the clustering center in SR-SIFT [1], and the method proposed in this paper can select the correct and reasonable clustering center, therefore, it is conducive to subsequent processing.

 

C parameter selection and model estimation

Since Lowe [7] uses TD = 0.8 and mikolajczyk [14] uses Ts = 0.4, in step 1 of our experiment, the TD value is 0.8, in the second step, take a large value of 0.9 to obtain as many key candidates as possible. In step 3), we assign a TS value of 0.4. For remote sensing images, TR = 0.5 (radians) is enough to strictly limit the RMO error. Finally, we use the mdsac [12] algorithm to eliminate isolated points and achieve faster convergence than the ransac algorithm. The least square method is used to obtain the most suitable radiation parameters between the associated image pairs.

 

4. Experiment and Result

In image processing, such as Motion Object Recognition and real-time remote sensing, there is no information about sensor parameters, rotation of large angles between image pairs, and scaling and translation, so these variables are used in the experiment to test the capabilities of the proposed algorithm. This typical method is more widely used in the application field and can avoid additional hardware or software overhead of remote sensing devices when obtaining and processing these parameters.

 

A. Image Set

To test the performance of the proposed algorithm, we divide remote sensing images into three image sets:

1) P-A image set: 200 multi-spectral image pairs from the same sensor, the sensor has Landsat TM and Daedalus. See figure 1 ().

2) P-B image set: 50 multi-spectral image pairs from Landsat TM sensor at different times. See Figure 1 (B ).

3) P-C image set: 50 multi-spectral image pairs from Landsat TM and spot, respectively. See Figure 1. (c ).

1. All remote sensing images are normalized to 512x512 grayscale images. Dislocation parameter of the image pair: the translation range is-100 ~ 100, the rotation range is 0 ~ 180 °, with a scale factor of 0.5 ~ 2.
2. B. Evaluation Criteria
3. Root-mean-square error (RMSE) and correct matching rate CMT are used as the evaluation criteria for evaluating the performance of the proposed method.
4. 1) RMSE: approximately 30 test-point (TP) pairs are manually selected in each test. Of the 30 points selected, 20 are used for training, and the remaining 10 are used for testing. Calculate the root mean square error of the 10 point pairs and determine the accuracy of the matching based on the algorithm described in this article. Use 20 training points to calculate the root mean square error of transformation model parameters as reference baseline standards. Use the following formula to calculate the root mean square error of the given two points set, that is
5. ,
6. Here (XI, Yi) and (x' I, y' I) are the coordinates of the I point, u = λ cos θ, V = λ sin θ, the parameter vectors (λ, θ, △x, △y) represent the scale factor, rotation angle, and offset. For each test example, perform the test five times and take the 5 average value as the final RMSE.
7. 2) CMT: the ratio of the number of correctly matched image pairs to the total number of image pairs.
9. Note: Only when the RMSE value of the image pair is smaller than 4 is the correct matching image pair.

C. Experiment results

The algorithms proposed in this article are compared with the CC [1], arrsi [12], sift [7], and srft [11] algorithms. Among the four algorithms, the CC algorithm is implemented based on [1], while the other three algorithms are based on the help of public code and authors. Table 1 shows the correct matching rate CMT obtained by these algorithms. Figure 7 shows the registration result of the synthesized image of the image pair in Figure 1.

50 image pairs are selected from the image pairs determined to be correctly registered by all test algorithms for RMSE calculation. Among them, 40 pairs from the P-A image set, 5 pairs from the P-B image set, the remaining 5 pairs from the P-C image set.

As shown in figure 8, the results of the algorithm proposed in this paper have higher alignment accuracy than those of the other three methods, and the benchmark value recently calculated by the selected test point pair. Considering that SR-SIFT is more accurate than sift, we didn't use the RMSE value of Sift to make the image clearer.

 

V Conclusion

In order to achieve automatic registration of remote sensing images, problems encountered when using sift for Remote Sensing Image Matching are analyzed first. To overcome the Brightness Difference between remote sensing image pairs, we redefine the feature description of each key point. In addition, we propose a criterion named "scale-direction" to achieve robust feature registration of key candidates. The experimental results show that the proposed method is compared with the brightness-based method, such as sift and SR-SIFT, it has a higher matching rate and registration accuracy, so it is helpful for the subsequent processing of remote sensing image fusion and analysis.

 

Thank you

Thanks to the Untitled reviewers for their helpful comments and suggestions. Thanks to Mr. A. Wong for his guidance in arrsi [12] and kovesi for providing the implementation phase consistency and the Open Source Matlab code for implementing sift [7] provided by vedaldi. Finally, I would like to thank Professor Cao Zhiguo for his guidance on SR-SIFT [11.

 

References
[1] B. zitova and J. flusser, "Image Registration methods: A Survey," image is. comput ., vol. 21, no. 11, pp. 977-1000, Oct. 2003.
[2] J. inglada, "Analysis of artifacts in subpixel Remote Sensing Image Registration," IEEE Trans. geosci. remote sens ., vol. 45, no. 1, pp. 254-264, Jan. 2007.
[3] C. hsu and R. beuker, "multiresolution feature-based image registration," In Proc. spie-Visual communicationsand image processing, 2002, Vol. 4067, pp. 1490-1498.
[4] C. y. wang, H. sun, S. yada, and. rosenfeld, "Some experiments in relaxation Image Matching Using corner features," pattern recognit ., vol. 16, no. 2, pp. 167-182,198 3.
[5] H. li, B. manjunath, and S. mitra, "a contour-based Approach to Multisensor Image Registration," IEEE Trans. image Process ., vol. 4, No. 3, pp. 320-334, Mar. 1995.
[6] Y. sheng and C. a. shah, "automatic image registry for hydrologic change detection in the lake-rich Arctic," IEEE geosci. remote sens. lett ., vol. 5, no. 3, pp. 414-418, Jul. 2008.
[7] D. g. lowe, "distinctive image features from scale-invariant keypoints," Int. j. comput. VIS ., vol. 60, no. 2, pp. 91-110, Nov. 2004.
[8] Y. KE and R. sukthankar, "PCA-SIFT: a more distinctive representation for local image descriptors," inproc. int. conf. comput. VIS ., washington, DC, 2004, pp. 506-513.
[9]. e. abdel-Hakim and. a. farag, "csift: A Sift descriptor with color invariant characteristics," inproc. int. conf. comput. VIS ., 2006, pp. 1978-1983.
[10] K. mikolajczyk and C. schmid, "a performance evaluation of local descriptors," IEEE Trans. pattern Anal. mach. intell ., vol. 27, No. 10, pp. 1615-1630, Oct. 2005.
[11] Z. yi, C. zhiguo, and X. yang, "multi-spectral Remote Image Registration Based on sift," electron. lett ., vol. 44, No. 2, pp. 107-108, Jan. 2008.
[12]. wong and D. a. clausi, "arrsi: Automatic registry of remotesensing images," IEEE Trans. geosci. remote sens ., vol. 45, no. 5, pp. 1483-1493, May 2007.
[13] K. mikolajczyk, T. tuytelaars, C. schmid,. zisserman, and J. matas, "A coparison of affine region detectors," Int. j. comput. VIS ., vol. 65, no. 1, pp. 43-72, Nov. 2005.
[14] K. mikolajczyk and C. schmid, "Scale and Affine Invariant interest point detectors," Int. j. comput. VIS ., vol. 60, no. 1, pp. 63-86, Nov. 2004.

[Translation] Application of robust scale-unchanged feature matching in Remote Sensing Image Registration (robust scale-invariant feature matching for Remote Sensing Image Registration)