K-L Transformation

The K-L transform (Karhunen-loeve Transform) is a transformation based on statistical characteristics, some of which are referred to as hotelling (Hotelling) transformations, since he first gave the method of transforming discrete signals into a series of unrelated coefficients in 1933. The outstanding advantage of the K-L transformation is that it is good to correlate and is the best transformation in the sense of Mean square error (Mse,mean square error), which occupies an important position in data compression technology.

K-L (karhunen-loeve) Transformation form

Set x= (X1,X2,...,XN)T is an N-dimensional random vector, mx=e (X) and cx=e{(X-MX) (X-MX)T} are their mean and covariance matrices respectively, and EI and λi are CX-based eigenvectors and corresponding eigenvalues, where i=1,...,n, and the eigenvalues have been sorted in descending order, That is λ1≥λ2≥ ... ≥λn, the K-L transformation is: [1] Y=a (X-MX) (1.1) wherein the behavior of the transformation matrix A CX eigenvalue, namely: in the formula: Eij represents the first I eigenvector of the J component. The property of the K-L transform ①y the mean vector to zero vector 0. namely: My=e{y} =e{a (X-MX)}=0 (1.2) The ②k-l transform makes the vector signal each component irrelevant, namely the transformation domain signal covariance is the diagonal matrix. ③k-l Inverse transform is: x=a-1y+mx=aty+mx (1.3) ④k-l transform is a transformation with the least distortion under the mean square error criterion, so it is also called the best transformation. This property is related to compression coding. The meaning is that if only the first n coefficients of the transformation in the data transmission of the vector, then according to the n coefficients obtained by the recovery value can be obtained the minimum mean square error, the value is: The above formula shows that, under the K-L transformation, the minimum mean square error value equals the minimum n-n variance of the vector signal in the transform domain. In particular, if the mean value of these components is zero, the mean square error can be minimized if the components are zeroed at the time of recovery. The K-L transform of the image signal is a one-dimensional transformation, when the image signal is transformed, the vector can be a picture or a sub-image in an image. The correlation between the components of a vector reflects the correlation between pixels. In order to get vector x, you can arrange the pixels of an image or sub-image in the order in which rows are connected or columns are connected, as shown in 1. (a) row-to-row (b) column-to-phase diagram 1 The vector signal is established by two-dimensional image signal after the vector signal is established, the covariance matrix CX is computed, and then the characteristic vectors are computed to obtain the K-L transformation matrix A. Thus, although the K-L transformation has the properties (2) and (4) The best de-correlation and error performance, it is not easy to solve eigenvalue and feature roots, especially when the dimension is high, and the transformation matrix is related to the content of the image, so it is difficult to meet the requirements of real-time processing. However, the K-L transformation in the transformation coding has theoretical guidance, people through comparison, to find some performance and K-L transformation approach, but the implementation is much easier "quasi-optimal" coding method.

The clustering transformation thinks that the important component is the component that can make the distance within the transformed class smaller. The distance within the class is small, which means that the group is tightly held. But, hug the group to hold tight, really must be easy to classify?

1, according to the principle of the cluster transformation, we want to leave a small variance of the component, the variance is large (large fluctuations) of the component discarded, so two ellipses are projected to the y -axis, so tragic, two overlapping together, the fundamental inseparable. Another situation can do so, the weight of the variance is thrown away, so the x -axis projection, very smooth can be separated. Therefore, the clustering transformation is not always successful.

Figure 1

The K-L transformation of Cuikulaxiu

The K-L transformation is the "best" transformation in theory: it is the best transformation in the sense of Mean square error (mse,meansquare error), which occupies an important position in data compression technology.

Another problem with clustering transformations is that they must be dealt with in a class, transforming each class into separate groups.

K-L transformation to put all the categories together to transform, hope that through this one-time transformation, so that they are divided enough to open.

The K-L transformation thinks: All kinds of holding group tight not necessarily good distinction. The goal should be how to make the distance between classes larger, or make different classes better differentiated. Thus corresponds to 2 K-L transformations.

First: The K-L transformation of the optimal description (dimensionality reduction along the distance between the classes)

Let's start with an example of two-dimensional class two, as shown in 2.

Figure 2

If a clustering transformation is used, the direction is the direction of the least variance, so the direction of descending dimension is projected, and the distance between the 2 classes is the distance between the 2 red lines, but this is not the farthest projection direction. The ellipse is projected into the direction and the distance between the 2 classes is 2 green lines. This direction is a feature vector obtained from the statistical averaging of autocorrelation matrices.

There are a total of M categories, and the prior probabilities of each type are

To represent a vector from class I . The self-correlation matrix for class I clusters is:

The autocorrelation matrix R for a mixed distribution is:

Then we find the eigenvectors and eigenvalues of R :

Sort the eigenvalues in descending order (note the difference from the clustering transformation)

In order to descend to m -dimensional, the first m eigenvector is taken to form a transformation matrix A

This completes the best description of the K-L transformation.

Why is the K-L transformation the best transformation in the sense of Mean square error (mse,meansquare error)?

which represents the J component of the n -dimensional vector y , representing the first characteristic component.

The error introduced

The mean square error is

Eigenvalues starting from m+1 are the smallest, so the mean square error is minimized.

The above method is called the best description of the K-L transform, which is the direction of the large distance between the classes, so that the mean square error is the best.

In essence, the K-L transformation of the best description throws away the most insignificant features, however, the salient features are not necessarily helpful for classification. Our goal is also to find out the characteristics of the role of classification, rather than the strength of these characteristics themselves. This is the birth of the 2nd kind of K-L transformation method.

Second: the optimal distinction of K-L transform (mixed whitening after extraction characteristics)

In view of the above problems, the optimal distinction of K-L transform first mixed distribution whitening, and then according to the degree of separation of the characteristic value of the order.

The process of K-L transformation with optimal distinction

First, the self-correlation matrix R of the mixed distribution

Then we find the eigenvectors and eigenvalues of R :

The above is the spindle transformation, which is actually the coordinate rotation, which has been introduced before.

Make transform Matrix

Then there are

This function is an albino R Matrix, which is a coordinate scale transformation, equivalent to rounding the ellipse into a circle, as shown in 3.

Figure 3

Take the two-class mixed distribution problem as an example.

The characteristic vectors and eigenvalues of class two are calculated separately, and there are

The eigenvector of the two is identical, and the only difference is the characteristic root, and also the negative correlation, that is, if the descending order is arranged in ascending order.

In order to obtain the optimal distinction, the eigenvalues of the two are sufficiently different. Therefore, it is necessary to discard those features whose eigenvalues are close to 0.5, while preserving the large ones, and selecting m eigenvector as the principle

Then the total optimal distinction of the K-L transformation is:

K-L Transformation