Understanding the principle of PCA and C++\matlab realization __c++

1. Principle of PCA

The main component analysis is often used in the field of image processing, and the advantage is that the dimension of the data to be analyzed is reduced, but the main information of the data can be retained. It works like this, for a given set of data (column vectors):

When it is centered, it is represented as:

where u is the mean value of the input column vector. The central data is U1 in the first spindle (both the main direction of the data, this assumes that the direction of the unit vector distribution is the most open, that is, in the U1 direction of the absolute value of the sum of the maximum (also can say the variance is the largest), the method of calculating the projection is the X and U1 do the inner product, because only need U1 direction, So set U1 is the unit vector. Which is the maximization of the following:

And that is maximizing:

The square can take away the absolute value symbol, the smooth curve is easy to handle. Two vectors do the inner product can be transformed into matrix multiplication form, so the object function bracket inside is the matrix multiplication to represent the inner product, transpose the row vector multiplied by the column vector to get a number. Because the transpose of a number is the same as itself, the objective function can be converted to:

After the above process has removed the absolute value symbol, the original problem conversion into a peer problem, then you can use the multiplication of the matrix to the matrix multiplication of the definition of the equation to further merge, so you can remove the parentheses, To remove a variable that is U1 (the largest principal direction previously defined) and I, you can get it out of the sum:

In fact the parentheses inside is a matrix multiplied by its own transpose. The first column of the X matrix is XI, and the target function is finally translated to:

In the absence of the front 1/n, it is a standard two-second type. and xx ' (in order to facilitate, with the ' expression transpose ') matrix is a semidefinite symmetric matrix (the specific proof process can refer to matrix teaching materials). Because it is a conditional extremum, the Lagrangian operator is directly guided: The optimization problem can be converted to:

The objective function and the constraint condition constitute a maximization problem (previously assumed U1 is the unit vector):

To construct a Lagrangian function:

The extremum of the function can be obtained by the derivative number of the U1, because the optimization function is a strict convex function, then it gets:

It can be seen from the upper-form that U1 is the characteristic vector corresponding to xx ' eigenvalue λ. All eigenvalues and eigenvectors of XX ' are satisfied with the above formula. So, to get the maximum eigenvalue, the target value is the highest. Therefore, in the process of image compression, select the largest number of eigenvalues, you can compress the image, cut off the relatively "redundant" part.

As for the use of PCA, should be the PCA method in face Recognition feature extraction and data dimension, for the input 200*200 size of the face image, extract its gray value only as the original feature, then this original feature will reach 40000 dimensions, which will bring great difficulty to the processing of the following classifier. The famous face recognition Eigenface algorithm uses the PCA algorithm, uses a low Wizi space to describe the face image, simultaneously uses the information which the recognition needs to save. The Gaussian face detection algorithm based on K-L transform has high detection rate, good real-time performance and strong robustness, and has important practical significance for the intelligent development of monitoring system.

There are two theories applied in PCA Principal component analysis: Maximum variance theory and minimum error theory. In signal processing, it is considered that the signal has a large variance, the noise has a small variance, and the signal-to-noise ratio is the variance ratio of the signal to the noise, the bigger the better. Therefore, it is considered that the best K-dimensional feature is the conversion of n-dimensional sample points to K-dimensional, and the variance of samples on each dimension is very large. The minimum error theory is to select the appropriate eigenvector to minimize the difference between the restored image data and the original data, and to select a balance between the data dimensionality reduction and the loss of the data. 2. Implementing 2.1 based on OPENCV implementation OPENCV functions involving PCA are mainly:

The main functions of PCA in OPENCV are://Parameters: Inputarray Data raw Matrix//parameter: Inputarray mean original matrix data mean value, input null function will calculate//parameter: int flags per line (C V_pca_data_as_row)/Column (Cv_pca_data_as_col) represents a sample//parameter: int maxcomponents = 0 preserves the number of eigenvalues, the default is fully retained PCA::P CA (Inputarray DATA, Inputarray mean, int flags, int maxcomponents = 0)//parameter: Inputarray data raw Matrix Date//parameter: Inputarray mean original matrix data mean, input empty letter Number will own calculation//parameter: int flags per line (CV_PCA_DATA_AS_ROW)/column (Cv_pca_data_as_col) represents a sample//parameter: Double retainedvariance retain how many eigenvalues, percentages, default  All-retained PCA::P CA (inputarray data, inputarray mean, int flags, double retainedvariance)//Parameters: Inputarray the raw matrix Date//parameter: Inputarray mean original matrix data mean value, input null function will calculate//parameter: int flags per row (Cv_pca_data_as_row)/column (Cv_pca_data_as_col) represents a sample//parameter: Double Retainedvariance retains the number of eigenvalues, percentages, default all pca& Pca::computevar (inputarray data, inputarray mean, int flags, double retainedvariance)//parameter in order: Raw data, the original data mean, input empty will be calculated on their own; each row/column represents a sample; retain the number of eigenvalues, percentages, default all//original images, projected into the new space Mat PCA::p roject ( Inputarray VEC) const//First data after project, reflected to original image C + +: MAt Pca::backproject (Inputarray VEC) const 

OpenCV Test Example:

 const int row_num = 4;
	const int col_num = 4;

	Float array[] = {1.2, 3.2, 4.1, 6.1, 3.2, 0.23, 4.1, 3.1, 5.1, 0.9, 0.12, 1.9, 3.2, 5.2, 9.0, 2.4};
	Cv::mat Pcaset (row_num, Col_num, CV_32FC1, Cv::scalar::all (0));
	float* data = nullptr;
		for (int i = 0; i < row_num i++) {data = pcaset.ptr<float> (i);
		for (int j = 0; J < Col_num; J + +) {*data++ = Array[i*col_num + j];

	} cout << "Origin matrix:\n" << pcaset << Endl; CV::P CA PCA (Pcaset, Cv::mat (), Cv_pca_data_as_row, 3);//parameter in order: Raw data, the original data mean, input null will be calculated on its own, each row/column represents a sample, retain how many eigenvalues, the default all reserved cout  << "to the matrix mean after the PCA transformation of the feature matrix:" << pca.mean << endl;//mean cout << "to the eigenvalue after the PCA transformation to the feature matrix: \ n" << Pca.eigenvalues << endl;//eigenvalues cout << "eigenvalues vector after PCA transformation of feature matrices: \ n" << pca.eigenvectors << endl;/ /eigenvector Cv::mat DST = pca.project (pcaset);//Mapping New space cout << "mapping space after PCA transformation of feature matrices: \ n" << DST << endl;// Levy Vector 

2.2 MATALB Implementation

function [PRO_MATRIX,MEAN_IMAGE]=MY_PCA (train_set,eigen_num)% input:%train_set: Training Sample set, each column is a sample, each row a class of characteristics, Dim*train_num%

Eigen_num: Projection dimension% output:%pro_matrix: Projection matrix%mean_image: mean value image [Dim,train_num]=size (Train_set);
    % when the training sample number is larger than the sample dimension, the direct factorization covariance matrix if Dim<=train_num Mean_image=mean (train_set,2);
    Train_set=bsxfun (@minus, train_set,mean_image);
    
    R=train_set*train_set '/(train_num-1);
    [Eig_vec,eig_val]=eig (R);
    Eig_val=diag (Eig_val);
    [~,ind]=sort (Eig_val, ' descend ');
    W=eig_vec (:, Ind);
    
Pro_matrix=w (:, 1:eigen_num); Else% constructs the small matrix, calculates its eigenvalues and eigenvectors, and then maps to the large matrix Mean_image=mean (train_set,2), and the average train_set=bsxfun of each row (@minus, Train_set,mea   n_image);% per line minus average R=train_set ' *train_set/(train_num-1);   % The elements of the matrix are converted to 200*200 after each divided by 199 [Eig_vec,eig_val]=eig (R);  % Compute eigenvector Eigenvalue eig_val=diag (eig_val);  % get eigenvector [Val,ind]=sort (Eig_val, ' descend ');   % eigenvalues are sorted to get the sort W=eig_vec (:, Ind) of the position size where the eigenvalues are located; % gets the eigenvector corresponding to the eigenvector pro_matrix=train_set*w (:, 1:eigen_num) *diag (Val (1:eigen_NUM). ^ ( -1/2));
Pro = Train_set*w (:, 1:eigen_num) *diag (Val (1:eigen_num). ^ ( -1/2)); End End

3. The result