UFLDL Teaching (iii) PCA and whitening exercise

EXERCISE:PCA and Whitening

No. 0 Step: Data preparation

UFLDL The downloaded file contains the dataset Images_raw, which is a 512*512*10 matrix, which is 10 images of 512*512

(a) data-loading

Using the Sampleimagesraw function, extract the numpatches image blocks from the Images_raw, each image block size is patchsize, and the extracted image blocks are stored in columns, respectively, in each column of the matrix patches, That is, patches (:, i) holds all the pixel values of the first image block

(b) Data de-value processing

The average pixel value of the image block is subtracted from all pixel values of each image block to achieve the de-averaging of the data.

is a randomly selected block of images displayed.

First step: Perform PCA

This section is divided into two parts

(1) PCA calculation, where only the data x is rotated to get xrot, without the extraction of the principal component

In particular:

① covariance matrix Sigma for calculating data x

② the characteristic decomposition of Sigma, using the EIG function of Matlab to get the matrix U of Sigma's characteristic vectors

[U,s,v]=eig (Sigma);

U=[u1,..., UI,..., un], each of its columns is a characteristic vector of Sigma, and N is the feature dimension of the input data

S =diag ([λ1,... λi,..., Λn]) is a diagonal array with the characteristic value of sigma as the diagonal element, and the UI and λi correspond;

In order to follow up the calculation, we need to change the order of each column of U, so that the corresponding characteristic value of each column is descending successively.

The changed matrix is still remembered as u, and the corresponding characteristic value diagonal array is still s, i.e.:

U=[u1,..., UI,..., Un],s=diag ([λ1,... λi,..., Λn]), meet: Λ1>=...>=λi>=...>=λn

③ uses the matrix U to rotate the data x to get Xrot, i.e. Xrot=u ' *x

(2) After the rotation of the data to solve the covariance matrix covar, and visualize it, to observe whether the selected data are correct

PCA guarantees that the covariance matrix of the selected data is a diagonal array, if the Covar is correct

Then its image should be a blue background, and there is a slash in the diagonal position

This shows the covariance matrix Covar utilizes the imagesc of MATLAB, and the function is really powerful.

The function of Imagesc (Covar) is to display the matrix Covar as an image, and the different values in the matrix will be given different colors.

The resulting covariance matrix image is as follows: You can see that the image is diagonally positioned outside the same color as the rest

　　

The second step: the number of principal components satisfying the condition

In this section, find the number of principal components that satisfy the condition K

That is to find the smallest k value, so that (λ1+...+λk)/(λ1+...+λn) > A percentage, such as 99%

Step three: Use the number of principal components found to reduce the dimension of the data

In the second step, the number K has been found, that is, the K-principal component of preserving the data satisfies the requirement

In this step, the data X will be reduced to a dimension, leaving only K principal components, to get Xtidle

At the same time, in order to observe the quality of the data after dimensionality reduction, we can use U (:, k) to transform the data of the reduced dimension to the original dimension, that is to get the approximate recovery data of the original data.

and using the mesh to restore the image displayed, compared with the original image, the following first picture is a reduced dimension after the restoration of the original data from the image, is the corresponding original data, you can find that the data after the reduced dimension can be restored to the original data very similar data

Fourth Step: PCA Whitening + Regularization

This section is divided into two steps

(1) Implementation of PCA with albinism and regularization

First, rotate the data (using the feature matrix U)

The rotated data is then scaled using the eigenvalues to achieve whitening

At the same time, the eigenvalues are fine-tuned by using the parameter ε to achieve regularization when using the eigenvalue scaling.

(b) Calculate the covariance matrix of the data after the hundred, and observe the covariance matrix

If a regularization item is added, the diagonal element of the covariance matrix is less than 1

If you do not include a regular item (that is, rotation + whitening only), the diagonal element of the covariance matrix is 1 (in fact, the ε is a very small number)

Is the image of the covariance matrix corresponding to the albino data, which is the result of joining regularization and not after regularization.

Fifth Step: Zca Whitening

Zca Whitening, is on the basis of PCA whitening made a rotation, that is

The first image below is the result graph of Zca whitening, and the second picture is the corresponding original

As you can see, the result diagram of Zca whitening appears to be the edge of the original image

Below, is the Pca_gen Code of the Section

Clcclearclose all%%================================================================%% Step 0a:load data% Here we Provide the code to load natural image data into x.% x would be a 144 * 10000 matrix, where the kth column x (:, K) Corresp Onds to% The RAW image data from the kth 12x12 image patch sampled.% "do not" need to change the code below.x = sample Imagesraw ();% reads some images from the Images_raw patchesfigure (' name ', ' raw images ');% shows a figure, titled Raw Imagesrandsel = Randi (Size (x, 2), 200, 1); % A random selection of samples for Visualizationdisplay_network (x (:, Randsel)),% display randomly selected image block percent Step 0b:zero-mean the data (by row)% mean and Repmat/bsxfun functions.%--------------------YOUR CODE here---------------- ----X=x-repmat (mean (x), size (x,1), 1), all elements of each column of%x are subtracted from the mean of the column%%===================================================== ===========%% Step 1a:implement PCA to obtain xrot% Implement PCA to obtain Xrot, the matrix in which the data is Expres sed% with respect to the eigenbasis of Sigma, which is the matrix u.%--------------------YOUR CODE here--------------------Xrot = zeros (size (x)); % need to compute this% compute covariance matrix and perform eigenvalue decomposition m=size (x,1);% input sample Count sigma=x*x '/m;% input data covariance matrix [U,s,v]=eig (sigma);% Eigenvalue decomposition of the covariance matrix [S_value,s_index]=sort (Diag (S), ' descend ');% extracts the diagonal elements of S, sorts them in descending order, Sindex is the sorted number U=u (:, S_index); S=diag (s_value);% rotates the data xrot=u ' *x;%% Step 1b:check Your implementation of pca% the covariance matrix for the ' Expre ' Ssed with respect to the basis u% should is a diagonal matrix with Non-zero entries only along the main% diagonal. We'll verify this here.% Write code to compute the covariance matrix, Covar. % when visualised as an image, you should see a straight line across the% diagonal (Non-zero entries) against a blue BAC Kground (zero entries).%--------------------YOUR CODE here--------------------Covar = zeros (Size (x, 1)); % need to compute Thiscovar=xrot*xrot '/m;% the data corresponding to the covariance matrix after the rotation of the data visualise the covariance matrix. You should see a line across the% diagonal against a blue background.figure (' name ', ' visualisation of covariance matrix '); Imagesc (Covar);%%============ ====================================================%% Step 2:find K, the number of the components to retain% Write code to Determine K, the number of the retain in order% to retain at least 99% of the variance.%--------------------Y Our CODE here--------------------k = 0; % Set K Accordinglys_diag=diag (S);    S_sum=sum (S_DIAG); for K=1:size (x,1) sk_sum=sum (S_diag (1:k));    if sk_sum/s_sum>=0.99 break; endend%%================================================================%% Step 3:implement PCA with dimension reduction% now there are found K, you can reduce the dimension of the data by% discarding the remaining dimensions.  In this "You can represent the% data" in K dimensions instead of the original 144, which would save you% computational time when running learning algorithms on the reduced% representation.%% following the DimensiOn reduction, invert the PCA transformation to produce% of the matrix xhat, the dimension-reduced data with respect to the Original basis.% visualise the data and compare it to the raw data.  You'll observe that% there is little loss due to throwing away the principal of that% correspond to dimensions With low variation.%--------------------YOUR CODE here--------------------% de-dimensionality of Data xtidle=u (:, 1:k) ' *x;% take advantage of reduced-dimensionality data Xtid  Le restores the data Xhat = zeros (size (x));  % need to compute thisxhat = U*[xtidle;zeros (M-k,size (x,2))];% visualise the data, and compare it to the raw data%  Should observe that the raw and processed data is of the comparable quality.% for comparison, you could wish to generate a PCA Reduced image which% retains only 90% of the variance.figure (' name ', [' PCA processed images ', sprintf (' (%d/%d dimensions ) ', K, size (x, 1)),;d Isplay_network (Xhat (:, Randsel)), figure (' Name ', ' Raw images ');d isplay_network (x (:, Randsel)) ;%%================================================================%% Step 4a:implement PCA with whitening and regularisation% Implement PCA with whitening and Regularisatio N to produce the matrix% xpcawhite. Epsilon =0.000001;%--------------------YOUR CODE here--------------------xpcawhite = zeros (size (x)); Xpcawhite=diag ( 1./SQRT (S_diag+epsilon)) *xrot;%% Step 4b:check Your implementation of PCA whitening% Check Your implementation of PCA W Hitening with and without regularisation. % PCA whitening without regularisation results a covariance matrix% that's equal to the identity matrix. PCA whitening with regularisation% results in a covariance matrix with diagonal entries starting close to% 1 and Gradua Lly becoming smaller. We'll verify these properties here.% Write code to compute the covariance matrix, Covar. Percent without regularisation (set Epsilon to 0 or close to 0),% if visualised as an image, you should see a red line ACR OSS the% Diagonal (one entries) against a blue background (zero entries).% with Regularisation, should see a red line then slowly turns% blue across the diagonal, corresponding to the one entries slowly% b Ecoming smaller.%--------------------YOUR CODE here--------------------covar=xpcawhite*xpcawhite '/m;% visualise the Covariance matrix. Should see a red line across the% diagonal against a blue background.figure (' name ', ' visualisation of covariance matrix '); Imagesc (Covar);%%================================================================%% Step 5:implement ZCA whitening% now implement ZCA whitening to produce the Matrix Xzcawhite. % visualise the data and compare it to the raw data. You should observe% this whitening results in, among other things, enhanced Edges.xzcawhite = zeros (size (x));%---------- ----------YOUR CODE here--------------------Xzcawhite=u*xpcawhite;%zca Whitening is done on PCA whitening based on a spin% visualise the data, and Compare it to the raw data.% your should observe that the whitened images has enhanced edges.figure (' name ', ' ZCA whitened i Mages ');d ISPLAY_NETWOrk (Xzcawhite (:, Randsel)), figure (' Name ', ' Raw images ');d isplay_network (x (:, Randsel)); 

　　

UFLDL Teaching (iii) PCA and whitening exercise