Linear decoder-solve the data Scaling Problem

Sparse self-Encoder works well, but it has an inherent defect that the range of input data must be scaled to (0, 1 ).

 

Imagine that if an image directly inputs a value greater than 1 into a sparse self-Encoder without preprocessing, it will be in the range of (0, 1) after being sigmoid, or in the range of (0, 1. In this way, the residual data cannot be calculated because the data after the sigmoid transformation cannot be equal to the input data.

 

However, if we replace the third-layer sigmoid function with the constant function, this will change.

 

In this way, we can use real-value input without the need to scale down the sample to the (0, 1) range. For example, this is difficult to achieve in processing color images and PCA whitening.

 

Therefore, we only need to slightly modify the residual of the last layer of the sparse self-encoding algorithm.

 

The Code is as follows:

 

Function [cost, grad, features] = sparseautoencoderlinearcost (Theta, visiblesize, hiddensize ,... lambda, sparsityparam, beta, patches) W1 = reshape (theta (1: hiddensize * visiblesize), hiddensize, visiblesize ); w2 = reshape (theta (hiddensize * visiblesize + * hiddensize * visiblesize), visiblesize, hiddensize ); b1 = theta (2 * hiddensize * visiblesize + 1:2 * hiddensize * visiblesize + hiddensize); b2 = theta (2 * hiddensize * visiblesize + hiddensize + 1: End); Cost = 0; w1grad = zeros (SIZE (W1); w2grad = zeros (SIZE (W2); b1grad = zeros (SIZE (B1); b2grad = zeros (SIZE (B2 )); numpatches = size (patches, 2); a2 = sigmoid (W1 * patches + repmat (B1, 1, numpatches); a3 = W2 * A2 + repmat (B2, 1, numpatches); % change the sum (A2, 2)/numpatches; penalty =-sparsityparam. /rho+ (1-sparsityparam ). /(1-rock); delta3 = (a3-patches); % change delta2 = (W2 '* delta3 + BETA * repmat (penalty, 1, numpatches )). * A2. * (1-a2); cost1 = sumsqr (a3-patches)/numpatches/2; cost2 = (sumsqr (W1) + sumsqr (W2) * lambda/2; cost3 = beta * sum (sparsityparam * log (sparsityparam. /rock) + (1-sparsityparam) * log (1-sparsityparam ). /(1-rock); Cost = cost1 + cost2 + cost3; w2grad = delta3 * A2 '/numpatches + Lambda * W2; b2grad = sum (delta3, 2) /numpatches; w1grad = delta2 * patches '/numpatches + Lambda * W1; b1grad = sum (delta2, 2)/numpatches; Grad = [w1grad (:); w2grad (:); b1grad (:); b2grad (:)]; endfunction sIgM = sigmoid (x) sIgM = 1. /(1 + exp (-x); End

The image is colored, so there are 3 channels, the input dimension is 8*8*3, the original image sample 1

Figure 1

After zca is used, the image becomes sharp.

Figure 2

With Linear decoder, we can learn 400 features 3.

Figure 3

Welcome to the discussion and follow up on this blog, Weibo, and zhihu personal homepage for further updates ~

Reprinted, please respect the work of the author and keep the above text and link of the article completely. Thank you for your support!