Deep Learning for Color Image Feature Extraction: Linear Decoder

In the previous discussion, all the incentive functions used are sigmoid functions:

We assume that the output range of the final output layer is [0, 1], and the motivation of Self-encoding learning is to make the output equal to the input, so all input must be adjusted to the range of [0, 1, however, the problem is that the input range of some datasets is easy to adjust, such as minist, but the input of PCA whitening processing does not meet []. therefore, we need to find a mechanism that can handle non-[0, 1] ranges.

To solve this problem, we make the output of the output layer equal to the input of this layer:A(3) =Z(3), which is equivalent to changing the incentive functionF(Z) =Z,This excitation function is called a linear excitation function, which can output values greater than 1 and less than 0. Then we can use real values as input to train the entire self-encoder, that is, the input range is no longer limited to [0, 1], and can be any real number.

In this linear decoder model :.

As the excitation function changes, the gradient method of the output unit also changes:

 

Because the output layer f (z) = z, f '(Z) = 1, so:

 

When inverse propagation is used to calculate the error, it is still the same as before:

This is because the incentive function of the hidden layer or the sigmoid function has not changed.

The following exercises use a linear encoder to learn the features of color images, dataset features:

After whitening:

Learned features:

It can be seen that, like a gray image, the learned part is still the edge part of some objects.

Experiment code: linear_autoencoder.rar,minfunc.rar, in which minfunc.rar is a tool function package used to optimize automatic coding cost functions.

 

Learning Source: http://deeplearning.stanford.edu/wiki/index.php/Exercise:Learning_color_features_with_Sparse_Autoencoders

 

 

Deep Learning for Color Image Feature Extraction: Linear Decoder