Stanford UFLDL Tutorial Linear decoder _stanford

Linear decoder Contents [hide] 1 sparse self-coding restatement 2 Linear Decoder 3 Chinese-English version of the sparse self-coding restatement

The sparse Self encoder contains 3 layers of neurons, namely the input layer, the hidden layer and the output layer. From the front (neural network) from the description of the encoder, the neurons in the neural network are all using the same excitation function. In the annotation, we modify the definition of the Self encoder to make certain neurons adopt different excitation functions. The resulting model is easier to apply, and the model is more robust to the parameter changes.

Recall that the output layer neuron calculation formula is as follows:

where a (3) is the output. In the Self Encoder, a (3) approximately reconstructs the input x = A (1).

The output range of the S-type excitation function is [0,1], and when F (Z (3)) takes the excitation function, the input is restricted or scaled so that it is in the [0,1] range. Some datasets, such as mnist, can easily scale output to [0,1], but it is difficult to meet the requirements for input values. For example, the input of PCA whitening processing does not meet [0,1] range requirements, and it is not clear whether there is the best way to scale the data to a specific range.

Linear decoder

Setting a (3) = Z (3) can be a simple solution to the above problem. Formally, the output end uses the identity function f (z) = Z as the excitation function, so there is a (3) = f (z (3)) = Z (3). We call this special excitation function a linear excitation function (known as the identical excitation function may be better).

It should be noted that the neurons in the hidden layer of the neural network still use an S-type (or Tanh) excitation function. The excitation formula of the implied element is that the S-type function, x is input, W (1) and B (1) are the weights and deviations of the hidden units respectively. We only use linear excitation functions in the output layer.

An S-type or tanh hidden layer and a linear output layer form a self encoder, which we call a linear decoder.

In this linear decoder model,. Since the output is a linear function of the implicit unit excitation output, the W (2) can be changed so that the output value a (3) is greater than 1 or less than 0. This allows us to train the sparse self encoder with real value input to avoid scaling the sample to a given range.

As the excitation function of the output unit changes, the gradient of the output unit changes accordingly. Before reviewing each output cell error entry is defined as:

where y = x is the desired output, the output from the encoder, is the excitation function. Because the excitation function in the output layer is f (z) = Z, so that F ' (z) = 1, so the above formula can be simplified to

Of course, if you use a reverse propagation algorithm to compute the error term for an implied layer:

Because the hidden layer uses an S-type (or tanh) excitation function f, in the above formula, it is still the derivative of the S-type (or Tanh) function.

Linear decoders sparse self-encoding Sparse autoencoder input layer input layer hidden layer hidden layer output layer layer neuron neuron neural network neu RAL Network self-encoder autoencoder excitation function activation function robust robust s-type excitation function sigmoid activation function tanh excitation functions tanh function Linear excitation function linear activation function identity excitation functions identity activation function hidden element hidden unit weight Weight deviation Item Error Term reverse propagation algorithm back Propagation From:http://ufldl.stanford.edu/wiki/index.php/%e7%ba%bf%e6%80%a7%e8%a7%a3%e7%a0%81%e5%99%a8