Autoencoders and sparsity (2)

In this problem set, you will implement the sparse autoencoder algorithm, and show how it
Discovers that edges are a good representation for natural images.

Step 1: generate training set

Step 2: sparse autoencoder objective

Step 3: gradient checking

Step 4: Train the sparse autoencoder

Step 5: Visualization

Process

1. Calculate the input value (the Z value in the Program) and output value (that is, the value in the program, and a is the sigmoid function value of Z) of each node in the network ).

2. Use the Z value and a value to calculate the error value of each node in the network (that is, the Delta value in the program ).

3. In this way, we can use the, Z, and delta of each node calculated above to express the loss function of the system and the partial derivative of the loss function.

In fact, step 1 is performed in the forward direction, that is, the calculation is performed in the direction of the input layer -- "hidden layer --" output layer. Step 2 is performed in the inverse direction (this is also the source of the BP algorithm), that is, the error value of each node is performed in the direction of the output layer-"hidden layer-" input layer.

MATLAB

Bsxfun-- C = bsxfun (fun, a, B) Expresses binary operations on elements between arrays A and B. Fun is a function handle, M file, or embedded function. In actual use, there are many options for fun, such as addition and subtraction. the symbol '@' must be used before '@'. generally, a and B must have the same size. If they are different, only one dimension is different. At the same time, a and B must have one dimension equal to 1. For example, bsxfun (@ minus, A, mean (A), where the sizes of A and mean (a) are different) expand to the same size as a, and then subtract the value of the expanded mean (a) element from each element of.

Rand-- Generate a pseudo-random number with a uniform distribution. Distributed in (0 ~ 1)
Syntax: rand (m, n) generates a pseudo-random number with a uniform distribution of n columns in m rows.
Rand (m, n, 'double') generates a pseudo-random number with a uniform distribution of the specified precision. The parameter can also be 'single'
Rand (randstream, m, n) uses the specified randstream (I think it is a random seed) to generate pseudo-random numbers.

Randn-- Generate a pseudo-random number with a standard normal distribution (Mean Value: 0, variance: 1)

Randi-- Generate a pseudo-random integer with a uniform distribution
Primary Syntax: Randi (IMAX) generates a pseudo-random integer with a uniform distribution in a closed interval (0, IMAX ).
Randi (IMAX, m, n) generates an mxn random matrix in the closed range (0, IMAX)
R = Randi ([Imin, IMAX], M, N) generate an mxn random matrix in the closed interval (Imin, IMAX)

Exist-- Test whether the parameter exists. For example, exist ('opt _ normalize ', 'var') indicates whether the variable opt_normalize exists, and 'var' indicates the meaning of the variable.

Colormap-- Set the current common color value table.

Floor-- Floor (a): returns the maximum integer not greater than.

Ceil-- Ceil (a): returns the smallest integer not less than.

Repmat-- This function extends a matrix and copies the data in the original matrix. For example, B = repmat (a, m, n) creates a matrix B, and B Copies a total of M * n a matrices, therefore, the size of the B matrix is [size (A, 1) * m size (A, 2) * m].

Technorati label: Machine Learning

Autoencoders and sparsity (2)