[Deep Learning] Wake-sleep Algorithm

This article is translated from 2007-to recognize shapes, first learn to generate images, Geoffrey Hinton.

The fifth strategy is designed to enable the high-level Feature Extraction Tool to communicate with the underlying layer, and at the same time, it is easy to use the layered network of random binary neurons.

The activation probability of these neurons is a smoothing nonlinear equation about the total input:

 

Si and SJ are the activity of neuron I and j, wij is the weight of I and J, and BJ is the offset of J.

Figure 1

 

If the training data is generated from top to bottom using a multi-layer image model of the type in Figure 1, it is usedTop-down)The binary state of the hidden layer neuron that generates the image can be used for training.Bottom-up (reco-weights)Expected output.

At first glance, this kind of use is from top to bottom.Generative connections)It is meaningless to provide the desired state to neurons in the hidden layer, because what we need to learn now is the ability to generateTraining data)OfGraphics Model).

However, if we already have some goodReco-connections), We can useFrom bottom to topOfPass)-- Use real data to activate neurons at each layer, so that we can rebuild the activity at each layer by trying the activity information at the previous layer, so as to learn the generated weight.

So this becomes a problem of chicken and eggs: GivenGenerative weights, gene-weights for short), We can learnRecognition weights, reco-weights)Given cognitive weights, we can learn to generate weights.

What is the result? Based on a small number of random values andPhases of learning)Switch, we can learn the above two weights at the same time!

In the wake phase ("Wake" phase), cognitive weights are used to drive neurons from bottom to top. The binary states of neurons in the adjacent layers can be used to train and generate weights;

In the sleep phase ("Sleep" pahse), a connection is generated from top to bottom to drive the network.Generative model)GenerateImage (fantasies)The state of neurons in the adjacent layer (0/1) can be used to learn cognitive connections from bottom to top (hinto et al .., 1995 ).

The learning rules are very simple. In the waking stage, the weight gkj is generated and updated according to the following formula:

Where the neuron K is on the upper layer of neuron J, e is the learning rate, and PJ is the activation probability when the neuron J is driven by the current state of the first neuron that uses the current generated weight.

During sleep, the cognitive weight wij is updated according to the following formula:

Qj is the activation probability when neuron J is driven by the current state of the first neuron using the current cognitive weight.