"Reprint" Deep belief Network

Deep belief Network

In order to better in the next discussion class to tell DBN, special open this post. is mainly
Introduce DBN knowledge and make a logically complete thing. Reference
Hinton of Things:

Reading list
RBM related
[1] about the Scholarwiki of Boltzmann machine
[2] The 11th chapter of Haykin Book
[3] The 7th chapter of Duda Book
[4] Exponential family expansion of RBM
[5] RBM modeling capabilities: as a universal approximator

Contrastive divergence related
[6] Proposed PoE article, also CD-presented article
[7] about the effect of CD in practice
[8] Analysis of convergence

Helmholtz Machine Related
A list of others
[9] The proposed concept
[10] Training algorithm Wake-sleep

PoE related
[One] Scholarwiki on Max Welling's article
[T-distr] PoE

DBN related
[NIPS2007] Tutorial on DBN
[Hinton] Classroom slides
[15] Philosophical considerations: depth is to improve coding efficiency
[16] Fast Learning algorithm: DBN + Softmax
[17] Reduction of dimensionality
[18] using DBN + GP classification

Other related articles
[Sigmoid] Belief network
[FoE]
[FoE] Original article

Brief introduction
Understand DBN in several steps, how to train an RBM for each layer of DBN,
How to glue these RBM together and finally how to use these as a
Belief network solves practical problems.

An RBM is a PoE [6], and therefore also a special MRF, which learns from
In that normalization factor caused difficulties, although can be used MCMC
To solve, but not good (slow, Fangcha). In about 2002 years
Hinton proposed a CD-based training algorithm [6] to replace the original MLE obtained
Good effect, why do the RBM can be see about RBM this
Universal Approximator's article [5].

The main reason for using deep architecture is encoding efficiency [15],
The use of multilayer requires a greedy algorithm pretraining [16], the algorithm
Ensure the distribution of the hidden layer trained on the first layer as the second layer of the explicit layer input
May increase the marginal likelihood [16]. Then, depending on the type of application, the model
Further fine tune.

such as classification problems, can introduce a layer of softmax neuron [16], with its
The loss on the callout data uses BP for fine tune (which may require
Helmholtz machine-related methods [10]); like Dimensionaly.
Reduction [17], in fact, the original network in turn can be reconstruction,
Then BP is used according to reconstruction error, and for example

Related connections
ML project of Super Big Cow soup
Hinton ' s page for the DBN project
Hinton ' s page for the Science paper
Hinton ' s reading list for DBN

Research questions
1. Can I use Nonparametric Bayesian method to change the Boltzmann
Machine

"Reprint" Deep belief Network