Reference papers: 1,convolutional deep Belief NetworksFor Scalable unsupervised learning of hierarchical representations
2.Stacks of convolutional Restricted Boltzmann machinesFor shift-invariant Feature Learning
Pre-Knowledge:http://blog.csdn.net/zouxy09/article/details/9993371
At the beginning of the article, the author presents the problem of the current multilayer generation model (such as DBN): It is difficult to make full-size measurements of high-dimensional images (scaling such models to full-sized). In detail, the traditional multilayer generation model DBN has two challenges: ① image is very high-dimensional, the algorithm should be able to model reasonably, and the calculation is simple; ② objects are often distributed locally in the image, requiring features to represent invariance of local transformations of the input.
Next, the authors recall that CNN's simple calculations and the extraction of local features are impressive. Bingo! CNN and DBN Combine--cdbn! convolutional deep belief networks convolution is convinced of the network, the key part of this method is max-pooling, a probability of dimensionality reduction technology means. The first thing to say about this approach is that the first to second and three layers of the network can learn edge detection, object parts, and objects, such as:
The text begins with the introduction of RBM and DBN in the preamble, and lists the RBM and gaussian-Bernoulli the energy function of the RBM is mainly the summary of the predecessors ' work. Detailed calculations and derivations of RBM or Gaussian-bernoulli RBM can be obtained in the article "Learning multiple Layers of Features from Tiny Images "by Alex Krizhevsky, and" deep learning reading notes of the RBM "self-baidu Google it.
next dry time, carefully introduced CDBN. Starting from the introduction of single-layer CRBM, first release two graphs, Figure 2 is a single filter in the network connection relationship, Figure 3 is the visual layer to the hidden layer of the convolution calculation method:
first, a single-layer CRBM network forward calculation process (positive phase) The input is a 2D image of Nvxnv, like CNN CRBM can set multiple feature filters (also known as convolution cores), assumingThere's a K .the size is NWXNWfeature Filters. Each filter can be interpreted as a channel, and the internal calculation of a channel is independent of the rest of the channel. The calculation of each filter is divided into convolution and pooling two parts: first, from the visual layer to the hidden layer of the calculation (convolution), figure 3 Image represents the calculation process conv2 (v,w, ' valid ') =H1, The sigmoid function is activated to get the value of the filter 1 in the hidden layer (H1 is called a group), and the second is the calculation of the pooling layer of the hidden layer to the lower sampling layer, where the Max-pooling method is chosen, that is, by the size of the pool (such as 2x2) each region selects the maximum value (p in Figure 2), Regions are zoned non-overlapping. The convolution and down-sampling process here is similar to CNN, as detailed in LeCun's CNN paper. The following two formulas correspond to the calculations of convolution and pooling respectively.
where I is, BA represents the pooling region
The second pooling formula is calculated by the probability of the max-pooling mechanism and P (h=1|v), P (p=0|v) =1-p (h=1|v), where P (H=1|v) represents a 1 probability of the H element appearing in the pooling region, Max-pooling P-Cell is 1 probability p (p=1|v) =p (h=1|v) →p (p=0|v) =1-p (p=1|v) =1-p (h=1|v).
Calculate all filter channels and get the group of K size NPXNP (where np=nv-nw+1). Finally, we need to do Gibbs sampling, so as to complete the CRBM forward propagation process positive phase. In fact, since the pooling layer has no parameters to train, pooling only acts as a descending and regularization operation.
second, the reverse calculation process of CRBM negative phase
such as the formula, where W with * indicates the transpose of the filter W. In the description of the forward propagation, it is mentioned that the pooling layer has no free parameters, so it can be transmitted back to the visible layer from the hidden layer when the CRBM is trained. each visual Layer Unit V is connected to the K filter, so the reconstruction of negative phase in the inverse process must be realized by the joint action of all filters, and the formula is expressed as the superposition σ of K filter action. The full conv2 function can be implemented in the calculation process, such as σ (Conv2 (h,w ', ' full ')). In the paperStacks of convolutional Restricted Boltzmann machinesFor shift-invariant Feature Learning also divides the visual layer into the edges and the center two parts to do the calculation (because the edge portion of the forward convolution calculation process weight is small), generally do not divide the matter, the goal of the task attention is often distributed in the middle of the image.
third, sparse regularizationbecause the CRBM hidden layer unit is much larger than the input visual layer, the model is super complete. Ultra-complete is easy to cause the filter only represents a single pixel rather than a local feature, a common solution is to add sparse constraints, forcing most of the hidden layer of the element is zero, set the entire hidden layer in a low activation value. Lee also stressed that: sparsity regularization during trianing was necessary for learning the oriented edge Filters;when this term is re Moved the algorithm failed to learn oriented edges. No sparse constraint algorithm can learn to have a directional edge line.
Iv. Energy Functionsafter the connection and computation of the network are finished, the energy function of the model is given (the individual thinks that the energy function is always put in front for the sake of convenience, and the actual design model is often designed to visualize the hidden layer and pool layer and then deduce the energy function on this basis).
Five, parameter calculationusing the contrastive divergence-like RBM algorithm, the algorithm is an approximate fast solution to the maximum likelihood function, and the specific content of the CD algorithm for RBM is Hinton the article of the Great God. The following is a CRBM parameter solver:
From for notes (Wiz)
Convolutional deep belief Networks convolution conviction Network paper notes
