Depth learning each layer structure of neural network in Caffe __caffe

Preface

Layer structure is the most basic unit of neural Network (neural Networks) modeling and computation. Because the neural network has different layer structure, different types of layers have different parameters. Therefore, each layer of caffe configuration is different, and the layer structure and parameters are predefined in the Prototxt file, here, we have the latest version of the Caffe model of the layer structure to make a brief summary.
If you need to reprint, please specify the source: http://blog.csdn.net/ws_20100
As the level is limited, if there are errors, please correct me. 1. Visual Layer (Vision Layers) header file:./include/caffe/vision_layers.hpp

The visual layer usually takes the image as input and takes the processed image as output. A typical image, typically having a channel (c=1 c=1, grayscale image) or three channels (c=3 C=3,rgb images). In this paper, the distinguishing feature of an image is its spatial structure: usually an image has a larger height h>1 h>1 and width w>1 w>1. This two-dimensional geometrical feature naturally leads us to properly handle image input. Usually, the visual layer operates on the local area of the input image and produces the corresponding local output. In contrast, other layers (with very few exceptions) ignore the spatial structure of the input image and treat it as a dimension for CHW CHW "large vectors."1. Convolution layer (convolution)Type: Convolution CPU Implementation code:./src/caffe/layers/convolution_layer.cpp CUDA GPU Implementation code:./src/caffe/layers/convolution_ Layer.cu parameters (Convolutionparameter convolution_param)
Necessary parameters:
Num_output (c_o):The number of filters.
kernel_size (or Kernel_h and Kernel_w):Specifies the height and width of each filter. Strongly recommended parameters:
Weight_filler:The initial distribution and distribution parameters of the filter.
Bias_filler:[Default: Type: ' constant ' value:0] Optional parameter:
Bias_term:[Default: True] Specifies whether additional offsets are learned and applied after the filter is exported.
pad (or Pad_h and Pad_w):[Default: 0] Specifies the number of pixels to add to each side of the input (stealth).
Stride (or Stride_h and stride_w):[Default: 1] Specifies the interval of input filters.
Group (g):[Default: 1] If g>1, we limit the connection of each filter to a subgroup of the input image, in particular, the input and output channels are divided into G groups, and the output channel of the I group I is only connected with the input channel of group I I. Input:
n * c_i * h_i * w_i output
n * c_o * h_o * w_o, where h_o = (h_i + 2 * pad_h-kernel_h)/stride_h + 1,w_o are computed by the same method. Examples (in./examples/imagenet/imagenet_train_val.prototxt can be seen):

Layer {
  name: "Conv1"
  type: "Convolution"
  bottom: "Data" top
  : "Conv1"
  # Learning rate and decay Multipliers for the filters
  param {lr_mult:1 decay_mult:1} # Learning rate and decay for the multipliers
  s
  param {lr_mult:2 decay_mult:0}
  convolution_param {
    num_output:96     # Learn
    filters  Ize:11    # Each filter was 11x11
    stride:4          # Step 4 pixels between each filter application
    Weight_filler {
      type: ' Gaussian ' # Initialize the filters from a Gaussian
      std:0.01        # Distribution with Stdev 0.01 (DEFA Ult mean:0)
    }
    Bias_filler {
      type: "Constant" # Initialize the biases to zero (0)
      value:0
    }
  
   }
}
  

2. Pool Layer (pooling)Type: Pooling CPU Implementation code:./src/caffe/layers/pooling_layer.cpp CUDA GPU Implementation code:./SRC/CAFFE/LAYERS/POOLING_LAYER.CU parameters ( Poolingparameter Pooling_param)
Necessary parameters:
kernel_size (or Kernel_h and Kernel_w):Specifies the height and width of each filter. Optional Parameters:
Pool:[Default: MAX] Pooling method, currently has max,ave,stochastic.
pad (or Pad_h and Pad_w):[Default: 0] Specifies the number of pixels to add to each side of the input (stealth).
Stride (or Stride_h and stride_w):[Default: 1] Specifies the interval of input filters. Input:
n * C * h_i * w_i output
n * C * h_o * w_o, wherein h_o and w_o are computed by the same method as the convolution layer. Examples (in./examples/imagenet/imagenet_train_val.prototxt can be seen):

Layer {
  name: "Pool1"
  Type: "Pooling"
  bottom: "conv1" top
  : "Pool1"
  pooling_param {
    Pool:max
    Kernel_size:3 # Pool over a 3x3 region
    stride:2      # Step two pixels (in the bottom blob) between pooling ONS
  }
}

3.LRN Layer (local Response normalization)Type: LRN CPU Implementation code:./src/caffe/layers/lrn_layer.cpp CUDA GPU Implementation code:./SRC/CAFFE/LAYERS/LRN_LAYER.CU parameters (Lrnparameter lrn_ Param
Optional Parameters:
local_size:[Default: 5] The number of channels to sum (cross-channel LRN), or the edges of the square area used to sum (len in the channel).
Alpha:[Default: 1] Scaling parameters (see below).
Beta:[Default: 5] Index (see below).
norm_region:[Default: Across_channels] Specifies whether to sum (across_channels) adjacent channels or to sum adjacent space positions (within_channels).

The local response regularization layer is used to localize the local input region to achieve a "side inhibition" effect. In across_channels mode, local areas extend to adjacent channels, but there is no space extension (for example, their size is local_size x 1 x 1). In Within_channel mode, local areas have spatial extensions, but they are in separate channels (for example, their size is 1 x local_size x local_size). Each input value is divided by (1+ (α/n) ∑ix2i) Beta (1+ (\alpha/n) \sum_{i}x_{i}^{2}) ^{\beta}, where n n is the size of the local area, and the area centered on the value is summed (0 padding is added if required). 4.im2col

Im2col is a useful tool for transforming an image into a column vector, and you don't need to know it in most cases. Im2col the entire block layout to a matrix and converts the Caffe original convolution operation into matrix multiplication. 2. Loss layer (Loss Layers)

The

lossy layer drives learning by comparing the difference between the output and the target and minimizing the cost. The loss itself is calculated by forward propagation, and the gradient of loss is w.r.t by reverse propagation. 1.Softmax type: Softmaxwithloss
Softmax loss layer calculates the polynomial logic loss of the input data Softmax. is conceptually equivalent to the cascade of a softmax layer after the polynomial logic loss layer. But it can provide a more stable gradient in numerical terms. 2 Sum-of-squares/euclidean type: Euclideanloss
Euclidean loss layer calculates the square sum of two input difference values. The formula is as follows:
12n∑ni=1| | x1i−x2i| | \frac{1}{2n}\sum_{i=1}^{n} | | x_i^1-x_i^2| | _2^2 3 hinge/margin type: Hingeloss CPU Implementation code:./src/caffe/layers/hinge_loss_layer.cpp CUDA GPU Implementation code: There are currently no parameters (Hingelossparameter hinge_loss_param)
Optional parameters:
norm: [Default: L1] used by the norm, There are L1 and L2 norms at present. Input:
N * C * H * w predictions n * 1 * 1 * 1 Labels output
1 * 1 * 1 * 1 computed Loss example:

# L1 Norm
Layer {
  name: ' Loss '
  type: ' Hingeloss ' bottom
  : ' pred ' bottom
  : ' label '
}

# L2 Norm
Layer {
  name: "Loss"
  type: "Hingeloss"
  Bottom: "pred"
  Bottom: "Label" Top
  : "Loss"
  Hinge_loss_param {
    norm:l2
  }
}

The hinge loss layer calculates the square of a pair of many hinge losses or hinge losses. 4.Sigmoid cross-entropy type: Sigmoidcrossentropyloss 5.Infogain type: Infogainloss 6.Accuracy and Top-k

The accuracy layer is used to calculate the correct rate of output and target output, in fact it is not a loss layer, and there is no post propagation this step. 3. Excitation layer (Activation/neuron Layers)

Typically, the excitation layer, or the neuron layer, is an element-level operation that inputs the underlying BLOB and produces a blob of the same size at the top level. In the excitation layer described below, we ignore the input and output size, uniformly defined as: input
n * C * H * w output
n * C * H * w 1.relu/rectified-linear and Leaky-relu type: Relu CPU Implementation code:./src/caffe/layers/relu_layer.cpp CUDA GPU Implementation code:./SRC/CAFFE/LAYERS/RELU_LAYER.CU parameters (Reluparameter relu_param)
Optional Parameters:
Negative_slope: [Default: 0] Specifies how negative values are removed. By multiplying by a slope value (1) or by setting a negative value of 0 (0). Example:

Layer {
  name: ' RELU1 '
  type: ' Relu '
  bottom: ' conv1 ' top
  : ' Conv1 '
}

For a given input value of x, if x > 0,relu layer output is x, if x < 0,relu layer output is negative_slope * x. If the Negative_slope parameter is not set, it is equivalent to the standard Relu function: Max (x,0) max (x, 0). It also supports in-situ computations, which means that the underlying BLOB and the top blob can be the same to reduce resource consumption. 2.Sigmoid <