C ++ convolutional neural network example: tiny_cnn code explanation (7) -- fully_connected_layer Structure Analysis

C ++ convolutional neural network example: tiny_cnn code explanation (7) -- fully_connected_layer Structure Analysis

In the previous blog, we have analyzed the convolution and downsampling layers. In this blog, we will analyze the fully_connected_layer class (full connection layer) of the last top layer structure:

I. Full connection layer in convolutional Neural Networks

In convolutional neural networks, the full connection layer is located at the final part of the network model. It is responsible for predicting the final output features of the network and obtaining the classification result:

The full connection layer in the LeNet-5 model is divided into full connection and Gaussian connection, the final output result of this layer is the prediction label, for example, here we need to classify and forecast the data in the MNIST database, there are 10 types of data (numbers 0 ~ 9). Therefore, the final output of the fully connected layer is a 10-dimensional prediction result vector. If the value of one dimension is non-zero, the prediction result corresponds to a number.

Ii. fully_connected_layer Class Structure

Different from the convolution layer and the lower sampling layer, the full connection layer fully_connected_layer class inherits from the base class layer. The class members can be divided into four parts: member variables, constructors, Forward Propagation functions, and reverse propagation functions.

2.1 member variables

Fully_connected_layer class has only one member variable, which is a Filter type variable:

Here, the Filter is a default filter_none type passed in through the class template parameter, as follows:

As for the filter_none type, it is determined from the name that it should be a class encapsulation related to the filter core. The specific definition is in the dropout. h file:

About dropout. I will give a detailed introduction to the relevant class information encapsulated in the H file in the Post-blog post. Here, I will leave a pitfall, but I will reveal it in advance, dropout in convolutional neural networks is designed to improve the network overfitting performance.

2.2 Constructor

The constructor is extremely simple. It simply calls the constructor of the base layer:

As for the base class layer, it encapsulates a large number of virtual functions and pure virtual functions, provides basic framework settings for the network layer, and instantiates the layer_base class in part, these points will be detailed later.

2.3 Forward Propagation Functions

As we all know, convolutional neural networks are very similar to BP neural networks in training, including a Forward Propagation Process of Sample Prediction and a back propagation process of error. First, the code of the Forward propagation function is as follows:

        const vec_t& forward_propagation(const vec_t& in, size_t index) {            vec_t &a = a_[index];            vec_t &out = output_[index];            for_i(parallelize_, out_size_, [&](int i) {                a[i] = 0.0;                for (int c = 0; c < in_size_; c++)                    a[i] += W_[c*out_size_ + i] * in[c];                a[i] += b_[i];            });            for_i(parallelize_, out_size_, [&](int i) {                out[i] = h_.f(a, i);            });            auto& this_out = filter_.filter_fprop(out, index);            return next_ ? next_->forward_propagation(this_out, index) : this_out;        }

From the code, we can see that this Forward propagation function is essentially a recursive function, which uses Recursion to implement layer-by-layer propagation:

In the process of Forward propagation, there are two main stages. One is to map the input data through the convolution kernel and offset of the current layer:

As can be seen from the code, the ing process of the convolution layer is essentially a convolution operation, and then accumulates the offset on the convolution result. The second stage is to send the ing result of the convolution layer to the activation function for processing:

The main function of activating a function is to standardize the ing output of the convolution layer and adjust the data distribution during the period. The typical activation function is the Sigmoid function, which smooths output features. Later, scholars proposed the Relu-type activation function, which is mainly to normalize the output features in a sparse manner, so that it is closer to the human brain's visual ing mechanism. In tiny_cnn, the author encapsulates activation functions such as sigmoid, relu, leaky_relu, softmax, tan_h, and tan_hp 1m2. These classes are defined in the activation namespace. Specifically, in the activation_function.h file, in the subsequent sections, I will write a blog article to analyze the activation functions of tiny_cnn in a centralized manner.

2.4 reverse propagation function

The back-propagation algorithm is a classic feature of the back-propagation neural network. Most of them use the random gradient descent method to evaluate and spread the error. The inverse propagation algorithm involves the concepts of deviation and sensitivity transfer, which makes it more complicated in principle than the Forward propagation process and complicated in code implementation, here we only give the reverse propagation code first. In the subsequent blog posts, we will analyze the propagation process in more detail. OK is a pitfall:

    const vec_t& back_propagation(const vec_t& current_delta, size_t index) {            const vec_t& curr_delta = filter_.filter_bprop(current_delta, index);            const vec_t& prev_out = prev_->output(index);            const activation::function& prev_h = prev_->activation_function();            vec_t& prev_delta = prev_delta_[index];            vec_t& dW = dW_[index];            vec_t& db = db_[index];            for (int c = 0; c < this->in_size_; c++) {                // propagate delta to previous layer                // prev_delta[c] += current_delta[r] * W_[c * out_size_ + r]                prev_delta[c] = vectorize::dot(&curr_delta[0], &W_[c*out_size_], out_size_);                prev_delta[c] *= prev_h.df(prev_out[c]);            }            for_(parallelize_, 0, out_size_, [&](const blocked_range& r) {                // accumulate weight-step using delta                // dW[c * out_size + i] += current_delta[i] * prev_out[c]                for (int c = 0; c < in_size_; c++)                    vectorize::muladd(&curr_delta[0], prev_out[c], r.end() - r.begin(), &dW[c*out_size_ + r.begin()]);                for (int i = r.begin(); i < r.end(); i++)                     db[i] += curr_delta[i];            });            return prev_->back_propagation(prev_delta_[index], index);        }

Iv. Notes

1. The forward/reverse propagation function of the convolution layer and the lower sampling Layer

In the fully_connected_layer class, we found that it contains the forward/reverse propagation function, however, we did not find the shadow of the forward/reverse propagation function in the convolution layer and the average sampling layer described earlier. However, the forward/backward propagation function is indeed a global process, it is impossible to have a fault. Therefore, we will find that the original author encapsulates the forward/reverse propagation functions corresponding to the convolutional_layer class and average_pooling_layer in their common base class: partial_connected_layer.

2. forward and reverse propagation Functions

In this blog post, I have dug a lot of pitfalls for subsequent blog posts, especially the essence of training for Convolutional neural networks such as forward/reverse propagation functions, this is the best part of the author's programming and framework design skills. One or two blog posts may not be clear, so please do not worry. I will understand the mystery as soon as possible, then explain it in plain language. Therefore, the pitfalls will be filled in one by one.