TINY-CNN Execution Process Analysis (MNIST)

Using Mnist as an example in http://blog.csdn.net/fengbingchun/article/details/50573841, the use of TINY-CNN is introduced, and the following analysis is carried out on its execution process:

Two kinds of loss functions are supported: (1), mean squared error (mean variance), (2), Cross entropy (crossover entropy). Using mean squared error in mnist, the code snippet:

Mean-squared-error loss function for Regressionclass MSE {public:    static float_t F (float_t y, float_t t) {        Retu RN (Y-T) * (Y-T)/2;    }    Static float_t df (float_t y, float_t t) {        return y-t;    }};

Support six kinds of activation functions: (1), Tanh; (2), sigmoid, (3), Softmax, (4), Rectifiedlinear (Relu), (5), Leaky Relu, (6), identity. Mnist is used in the Tanh, code snippet:

Class Tan_h:public function {public:    float_t F (const vec_t& V, size_t i) const override {        Const float_t EP = Std::exp (V[i]);        Const float_t em = Std::exp (-v[i]);         Return (EP-EM)/(ep + EM);    }    Fast approximation of tanh (improve 2-3% speed in LeNet-5)    /*float_t F (float_t x) const {        const float_t x2 = X * x;        X *= 1.0 + x2 * (0.1653 + x2 * 0.0097);        Return X/std::sqrt (1.0 + x * x);//Invsqrt (static_cast<float> (1.0 + x * x));    } *    /float_t df (float_t y) const override {return 1.0-sqr (y);}    std::p air<float_t, float_t> scale () const override {return Std::make_pair (-0.8, 0.8);}

Design the CNN structure for Mnist, similar to the LENET-5 structure, removing the F6 layer:

Input layer Inputs: image size 32*32, number of neurons 32*32=1024, code snippet:

    const int width = header.num_cols + 2 * x_padding;    const int height = header.num_rows + 2 * y_padding;    Std::vector<uint8_t> Image_vec (header.num_rows * header.num_cols);    Ifs.read ((char*) &image_vec[0], header.num_rows * header.num_cols);    Dst.resize (Width * height, scale_min);    for (size_t y = 0, y < header.num_rows; y++) for      (size_t x = 0; x < Header.num_cols, x + +)        Dst[width * (y + y _padding) + x + x_padding]        = (Image_vec[y * header.num_cols + x]/255.0) * (scale_max-scale_min) + scale_min;

C1 layer: Convolution window size 5*5, the number of output features 6, convolution window type 6, output feature map size 28*28, training parameters 5*5*6+6=156, the number of neurons 28*28*6=4704;

S2 layer: Convolution window size 2*2, the output of the sample number of 6, the volume of the window 6, the output of the sample size 14*14, training parameters 1*6+6=12, the number of neurons 14*14*6=1176;

C3 layer: Convolution window size 5*5, the number of output features 16, convolution window type 16, output feature map size 10*10, training parameters 6*16*5*5+16=2416, the number of neurons 10*10*16=1600;

S4 layer: Convolution window size 2*2, the output of the sample number of 16, the volume of the window 16, the output of the sample size 5*5, training parameters 1*16+16=32, the number of neurons 5*5*16=400;

C5 layer: Convolution window size 5*5, the number of output features 120, convolution window type 120, output feature map size 1*1, training parameters 5*5*16*120+120=48120, the number of neurons 1*120=120;

Output layer outputs: Output feature graph Number 10, convolution window type 10, output feature graph size 1*1, can train parameter 120*10+10=1210, neuron number 1*10=10.

The original mnist image size is 28*28, here is 32*32, up and down each fill 2 pixels, fill the pixel value is 1, the other pixel value range is [ -1,1].

Weights and thresholds (bias) initialization: Weights are generated using uniform random numbers, and the thresholds are assigned 0.

C1 layer weights, initialization range [sqrt (6.0/(25+150)), sqrt (6.0/(25+150))];

S2 layer weights, initialization range [sqrt (6.0/(4+1)),-sqrt (6.0/(4+1))];

C3 layer weights, initialization range [sqrt (6.0/(150+400)),-sqrt (6.0/(150+400))];

S4 layer weights, initialization range [sqrt (6.0/(4+1)),-sqrt (6.0/(4+1))];

C5 layer weights, initialization range [sqrt (6.0/(400+3000)),-sqrt (6.0/(400+3000))];

Output layer weights, initialization range [sqrt (6.0/(120+10)),-SQRT (6.0/(120+10))].

Forward Propagation:

C1 Layer Code Snippet:

        vec_t &a = A_[worker_index]; W*x vec_t &out = Output_[worker_index]; Output const vec_t &in = * (Prev_out_padded_[worker_index]);        Input Std::fill (A.begin (), A.end (), (float_t) 0.0);                For_i (Parallelize_, Out_.depth_, [&] (int o) {for (layer_size_t inc = 0; Inc < In_.depth_; inc++) {                if (!tbl_.is_connected (O, Inc)) continue;                Const float_t *PW = &this->w_[weight_.get_index (0, 0, In_.depth_ * o + Inc)];                Const float_t *PI = &in[in_padded_.get_index (0, 0, Inc)];                float_t *pa = &a[out_.get_index (0, 0, O)]; for (layer_size_t y = 0, y < out_.height_; y++) {for (layer_size_t x = 0; x < out_.width_; × x + +)                        {Const FLOAT_T * PPW = PW;                        Const float_t * PPI = pi + (y * h_stride_) * in_padded_.width_ + x * W_STRIDE_; float_t sum = (float_t) 0.0;                        Should is optimized for small kernel (3x3,5x5) for (layer_size_t WY = 0; WY < Weight_.height_;                                wy++) {for (layer_size_t wx = 0; wx < weight_.width_; wx++) {                            Sum + = *ppw++ * Ppi[wy * in_padded_.width_ + WX];                    }} pa[y * out_.width_ + x] + = sum; }}} if (!this->b_.empty ()) {float_t *pa = &a[out_.get_index (                0, 0, O)];                float_t B = this->b_[o];            Std::for_each (PA, PA + out_.width_ * out_.height_, [&] (float_t& f) {f + = B;});        }        });        For_i (Parallelize_, Out_size_, [&] (int i) {Out[i] = H_.F (A, I); });

S2 Layer Code Snippet:

        vec_t& a = A_[index];             For_i (Parallelize_, Out_size_, [&] (int i) {            const wi_connections& connections = Out2wi_[i];            A[i] = 0.0;            for (auto connection:connections)//13.1%                a[i] + = W_[connection.first] * In[connection.second];//3.2%            A[i] *= Scale_factor_;            A[i] + = B_[out2bias_[i]];        });        For_i (Parallelize_, Out_size_, [&] (int i) {            output_[index][i] = H_.F (A, I);        });

The C3 layer, C5 layer code snippet is the same as the C1 layer.

The S4 layer code snippet is the same as the S2 layer.

Output Layer Code snippet:

        vec_t &a = A_[index];        vec_t &out = Output_[index];        For_i (Parallelize_, Out_size_, [&] (int i) {            a[i] = 0.0;            for (layer_size_t c = 0; c < in_size_; C + +) {                A[i] + = W_[c*out_size_ + i] * in[c];            }            if (Has_bias_)                a[i] + = B_[i];        });        For_i (Parallelize_, Out_size_, [&] (int i) {            out[i] = H_.F (A, I);        });

Reverse propagation:

Output Layer Code snippet:

        vec_t Delta (Out_dim ());        Const activation::function& h = layers_.tail ()->activation_function ();        if (Is_canonical_link (h)) {            For_i (Out_dim (), [&] (int i) {delta[i] = Out[i]-t[i];});        } else {            vec_t de_d y = gradient<e> (out, t);            Delta = De/da = (de/dy) * (Dy/da) for            (size_t i = 0; i < Out_dim (); i++) {                vec_t Dy_da = h.df (out, i);                Delta[i] = vectorize::d ot (&de_dy[0], &dy_da[0], Out_dim ());            }        }

C5 Layer Code Snippet:

        Const vec_t& prev_out = prev_->output (index);        Const activation::function& Prev_h = Prev_->activation_function ();        vec_t& Prev_delta = Prev_delta_[index];        vec_t& DW = Dw_[index];        vec_t& db = Db_[index]; for (layer_size_t c = 0; c < this->in_size_; C + +) {//Propagate delta to previous layer//PR EV_DELTA[C] + = current_delta[r] * w_[c * out_size_ + r] prev_delta[c] = vectorize::d ot (&curr_delta[0], &amp ;            W_[c*out_size_], out_size_);        PREV_DELTA[C] *= prev_h.df (Prev_out[c]); } for_ (Parallelize_, 0, (size_t) out_size_, [&] (const blocked_range& R) {//Accumulate weight-st EP using Delta//dw[c * out_size + i] + = current_delta[i] * Prev_out[c] for (layer_size_t c = 0; c < In_size_; C + +) Vectorize::muladd (&curr_delta[r.begin ()], prev_out[c], r.end ()-R.begin (), &dw[c*out_size_ +    R.begin ()]);        if (Has_bias_) {for (int i = R.begin (); I < R.end (); i++) Db[i] + = Curr_delt            A[i]; }        });

S4 Layer Code Snippet:

        Const vec_t& prev_out = * (Prev_out_padded_[index]);        Const activation::function& Prev_h = Prev_->activation_function (); vec_t* Prev_delta = (Pad_type_ = = padding::same)?        &prev_delta_padded_[index]: &prev_delta_[index];        vec_t& DW = Dw_[index];        vec_t& db = Db_[index];        Std::fill (Prev_delta->begin (), Prev_delta->end (), (float_t) 0.0); Propagate Delta to previous layer For_i (in_.depth_, [&] (Int Inc.) {for (layer_size_t OUTC = 0; ou TC < Out_.depth_;                outc++) {if (!tbl_.is_connected (OUTC, Inc) continue;                Const float_t *PW = &this->w_[weight_.get_index (0, 0, In_.depth_ * OUTC + Inc)];                Const float_t *PDELTA_SRC = &curr_delta[out_.get_index (0, 0, OUTC)];                float_t *PDELTA_DST = & (*prev_delta) [In_padded_.get_index (0, 0, Inc)]; for (layer_size_t y = 0, y < out_.height_; y++) {for (layer_size_t x = 0 × < Out_.width_; x + +)                        {Const FLOAT_T * PPW = PW;                        Const FLOAT_T PPDELTA_SRC = pdelta_src[y * out_.width_ + x];                        float_t * Ppdelta_dst = pdelta_dst + y * h_stride_ * in_padded_.width_ + x * W_STRIDE_; for (layer_size_t WY = 0, WY < Weight_.height_; wy++) {for (layer_size_t wx = 0; WX < we Ight_.width_;                            wx++) {Ppdelta_dst[wy * in_padded_.width_ + WX] + = *ppw++ * PPDELTA_SRC;        }                        }                    }                }            }        });        For_i (Parallelize_, In_padded_.size (), [&] (int i) {(*prev_delta) [i] *= prev_h.df (Prev_out[i]);        });  Accumulate DW For_i (in_.depth_, [&] (int Inc) {for (layer_size_t OUTC = 0; OUTC < out_.depth_; outc++) {if (!tbl_.is_connected (OUTC, Inc) continue; for (layer_size_t WY = 0, WY < Weight_.height_; wy++) {for (layer_size_t wx = 0; WX < weight_.wi Dth_;                        wx++) {float_t dst = 0.0;                        Const FLOAT_T * Prevo = &prev_out[in_padded_.get_index (WX, WY, Inc);                        Const float_t * Delta = &curr_delta[out_.get_index (0, 0, OUTC)]; for (layer_size_t y = 0; y < out_.height_; y++) {DST + vectorize::d ot (Prevo + y * in_padde                        D_.width_, Delta + y * out_.width_, out_.width_);                    } dw[weight_.get_index (WX, WY, In_.depth_ * OUTC + inc)] + = DST;        }                }            }        });                Accumulate DB if (!db.empty ()) {for (layer_size_t OUTC = 0; OUTC < out_.depth_; outc++) {                Const float_t *delta = &curr_delta[out_.get_index (0, 0, OUTC)]; DB[OUTC] + = Std::accumulate (Delta, Delta + out_.width_ * out_.height_, (float_t) 0.0); }        }

C3 Layer Code Snippet:

        Const vec_t& prev_out = prev_->output (index);        Const activation::function& Prev_h = Prev_->activation_function ();        vec_t& Prev_delta = Prev_delta_[index]; For_ (parallelize_, 0, (size_t) in_size_, [&] (const blocked_range& R) {for (int i = R.begin (); I! = R.en D ();                i++) {Const wo_connections& connections = In2wo_[i];                float_t delta = 0.0; for (auto connection:connections) Delta + = W_[connection.first] * Current_delta[connection.second]; 40.6% Prev_delta[i] = Delta * scale_factor_ * PREV_H.DF (prev_out[i]);        2.1%}}); For_ (parallelize_, 0, Weight2io_.size (), [&] (const blocked_range& R) {for (int i = R.begin (); I < R . end ();                i++) {Const io_connections& connections = Weight2io_[i];                float_t diff = 0.0; for (auto connection:connections)//11.9% diff + = prev_out[connection.first] * Current_delta[connection.second];            Dw_[index][i] + = diff * SCALE_FACTOR_;        }        }); for (size_t i = 0; i < bias2out_.size (); i++) {Const std::vector<layer_size_t>& outs = bias2out_[            I];            float_t diff = 0.0;                for (auto o:outs) diff + = Current_delta[o];        Db_[index][i] + = diff;  }

The S2 layer, the input layer code snippet, and the S4 layer are the same.

The C1 layer code snippet is the same as the C3 layer.

Weight and Offset update code snippet:

    void Update (const vec_t& DW, const vec_t&/*hessian*/, vec_t &w) {        vec_t& g = get<0> (W);        For_i (W.size (), [&] (int i) {            g[i] + = dw[i] * Dw[i];            W[i]-= Alpha * Dw[i]/(STD::SQRT (G[i]) + EPS);}        );    

For the 60,000 training samples in the Mnist, perform the above operation sequentially, and update the weights and biases.

After performing 60,000 training samples per cycle, 10,000 test samples are tested and the recognition rate is obtained.

The total iteration is 30 times, then the final weights, biases and other related parameters are persisted to the specified file.

TINY-CNN Execution Process Analysis (MNIST)