Derivation of __BP algorithm by neural network and BP algorithm

Introduction

Neural network is the foundation of deep learning, and BP algorithm is the most basic algorithm in neural network training. Therefore, it is an effective method to understand the depth learning by combing the neural network structure and the BP algorithm. Reference UFLDL,BP derivation, neural network textbook. Neural network Structure

Typical network is shallow layer network, general 2~4 layer. Its structure is shown in the following illustration:

Suppose the neural network has an L layer, the 1th 1 layer is the input layer, the last layer (L layer) is the output layer, the middle 2, 3, ... L−1 2, 3, ... The L-1 is a hidden layer. There are sl,l=1,2 on each floor,...... L s_{l},l=1,2,...... L nodes, +1 + 1 nodes are used to represent the bias (bias); B (L) I b_{i}^{(l)} represents the right (bias) of the +1 + 1 node of the L-layer and the connection to the first node of the l+1 l+1 layer. The nodes of each layer are connected to the upper layer (not necessarily fully connected), and the connection weight W (l) IJ w_{ij}^{(L)} represents the weight between node J J on the layer L and the node I of L+1 l+1. The training set contains K K group input and output samples, {(x1→,y1→), (x2→,y2→),..., (xk→,y