Neural networks and deep learning (III.)--Reverse propagation works

How the reverse propagation algorithm works

In the previous article, we saw how neural networks learn through gradient descent algorithms to change weights and biases. However, before we discussed how to calculate the gradient of the cost function, this is a great pity. In this article, we will introduce a fast computational gradient algorithm called reverse propagation.

Neural networks that are learned using the reverse propagation algorithm are much faster than other early methods, which allows the use of neural networks to solve previously unresolved problems. Today, the inverse propagation algorithm is the most important part of the neural network.

You can completely ignore the reverse propagation algorithm and treat it as a black box to use the neural network. So why do we need to know this algorithm in detail. This is of course to understand the learning process of neural networks. The core of the inverse propagation algorithm is an expression that calculates the partial derivative of the cost function for any weight or deviation. This expression tells us how quickly the cost function changes when weights and deviations are changed. Although this expression is complex, it has its own mathematical beauty, with each element having a natural visual interpretation. So the reverse propagation is not just a fast algorithm for learning, it actually gives us an insight into the changes in the behavior of the network as it changes weights and biases. This is where we understand the meaning of the reverse propagation algorithm in detail.

Warm-up: a fast way to calculate neural network output based on matrix

Before we discuss the inverse propagation algorithm, we use a matrix-based method to calculate the output of neural networks for preheating. This allows us to be more familiar with the various notations and representations when learning to reverse-propagate.

First we give a clear sign of the weight. We will use WLJK W^{L}_{JK} to define the weights between the K-K neurons (l−1) (L-1) layer and the J-J neurons of the L-l layer. For example, the following figure shows the weights between the 4th neuron in the 2nd layer and the 2nd neuron in the 3rd layer:

This sign is just beginning to go around, it takes a little time to get acquainted. After a period of mastery, it is easy and natural to find such expressions. This is a very awkward mark in the order of J and K, you might as well as I just started to feel that J and K locations need to change a bit more comfortable, but below I will tell you why.

We use a similar notation for the activation value of the network (the neuron's output value) and the deviation. We use the BLJ b_j^l to represent the deviation of the J-J neurons of the L-l layer, using the ALJ a_j^l to represent the activation values of the J-J neurons of the L-l layer. The following illustration shows an example:

With these tokens, the activation value ALJ A_j^l can be calculated by the activation value of the l−1 L-1 layer:

The summation is performed on all neurons in the l−1 l-1 layer. In order to rewrite the above expression using the matrix, we define a weight matrix WL w^l For each layer L, each element of the matrix represents the weight of the neuron connected to the L-l layer, that is, the value of the element in the K-K column of the J-J Row is WLJK W^L_{JK}. Similarly, we define a deviation matrix (vector) BL b^l for each layer. I think you can give it your own definition, obviously it is a column of more than one row of the matrix, its constituent elements of the expression value is the Blj b_j^l in the previous paragraph. Finally we also give the activation matrix (vector) Al A^l, which has a value of each element