Neural network Learning (ii) Universal Approximator: Feedforward Neural Networks

1. Overview

We have already introduced the earliest neural network: Perceptron. A very deadly disadvantage of the perceptron is that its linear structure, which can only make linear predictions (even if it does not solve the regression problem), is a point that was widely criticized at the time.

Although the perceptual machine can not solve the nonlinear problem, it provides a way to solve the nonlinear problem. The limitation of the perceptron comes from its linear structure, if we can add a nonlinear structure to it, such as to make a nonlinear transformation of the input, so that it can fit the nonlinear problem. So this is the forward neural network we're going to talk about.

2. Structure

The Forward neural Network (Feed-forward Neural Network) is a multi-layered networking structure that is typically three layers: the input layer, the hidden layer, and the output layer. The layer is connected to the layer by a weighted value (wire). Generally only the value of the previous layer can be used as input to the value of the latter layer, and in turn cannot, in other words, not have feedback (feed-back), which is also its name. This structure is very important, because the famous BP algorithm is based on this structure.

Each point (neuron) in the network has a value that is set to $a_j$. Where a of the input layer is equal to x, the other layer's a is determined by the weight value and the value of the previous layer. Each neuron does not directly use its own value when it participates in the calculation of the value of the neuron behind it, but passes a non-linear activation function (typically tanh or sigmoid), obtains the activation value $z_j$, and then participates in the calculation. The input layer is represented by x, and the output layer is generally denoted by y, and the weights are generally expressed as w (including bias). The following is an example of a three-tier network structure that represents these values.

　　

$a ^{(0)}_j$ = $x _j$

$z ^{(0)}_j$ = $h (a^{(0)}_j) $

　　　　　　

$a ^{(1)}_j$ = $\sum_{i}w^{(1)}_{ji}a^{(0)}_i$

$z ^{(1)}_j$ = $h (a^{(1)}_j) $

　　　　　　$a ^{(2)}_j$ = $\sum_{i}w^{(2)}_{ji}a^{(1)}_i$

　　　　　　$y _j$ = $\sigma (a^{(2)}_j) $

where $x_0$ = 1, $a _0$ = 1 (The purpose is to omit the bias entry).

Our example here is that the value of the latter layer is determined only by the value of the previous layer, which, of course, is not necessarily a definite one. As long as there is no feedback structure, it can be counted as the forward neural network. So here is the derivation except for a structure called the skip layer, where the current layer is not determined by the previous layer, but by the values of the first few layers. This network structure is as follows:

This structure looks like the input skips over the hidden layer's direct action and output, which is also the origin of its name.

3. Symmetry of weighted value

This is one of the few properties of neural networks and uses less. Taking the three-layer neural network as an example, assuming that the middle layer has m neurons, if we exchange the positions of two of those neurons (and the weights associated with them also move accordingly), then the network's mapping relation y = f (x) does not change. So for this three-layer network, there is a total of m! network and its equivalent. This is the symmetry of the network.

Neural network Learning (ii) Universal Approximator: Feedforward Neural Networks