"Artificial Neural Network Fundamentals" Why do Neural Networks choose "depth"?

Now that the "neural network" and "Deep neural network" are mentioned, there is no difference between the two, the neural network can not be "deep"? Our usual logistic regression can be thought of as a neural network with sigmoid (logistic) for output layer activation functions without hidden layers, and it is clear that the logistic regression is not deep. However, now the neural network is basically deep, that is, contains multiple hidden layers. Why?

1. Universality approximation theorem (general approximation theorem)

Any continuous function $f: r^n \to r^m$ can be represented by a neural network with only one hidden layer. (Enough hidden layer neurons)

Figure 1: Neural network with only one hidden layer

A neural network can be seen as a mapping from input to output, so since a neural network with only one hidden layer can represent any continuous function, why do we need multiple hidden layers of neural networks?

2. Why deep?

"Yes, shallow network can represent any function.

However, using the deep structure are more effective. "

We can call a neural network structure with only one hidden layer called shallow, and the neural network structure with multiple hidden layers is called deep.

In his machine learning video, Professor Li Hongyi presents an explanation called modularization (modular).

Figure 2:modularization

In a multilayer neural network, the first hidden layer learns the simplest feature, and then each hidden layer learns by using the features obtained from the previous layer, and the features that are learned are becoming more complex. Shown in 3 and 4.

Figure 3: Characteristics of different level-1

Each feature in the low level is more or less used at the level of high, so that for each of the high level features, only one set of low level features is trained. Yes, the low level feature is shared, which is equivalent to a single module that extracts the low levels feature to be called by the upper layer. For each of the high level features, you do not need to train the low level feature every time. This is the benefit of deep.

Figure 4: Different level Features-2

When comparing the effect of deep neural network with only one hidden layer neural network, it is necessary to control the number of trainable parameters of two networks, otherwise there is no comparability. In his machine learning video, Professor Li Hongyi, for example, deep performance is better with the same number of parameters, which means that deep parameters will be less if the same effect is achieved.

It is not denied that theoretically only a neural network with a hidden layer can realize the effect of deep neural network, but the training is more difficult than the deep neural network.

In fact, in a deep neural network, a hidden layer contains a number of neurons, such as AlexNet and VGG-16 the last fully connected layer of 4,096 neurons. At the same time in deep, fat is not to say no, but not as extreme as a layer of hidden layers, each of the number of hidden layer neurons is one of the parameters we need to adjust.

References

Li Hongyi machine learning

Universal approximation Theorem-wikipedia