On the activation function in deep learning

Situ

the role of the activation function

First, the activation function is not really going to activate anything. In the neural network, the function of activating function is to add some nonlinear factors to the neural network, so that the neural network can better solve the more complicated problems.
For example, in the following question:

As pictured above (picture source), in the simplest case, the data is linearly divided, and a straight line is required to properly classify the sample.

But if things get a little more complicated. In the image above (image source), the data becomes a linear non-divided case. In this case, a simple straight line is no longer able to classify the sample well.

So we try to introduce non-linear factors and classify the samples.

Similar in neural networks, we need to introduce some nonlinear factors to better solve complex problems. And the activation function is exactly the one that can help us introduce nonlinear factors, so that our neural network can better solve the more complicated problems.

definition of activation function and its related concepts

In a paper by ICML2016 noisy Activation functions, the author defines the activation function as an almost everywhere h:r→r.

In practical applications, we will also cover some of the following concepts:
A. Saturation
When an activation function h (x) satisfies limn→+∞h′ (x) =0 limn→+∞h′ (x) =0 We call the right saturation.

When an activation function h (x) satisfies limn→−∞h′ (x) =0 limn→−∞h′ (x) =0 We call the left saturation. When an activation function satisfies both left saturation and saturation, we call it saturation.

B. Hard saturation and soft saturation
For any x x, if there is a constant C C, when the X>c x>c is constant h′ (x) =0 h′ (x) =0 is called the right hard saturation, when