What is the advantages of ReLU over sigmoid function in deep neural network?

The state of the art of Non-linearity was to use ReLU instead of sigmoid function in deep neural network, what is the Adva Ntages? I know that training a network when ReLU was used would be faster, and it's more biological inspired, what's the other A Dvantages? (That's, any disadvantages of using sigmoid)?

Best answer in Stackexchange:

Additional major benefits of Relus is sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU ish =max (0, A) h=max (0,a)  where  a = W x + b "> a= Wx+ b a=wx+b.

One major benefit is the reduced likelihood of the gradient to vanish. This arises When  a > 0 " > A>0 a>0. The regime the gradient has a constant value. In contrast, the gradient of sigmoids becomes increasingly small as the absolute value of x increases. The constant gradient of relus results in faster learning.

The other benefit of Relus is sparsity. sparsity arises When  a & #x2264; 0 " > A≤0 a≤0. The more such units this exist in a layer of the more sparse the resulting representation. Sigmoids on the other hand is always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to is more beneficial than dense representations.

reference:http://stats.stackexchange.com/questions/126238/ What-are-the-advantages-of-relu-over-sigmoid-function-in-deep-neural-network

ReLU

the full name of Relu is rectified linear unit. The above answer basically covers several aspects of it over sigmoid function:

1. Faster
2. More biological inspired
3. Sparsity
4. Less chance of vanishing gradient (gradient vanishing problem)

An earlier DL using sigmoid or Tanh activation functions was doing unsupervised learning because the gradient vanishing problem problem would not converge. Relu This is not the problem.

What is the advantages of ReLU over sigmoid function in deep neural network?