Neural network and deep Learning notes (1)

Neural network and deep learning the book has been read several times, but each time there will be a different harvest.

The paper of DL field is changing rapidly. There's a lot of new idea coming out every day, I think. In-depth reading of classic books and paper, you will be able to find Remian open problems. So there's a different perspective.

Ps:blog is a summary of important contents in the main extract book.

Summary section

1. Neural networks, a beautiful biologically-inspired programming paradigm which enables a computer to learn from observation Al data.

2. Deep learning, a powerful set of techniques for learning in neural networks.

   CHAPTER 1 Using neural nets to recognize handwritten digits

3. The neural network uses the examples to automatically infer rules for recognizing handwritten digits.

#

The exact form of the active function isn ' t important-what really matters is the shape of the function when plotted.< /c0>

#

4.The Architecture of Neural networks

2. The design of the input and output layers of a neural network is often straightforward, there can being quite an art to the D Esign of the hidden layers. But researchers has developed many design heuristics for the hidden layers, which help people get the behaviour they want Out of their nets.

4. Learning with gradient descent

   1. The aim's our training algorithm would be is to minimize the cost C as A function of the weights and biases. We'll do the using an algorithm known as gradient descent.
   2. Why introduce the quadratic cost? It's a smooth function of the weights and biases in the network and it turns off to being easy-to-figure out how to make smal L changes in the weights and biases so as to get a improvement in the cost.
   3. MSE cost function is ' t ' the only cost function used in neural network.
   6. Mini Batch : SGD randomly picking out a small number m of randomly chosen Trai ning inputs; Epoch: randomly choose mini-batch and training until we ' ve exhausted the training inputs.

5. thinking about Hyper-parameter choosing
   "If we were coming to this problem for the first time then there wouldn ' t is much in the output to guide us on what to do. We might worry not only on the learning rate, but on every other aspect's our neural network. We might wonder if we ' ve initialized the weights and biases in a-a-to-the-makes it hard for the network to learn? Or maybe we don ' t have enough training data to get meaningful learning? Perhaps we haven ' t run for enough epochs? Or Maybe it ' s impossible for a neural network with this architecture to learn to recognize handwritten digits?

   Maybe the learning rate was too low? Or, maybe, the learning rate was too high?

   When you ' re coming to a problem for the first time, you're not always sure.
   The lesson to take away from the () debugging a neural network is not trivial, and, just as for ordinary program Ming, there is a art to it. You need to learn this art of debugging in order to get good results from neural networks. More generally, we need-develop heuristics for choosing good hyper-parameters and a good architecture. "

7. inspiration from face Detection:
   "The end result is a network which breaks down a very complicated question-does this image show a face or Not-into ver Y simple questions answerable at the level of a single pixels. It does this through a series of many layers, with early layers answering very simple and specific questions about the INP UT image, and later layers building up a hierarchy of ever more complex and abstract concepts. Networks with this kind of many-layer structure-two or more hidden layers-are called deep neural Networks. "

CHAPTER 2 How the backpropagation algorithm works

2. BackPropagation (BP): A fast algorithm for computing, the gradient of the cost function.
   

4. For backpropagation-to-work we need-to-make, main assumptions about the form of the cost function.

   1. Since What BP let us do are compute the partial derivatives for a single training example,so we need so the cost function Can be written as a average over all individual example.
   2. It can written as a function of the outputs from the neural network. Since y is not something which the neural network learns.

6. The four fundamental equations behind backpropagation
   

8. What's clever about BP are that it enables us to simultaneously compute all the partial derivatives using just one forward Pass through the network, followed by one backward pass through the network.

10. What indeed the BP does and how someone could ever has discovered BP?

    1. A small perturbations would cause a change in the Activation,then next and so on all the the-the-through to-causing a change in The final layer,and then the cost function.
       
       A clever the keeping track of small perturbations to the weights (and biases) as they propagate through the network, re Ach the output, and then affect the cost.

    2. (not to be continued ...) ）

Neural network and deep Learning notes (1)