Learning how to Code neural Networks

Original: https://medium.com/learning-new-stuff/how-to-learn-neural-networks-758b78f2736e#.ly5wpz44d

The second post in a series of me trying to learn something new over a short period of time. The first time consisted of learning how to does machine learning in a week.

This time I ' ve tried to learn neural networks. While I didn ' t manage to does it within a week, due to various reasons, I did get a basic understanding of it throughout the Summer and autumn of 2015.

By the basic understanding, I mean that I finally know the code simple neural networks from scratch on my own.

In this post, I'll give a few explanations and the resources I ' ve used, in case you ' re interested in doing th Is yourself.

Step 1:neurons and forward propagation

Why is a neural network? Let's wait with the network part and start off with one single neuron.

a neuron is a function, it takes a few inputs and calculates an output.

The circle below illustrates an artificial neuron. Its input is 5 and its output is 1. The input is the sum of the "three synapses connecting to the neuron" (the three arrows at the left).

At the far left we see both input values plus a bias value. The input values is 1 and 0 (the green numbers), while the bias holds a value of-2 (the brown number).

The

inputs here might is numerical representations of the different features. If we ' re building a spam filter, it could be wether or not the e-mail contains more than one capitalized WORD and wether or Not it contains the word ' viagra '.

The inputs is then multiplied by their so called weights, which is 7 and 3 (the blue numbers).

Finally we add it up with the bias and end up with a number, in this case:5 (the red number). This is the input to our artificial neuron.

The neuron then performs some kind of computation on this number?—? in our case the Sigmoid function, and then spits off an Output. This happens to is 1, as Sigmoid of 5 equals to 1, if we round the number up (more info on the Sigmoid function follows LA ter).

If This is a spam filter, the fact that we ' re outputting 1 (as opposed to 0) probably means that the neuron have lab Eled the text as ' spam '.

A Neural network illustration from Wikipedia.

IF you connect a network of these neurons together, you had a neural network, which propagates forward?—? from input OUTPU T, via neurons which is connected to all other through synapses, like on the image to the left.

I can strongly recommend Thewelch Labs videos on YouTube for getting a better intuitive explanation of the this process.

Step 2:understanding the Sigmoid function

After you ' ve seen the Welch Labs videos, it a good idea to spend some time watching Week 4 of the Coursera's machine Lear Ning course, which covers neural networks, as it ' ll give you more intuition of what they work.

The course is fairly mathematical, and its based around Octave, while I prefer Python. Because of this, I didn't do the the programming exercises. Instead, I used the videos to help me understand what I needed to learn.

The first thing I realized I needed to investigate further is the Sigmoid function, as this seemed to is a critical part of many neural networks. I knew a little bit about the function, as it is also covered in Week 3 of the same course. So I went back and watched these videos again.

The Sigmoid function simply maps your value (along the horizontal axis) to a value between 0 and 1.

But watching videos won ' t get to all the the the The. To really understand it, I felt I needed to code it from the ground up.

So I started to code a logistic regression algorithm from scratch (which happened to use the Sigmoid function).

It took a whole day, and it's probably not a very good implementation of logistic regression. But this doesn ' t matter, as I finally understood how it works. Check the code here.

You don ' t need to perform this entire exercise yourself, as it requires some knowledge about and cost functions and Gradie NT descent, which you might is not a.

But make sure you understand how the Sigmoid function works.

Step 3:understanding backpropagation

Understanding how a neural network works from input to output isn ' t that difficult to understand, at least conceptually.

More difficult though, was understanding how the neural network actually learns from looking at a set of data samples.

The concept is called backpropagation.

This

essentially means the "how to Wrong" the network guessed, and then adjust the networks weights AC cordingly.

The weights were the blue numbers on our neuron in the beginning of the article.

This process happens backwards, because your start at the end of the network (observe how wrong the networks ' guess ' is), a nd then move backwards through the network and while adjusting the weights on the the-the-the-same, until you-finally reach the inputs.

To calculate the hand requires some calculus, as it involves getting some derivatives of the networks ' weights. The Kahn Academy calculus courses seems like a good-to-start, though I haven ' t used them myself, as I took calculus on University.

Note:there is a lot of libraries this calculates the derivatives for you and so if you're d like to start coding neural Networks before completely understanding the math, you ' re fully able to does this as well.

Screehshot from Matt mazurs tutorial on backpropagation.

The three best sources I found for understanding backpropagation is these:

• A Step by step backpropagation Example?—? by Matt Mazur
• Hackers Guide to neural nets-by Andrej karpathy
• Neuralnetworksanddeeplarning?—? by Michael Nielsen

You should definitely code along while you ' re reading the articles, especially the first ones. It ' ll give you some sample code-to-look-back at when you ' re confused in the future.

Plus, I can ' t really emphasize this enough:

You

don ' t learn much by reading about neural nets, and you need to practice it to make the knowledge stick.

The third article is also fantastic, but I ' ve used this more as a wiki than a plain tutorial, as it's actually an entire B Ook. It contains thorough explanations all the important concepts in neural networks.

These articles would also help you understand important concepts as cost functions and gradient descent, which play equally Important roles in neural networks.

Step 4:coding your own neural networks

In some articles and tutorials you'll actually end up coding small neural networks. As soon as you ' re comfortable with that, I-recommend you-to-go all in the this strategy. It's both fun and an extremely effective the learning.

One of the articles I also learned a lot from is a neural Network in all Lines of Python by Iamtrask. It contains an extraordinary amount of compressed knowledge and concepts in just one lines.

Screenshot from the Iamtrask tutorial

After your ' ve coded along with this example, you should does as the article states at the bottom, which are to implement it on Ce again without looking at the tutorial. This forces really understand the concepts, and would likely reveal holes in your knowledge, which isn ' t fun. However, when your finally manage it, you'll feel like you ' ve just acquired a new superpower.

A Little side note:when doing exercises I was often confused by the vectorized implementations some tutorials use, As it requires a little bit of linear algebra to understand. Once again, I turned myself back to the Coursera ML course, as Week 1 contains a full section of linear algebra review. This helps-understand how matrixes and vectors is multiplied in the networks.

When you've done it, you can continue with the Wild ML tutorial, by Denny Britz, which guides you through a little more Robust neural network.

Screenshot from the Wildml tutorial.

At the could either try and code your own neural network from scratch or start playing around with some of the Networks you has coded up already. It's great fun-to-find a dataset that interests-you and try-to-make some predictions with your neural nets.

To get a hold of the a dataset, just visit my side project datasets.co (←shameless self promotion) and find one.

Visit datasets.co to get hold of a dataset

Anyway, the point is so you ' re now better off experimenting with stuff that interests you rather than following my Advic Es.

Personally, I ' m currently learning how to use Python libraries that makes it easier to code up neural networks, like Thean O, lasagne and nolearn. I ' m using this to does challenges on Kaggle, which are both great fun and great learning.

Good luck!

And don ' t forget to press the heart button if you liked the article:)

Thanks for reading! My name is Per Borgen and I mostly write about learning new stuff.

Feel free to reach out to me if you have and questions or would like to connect, either through Twitter, GitHub or just pl Ain old Email:[email protected]

Learning how to Code neural Networks