Neural networks used in machine learning (iii)

A simple model of neurons

Idealized neurons
to model things we have to idealize them (e.g. atoms)
–idealization removes complicated details that is not essential for understanding the main principles.
– it allows us to apply mathematics and to make analogies to other, familiar systems.
–once we understand the basic principles, its easy-to-add complexity to make the model more faithful.
• It is often worth understanding models that's known to being wrong (but we must isn't forget that they was wrong!)
–e.g. Neurons that communicate real values rather than discrete spikes of activity.
Linear neurons
these is simple but computationally limited
–if we can make them learn we could get insight into more complicated neurons

Binary threshold neurons two-value threshold neuron
Mcculloch-pitts (1943): Influenced Von Neumann.
–first compute a weighted sum of the inputs.
–then send out a fixed size spike of activity if the weighted sum exceeds a threshold.
–mcculloch and Pitts thought that each spike are like the truth value of a proposition and compute the truth value of Anot Her proposition!

There is equivalent ways to write the equations for a binary threshold neuron:

Rectified Linear neurons
(sometimes called linear threshold neurons)
They compute a linear weighted sum of their inputs.
The output is a non-linear function of the total input


Sigmoid neurons this neuron is often used
These give a real-valued output is a smooth and bounded function of the their total input.
–typically They use the logistic function
–they has nice smooth derivatives, the derivatives change continuously and they ' re nicely behaved and they make it easy To do learning.



Stochastic binary neurons random binary neurons

These use the same equations as logistic units.
–but They treat the output of the logistic as the probability of producing a spike in a short time window.

Instead of outputtiing that probability as a real number they actually make a probabilistic decision, and so what they acu Tally output is either a one or a zero. They ' re intrisically random. So they ' re treating the P as the probability of producing a and not as a real number.

we can do a similar trick for rectified (corrected) linear units:
, haven output is treated as the Poisson rate for spikes.

The rectified linear unit determines the rate, but intrinsic randomness in the unit determines when the spikes is ACTU Ally produced.

Neural networks used in machine learning (iii)