Introduction to deep neural Networks

Introduction to deep neural Networks

Neural networks is a set of algorithms, modeled loosely after the human brain, which is designed to recognize patterns. They interpret sensory data through a kind of machine perception, labeling or clustering raw input. The patterns they recognize is numerical, contained in vectors, to which all real-world data, is it images, sound, text O R time series, must be translated.

Neural networks cluster and classify. You can think the them as a clustering and classification layer on top of the data you store and manage. They group unlabeled data according by similarities among the example inputs, and they classify data when they has a labe LED training set to work with.

As you think about one problem deep learning can solve, ask Yourself:what categories does I care about? What can I Act on? Those is labels:spam or not spam, good guy or bad guy, angry customer or happy customer. Then Ask:do I had data to accompany those labels?

For example, if want to identify a group of people at risk for cancer, your training set would be a list of cancer pat Ients along with all the data associated to their unique IDs, which could include everything from explicit features like a GE and smoking habits to raw data such as time series, their motion, or logs of their behavior online, which lik Ely indicate a great deal about lifestyle, habits and interests.

Searching for potential dating partners, the future major-league superstars, a company's most promising employees, or Potentia L bad actors, involves much the same process of contructing a training set by amassing vital stats, social graphs, raw Tex T communications, click streams, etc.

Neural Network Elements

Deep learning are a name for a certain set of stacked neural networks composed of several node layers. A node is a place where computation happens, loosely patterned on the human neuron and firing when sufficient stimuli pass Through. It combines input from the data with a set of coefficients, or weights, which either amplify or mute that input. These input-weight products was summed and the sum is passed through a node ' s so-called activation function.

Here's a diagram of what one node might look like.

A node layer is a row of those neuronlike switches this turn on or off as the input is fed through the net. Each layer's output is simultaneously the subsequent layer's input, starting from an initial input layer receiving your DA Ta.

Pairing adjustable weights with input features is what we assign significance to those features with regard to how the NETW Ork classifies and clusters input.

Key concepts of deep neural Networks

Deep-learning Networks is distinguished from the more commonplace single-hidden-layer neural networks by their depth< /c0>; That's, the number of node layers through which data passes in a multistep process of pattern recognition.

Traditional machine learning relies on shallow nets, composed of one input and one output layer, and at the most one hidden LA Yer in between. More than three layers (including input and output) qualifies as "deep" learning.

In deep-learning networks, each layer of nodes was trains on a distinct set of features based on the previous layer ' s OUTPU T. The further you advance to the neural net, the more complex the features your nodes can recognize, since they aggrega Te and recombine features from the previous layer.

This was known as feature hierarchy, and it is a hierarchy of increasing complexity and abstraction. It makes deep-learning networks capable of handling very large, high-dimensional data sets with billions of parameters pas Sing through nonlinear functions.

Above all, these nets was capable of discovering latent structures withinunlabeled, unstructured data, which is T He vast majority of data in the world. Another word for unstructured data is simply raw media; i.e. pictures, texts, video and audio recordings. Therefore, one of the problems deep learning solves best are in processing and clustering the world ' s raw, unlabeled media, Discerning similarities and anomalies in data, no human have organized in a relational database or put a name to.

For example, deep learning can take a million images, and cluster them according to their similarities:cats in one corner , Ice breakers in another, and in a third all the photos of your grandmother. This is the basis of so-called smart photo albums.

Now apply this same idea to other data Types:deep learning might cluster raw text such as emails or news articles. Emails full of angry complaints might cluster in one corner of the vector space, while satisfied customers, or spambot mes Sages, might cluster in others. This is the basis of various messaging filters, and can being used in Customer-relationship management (CRM). The same applies to voice messages.

With time series, data might cluster around normal/healthy behavior and anomalous/dangerous behavior. If The time series data is being generated by a smart phone, it'll provide insight into the users ' health and habits; If it is being generated by a autopart, it might be used to prevent catastrophic breakdowns.

Deep-learning Networks perform automatic feature extraction without human intervention, unlike most traditional m Achine-learning algorithms. Given that feature extraction are a task that can take teams of the data scientists years to accomplish, deep learning is a-a- To circumvent the chokepoint of limited experts. It augments the powers of small data science teams, which by their nature does not scale.

When training on unlabeled data, each node layer in a deep network learns features automatically by repeatedly trying to r Econstruct the input from which it draws its samples, attempting to minimize the difference between the network ' s guesses and the probability distribution of the input data itself. Restricted Boltzmann machines, for examples, create so-called reconstructions in this manner.

In the process, these networks learn to recognize correlations between certain relevant features and optimal results–the Y draw connections between feature signals and what those features represent, whether it is a full reconstruction, or with Labeled Data.

A deep-learning Network trained on labeled data can then is applied to unstructured data, giving it access to much more in Put than machine-learning nets. This was a recipe for higher performance:the more data a net can train on, and the more accurate it was likely to be. (Bad algorithms trained on lots of the data can outperform good algorithms trained on very little.) Deep Learning's ability to process and learn from huge quantities of unlabeled data give it a distinct advantage over Prev IOUs algorithms.

Deep-learning networks end in an output layer:a logistic, or Softmax, classifier this assigns a likelihood to a particula R outcome or label. We call this predictive, but it's predictive in a broad sense. Given raw data in the form of an image, a deep-learning network may decide, for example, which the input data is a percent Likely to represent a person.

Example:feedforward Networks

Our goal with using a neural net is to arrive at the point of least error as fast as possible. We are running a race. The starting line for the race was the state of which our weights was initialized, and the finish line was the state of thos E parameters when they is capable of producing accurate classifications and predictions.

The race itself involves many steps, and each of the those steps resembles the others. Just like a runner, we'll engage in a repetitive act over and over to arrive at the finish. Each step for a neural network involves a slight update with its weights, an incremental adjustment to their quantities.

A collection of weights, whether they was in their start or end state, was also called a model, because it is an attempt to Model data ' s relationship to Ground-truth labels, to grasp its structure. Models normally start out bad and end up less bad, changing over time as the neural network updates its parameters.

This is because a neural network are born in ignorance. It does not know which weights and biases would translate the input best to make the correct guesses. It has the to-start out with a guess, and then try the better guesses sequentially as it learns from its mistakes. (You can think of a neural network as a miniature enactment of the scientific method, testing hypotheses and trying again –only It is the scientific method with a blindfold on.)

Here's a simple explanation of what happens when a neural network learns (more precisely, we'll discuss a feedforward neu Ral NET, the simplest architecture to explain.)

Input enters the network. The coefficients, or weights, map this input to a set of guesses the network makes at the end.

input * weight = guess

Weighted input results in a guess-about-what-why input is. The neural then takes their guess and compares it to a ground-truth on the data, effectively asking an expert "did I get This right? "

ground truth - guess = error

The difference between the network ' s guess and the ground truth is it error. The network measures that error, and walks the "error back" on its model, adjusting weights to the extent that they contri Buted to the error.

error * weight‘s contribution to error = adjustment

The three pseudo-mathematical formulas above account for the three key functions of neural networks:scoring input, calcul ating loss and applying an update to the Model–to begin the three-step process over again. A neural network is a corrective feedback loop, rewarding weights, it correct guesses, and punishing weights That leads it to err.

The name for one commonly used optimization function This adjusts weights according to the error they caused is called "GR Adient descent. "

Gradient is another word for slope, and slope, in its-typical form on an X-y graph, represents how both variables relate to Each of the other:rise over run, the "change" in Money over the "change" in time, etc. In this particular case, the slope we care about describes the relationship between the network's error and a single weigh T i.e. that's, how does the error vary as the weight is adjusted.

To put a finer point on it, which weight would produce the least error? Which one correctly represents the signals contained in the input data, and translates them to a correct classification? Which one can hear "nose" in an input image, and know that should is labeled as a face and not a frying pan?

As a neural network learns, it slowly adjusts many weights so this they can map signal to meaning correctly. The relationship between network Error and each of the those weights are a derivative, de/dw, that M Easures the degree to which a slight change in a weight causes a slight the error.

Each weight are just one factor in a deep network, that involves many transforms; The signal of the weight passes through activations and sums over several layers, so we use the chain rule of calculus to March back through the networks activations and outputs and finally arrive at the weight in question, and its relationship to overall error.

The chain rule in calculus states that

In a feedforward network, the relationship between the net's error and a single weight would look something like this:

That's, given, variables, Error and weight, that's mediated by a third variable, activation through which the weight is passed, you can calculate how a change in weight affects a change in error by first calculating what a change inactivation affects a change in Error, and how a change in Wei Ght affects a change in activation.

The essence of learning in deep learning are nothing more than that:adjusting a model ' s weights in response to the error I T produces, until you can ' t reduce the error any more.

Logistic Regression

On a deep neural network of many layers, the final layer has a particular role. When dealing with labeled input, the output layer classifies each example, applying the most likely label. Each node in the output layer represents one label, and that node turns on or off according to the strength of the signal It receives from the previous layer ' s input and parameters.

Each output node produces-possible outcomes, the binary output values 0 or 1, because an input variable either deserve s a label or it does not. After all, the There is no such thing as a little pregnant.

While neural networks working and labeled data produce binary output, the input they receive is often continuous. That's, the signals that network receives as input would span a range of values and include any number of metrics, DEP Ending on the problem it seeks to solve.

For example, the a recommendation engine has a binary decision on whether to serve an ad or not. But the input it bases it decision on could include how much a customer have spent on Amazon in the last week, or how Ofte N that customer visits the site.

So the output layer have to condense signals such as $67.59 spent on diapers, and visits to a website, into a range BETW Een 0 and 1; i.e. a probability that a given input should is labeled or not.

The mechanism we use to convert continuous signals into binary output is called logistic regression. The name is unfortunate, since logistic regression are used for classification rather than regression in the linear sense t Hat Most people is familiar with. It calculates the probability that a set of inputs match the label.

Let's examine this little formula.

For continuous inputs to being expressed as probabilities, they must output positive results, since there is no such thing As a negative probability. That's why you see input as the exponent of  e  in The denominator–because exponents, our results To is greater than zero. Now consider the relationship of , e ' s exponent to the fraction 1/1. One, as we know, is the ceiling of a probability, beyond which our results can ' t go without being absurd. (We ' re 120% sure of that.)

As the input x that triggers a label grows, the expression e to the x shrinks toward zero, leaving us WI Th the fraction 1/1, or 100%, which means we approach (without ever quite reaching) absolute certainty that the label APPL ies. Input that correlates negatively with your output would have their value flipped by the negative sign on e' s Exponen T, and as that negative signal grows, the quantity eto the x becomes larger, pushing the entire fraction ever CLO Ser to zero.

With this layer, we can set a threshold above which an example are labeled 1, and below which it is not. can set different thresholds as prefer–a low threshold would increase the number of false positives, and a high O NE would increase the number of false negatives–depending on which side your would like to err.

Neural Networks & Artificial Intelligence

In some circles, neural networks is thought of as "brute force" AI, because they start with a blank slate and hammer thei R-the-through to an accurate model. They is effective, but to some eyes inefficient in their approach to modeling, which can ' t do assumptions about functio nal dependencies between output and input. That said, gradient descent are not recombining every weight with every other to find the best Match–it ' s method of PATHF Inding shrinks the relevant weight space therefore the number of updates by many orders of magnitude.

Introductory Resources

For people just getting started with deep learning, the following tutorials and videos provide a easy entrance to the fun Damental Ideas of Feedforward networks:

• Neural Networks demystified (A seven-video series)
• A Neural Network in Lines of Python
• A Step-by-Step backpropagation Example
• Restricted Boltzmann Machines

Introduction to deep neural Networks