Neural NETWORKS, part 3:the NETWORK

Neural NETWORKS, part 3:the NETWORK

We have learned on individual neurons in the previous section, now it's time to put them together to form an actual neu RAL Network.

The idea was quite simple–we line multiple neurons up to form a layer, and connect the output of the first layer to the I Nput of the next layer. Here are an illustration:

Figure 1:neural the network with the hidden layers.

Each red circle in the diagram represents a neuron, and the blue circles represent fixed values. From there is four columns:the input layer, the hidden layers, and an output layer. The output from neurons on the previous layer is directed to the input of each of the neurons in the next layer.

We have a 3 features (vector space dimensions) in the input layer, the We use for learning:x1 ,< Span id= "mathjax-span-7" class= "Mrow" > x2  and  x 3 . The first hidden layer has 3 neurons, the second one has 2 neurons, and the output L Ayer has 2 output values. The size of these layers is up to you–on complex real-world problems we would use hundreds or thousands of neurons in EA CH layer.

The number of neurons in the output layer depends the task. For example, if we had a binary classification task (something is true or false) and we would only had one neuron. But if we had a large number of possible classes to choose from, our network can has a separate output neuron for each C Lass.

The network in Figure 1 was a deep neural network, meaning. It has both or more hidden layers, allowing the network to L Earn more complicated patterns. Each neuron in the first hidden layer receives the input signals and learns some pattern or regularity. The second hidden layer, in turn, receives input from these patterns from the first layer, allowing it to learn "patterns of patterns "and higher-level regularities. However, the cost of adding more layers are increased complexity and possibly lower generalisation capability, so finding T The He right network structure is important.

Implementation

I have implemented a very and simple neural network for demonstration. You can find the code Here:SimpleNeuralNetwork.java

The first important method is Initialisenetwork (), which sets up the necessary structures:

123456789

public void initialiseNetwork(){    input = new double[1 + M]; // 1 is for the bias    hidden = new double[1 + H];    weights1 = new double[1 + M][H];    weights2 = new double[1 + H];    input[0] = 1.0; // Setting the bias    hidden[0] = 1.0;}

M is the number of features in the feature vectors, and H is the number of neurons in the Hidden Lay Er. We add 1 to these, since we also use the bias constants.

We represent the input and hidden layer as arrays of doubles. For example, Hidden[i] stores the current output value of the i-th neuron in the hidden layer.

The first set of weights, between the input and hidden layer, is stored as a matrix. Each of the(1+M) Neurons in the input layer connects toHNeurons in the hidden layer, leading to a total of(1+M)xH Weights. We only has one output neuron, so the second set of weights between hidden and output layers is technically a (1+H)x1 Matrix, but we can just represent that as A vector.

The second important function is Forwardpass (), which takes a input vector and performs the computation to reach an OUTPU T value.

123456789101112131415

public void forwardPass(){    for(int j = 1; j < hidden.length; j++){        hidden[j] = 0.0;        for(int i = 0; i < input.length; i++){            hidden[j] += input[i] * weights1[i][j-1];        }        hidden[j] = sigmoid(hidden[j]);    }    output = 0.0;    for(int i = 0; i < hidden.length; i++){        output += hidden[i] * weights2[i];    }    output = sigmoid(output);}

The first For-loop calculates the values in the hidden layer, by multiplying the input vector with the weight vector and a Pplying the sigmoid function. The last part calculates the output value by multiplying the hidden values with the second set of weights, and also Applyi ng the sigmoid.

Evaluation

To test out this network, I has created a sample dataset using the database at quandl.com. This dataset contains sociodemographic statistics for 141 countries:

• Population density (per suqare km)
• Population growth rate (%)
• Urban Population (%)
• Life expectancy at birth (years)
• Fertility rate (births per woman)
• Infant mortality (deaths per births)
• Enrolment in tertiary education (%)
• Unemployment (%)
• Estimated control of corruption (score)
• Estimated government effectiveness (score)
• Internet users (per 100)

Based on this information, we want to train a neural network that can predict whether the GDP per capita are more than aver Age for the country (label 1 if it is, 0 if it's not).

I ' ve separated the dataset for training (121 countries) and testing (+ countries). The values are been normalised, by subtracting the mean and dividing by the standard deviation, using a script from a pre Vious article. I ' ve also pre-trained a model that we can load into this network and evaluate. You can download these from here:original data, training data, test data,pretrained model.

You can then execute the neural network (remember to compile and link the binaries):

1	java neuralnet.SimpleNeuralNetwork data/model.txt data/countries-classify-gdp-normalised.test.txt

The output should is something like this:

12345678910

label:0    prediction:0.01 label:0     prediction:0.00 label:1    prediction:0.99 label:0    prediction:0.00 ... label:0    prediction:0.20 label:0    prediction:0.01 label:1    Prediction: 0.99 label:0    prediction:0.00 accuracy:0.9

The network is in verbose mode, so it prints out the labels and predictions for each test item. At the end, it also prints out the overall accuracy. The test data contains positive and negative examples; A random system would has had accuracy 50%, whereas a biased system would has accuracy 65%. Our network managed 90%, which means it had learned some useful patterns in the data.

In this case we simply loaded a pre-trained model. In the next section, I'll describe how to learn this model from some training data.

Neural NETWORKS, part 3:the NETWORK