A programmer's neural network reverse communication

It can be considered that artificial neural network is a meta function, it can receive a fixed number of digital input and generate a fixed number of digital output. In most cases, the neural network has a layer of hidden neurons in which the hidden neurons and the input neurons and the output neurons are fully connected. Related to each single hidden layer neuron and each individual output neuron is a set of weighted values and a single offset value. Weights and offsets determine the output values of a set of established input values.

When using a neural network to model existing data to predict new data, the main challenge is to find a set of weights and offsets that can produce the output values that best match the existing values. The most common technique for determining the weights and biases of optimal neural networks is called reverse propagation. Although many of the best references describe complex mathematical applications that support reverse propagation, there are few reference guides that programmers can use to clearly explain how to program back-propagation algorithms. This article explains how to implement reverse propagation. I used the C # language, but you should be able to refactor the code in this article in other languages with ease.

To understand what I'm talking about, it's best to look at the screenshot of the demo program shown in Figure 1. The demo program creates a neural network that has three input neurons, a hidden layer with four neurons, and two output neurons. A neural network with a hidden layer requires two activation functions. However, in many cases the two activation functions are the same, usually the sigmoid function. However, in this demo, to illustrate the relationship between activation functions and reverse propagation, I will use different activation functions: the sigmoid function "input to hide" and the tanh (hyperbolic tangent function) function of "Hide to output" calculation.

Figure 1 The reverse propagation algorithm is running