Realization of handwritten numeral recognition (mnist) by neural network

first, the Origin

Originally wanted to follow the traditional recursive algorithm to achieve maze game--> genetic algorithm to achieve maze game--> neural network maze game ideas, in this article also write how to use the neural network to achieve the maze, but the study, feel some trouble is not very good, so I chose the more common way, Realization of handwritten digit recognition (so-called mnist). introduction of Artificial neural network

From the small to the ants (no specific number, some say that the ant brain has 250,000 nerve cells, also said to be 500,000), large to elephants, blue whales and other animals, the brain contains a large number of neurons. The various behaviors or skills of humans or animals are more or less due to the neural networks that are connected by a large number of neurons. and the artificial neural network is The imitation of biological brain to operate, so first to understand the laughter of the biological brain (neural network, neurons). 2.1 Biological Neural network

First Look at a table (from "AI in Game Programming", chapter seventh, section II, Table 7.1):

Animal	Number of nerve cells (order of magnitude)
Snail	10,000 (=104)
Bees	100,000 (=105)
Bee Sparrow	10, 000, 000 (=107)
Mouse	100, 000, 000 (=108)
Human	10, 000, 000, 000 (=1010)
Elephant	100, 000, 000, 000 (=1011)

From the table can even small as the snail also has a large number of nerve cells, the nature of the animals need to perceive the environment, adapt to the environment and so on, we must have a strong "processing" system to deal with a variety of unexpected events.

Judging by the number of nerve cells in humans and elephants, it can be inferred that the intelligence intensity shown is not necessarily proportional to the number of neurons or that the more nerve cells are, the smarter they are (purely personal opinions). Just like the artificial neural network, machine learning, depth learning and other networks, not the more the number of layers, the longer the neuron node the better.

The biological nerve cells are shown in the following illustration:

The signal transduction between nerve cells is accomplished through electrochemical processes, these signals pass from the axon of a nerve cell to the dendrites of the next nerve cell through a synapse (a ring or a globular structure that contacts other nerve cells, dendrites, and synapses), and then enter the cell body from the dendrites of the nerve cells, Finally, according to the actual situation to choose whether to pass to the next nerve cell. Here we need to talk about two states of nerve cells excited and not excited (ie inhibition). Nerve cells use a method we don't yet know to add all the signals from the dendrites into the cell body, if the signal sum reaches a threshold value (Note: "Threshold", Yu, tone, instead of "valve value" Oh, didn't know before, asked a little Niang, just know the difference is very big), will stimulate nerve cells into the excited state , the electrical signals pass from the current nerve cells through their axons to the next nerve cell, whereas the current nerve cells are in a state of inhibition.

These are just simple or pure nerve cell models, in the actual brain, electrical signals are transmitted in nerve cells, not only from the axon of one cell to the dendrites of another cell, but from the other direction of transmission: axon-dendrites-dendritic-cellular axon-cell body-dendritic dendrites-cell body-axon

For the biological neural network interested can go to the Internet to find the corresponding information, here is just a simple introduction, let us to the basic structure of nerve cells and nerve impulse (electrical signal) transmission of the basic way to have an image of understanding. The emphasis is also on the description of the artificial neural network. 2.2 Artificial Neural network--nerve cells

Artificial neural Network (ann,artificial Neuron Network) is a neural network structure simulating the biological brain, which is composed of many small structural units called artificial nerve cells (Artificial Neuron, also known as artificial neurons). The artificial nerve cells here can be seen as a simplified version or model of the biological nerve cells.

As shown above, it is a schematic diagram of an artificial neural cell, in which X1 ... xn: the input of nerve cells, which is the signal of input neurons. W1 WN: The weight of each input is the same as the thickness and strength of each axon and dendrites in a biological neural network. B: Bias weight threshold: bias (You can view threshold * B as the threshold of biological nerve cells mentioned above) blue part: Cell body. The yellow sphere is the sum of all the input signals. The red part is the excitation function of the signal after summation (that is, to reach the threshold is in the excited state, the contrary inhibition, of course, as artificial nerve cells, its excitation function is many, step (type) excitation function, sigmoid (S-type) excitation function, hyperbolic tangent (tanh) excitation function, Relu (rectified Linear Units) excitation function, etc.).

The mathematical expression (or model) of the original Perceptron (for example, what is perceived as a perceptual machine) is as follows:

Output={01if∑jwjxj<thresholdif∑jwjxj≥threshold o u t p u t = {0 I f∑j w j x J < T h R e s H o l d 1 I f∑j W j x J≥t h r e s h o l D output=\begin{cases} 0 &if \sum_{j} w_jx_j

As you can see from the expression above, the threshold (threshold) is also a parameter of the neural network. More specifically, to make nerve cells in a state of excitement (i.e. the output of the expression above), the following conditions need to be met:
W1x1+w2x2+w3x3+⋯+wnxn≥t W 1 x 1 + w 2 x 2 + w 3 x 3 +⋯+ w n x n≥t w_1x_1+w_2x_2+w_3x_3+\cdots+w_nx_n \geq t

Note: T:threshold (threshold value)

Network in the process of learning, the artificial Neural Network (ANN) Ownership weight needs to evolve (evolution), threshold (threshold) of the data should not be exceptional, also need to do the corresponding evolution, so it is necessary to threshold (threshold) into the form of weight. Subtract t from both sides of the equation above to obtain the following form:

W1x1+w2