Neural Network and machine learning--basic framework Learning

August 6, 2016, Saturday

• The nature and ability of neural networks:
• Nonlinearity and linearity, nonlinearity are very important properties
• Input and output mapping (mapping is an interesting concept, such as a matrix can also be considered a mapping)
• Self-adaptability (adjust neuron weights to adapt to environmental changes, i.e. auto-tuning)
• Ability to context information
• Fault tolerance (can be understood as redundancy of parameters to ensure proper operation)
• Evidence response, which not only provides decision choices, but also provides confidence-based information (used to reject what may appear to be ambiguous patterns and improve classification performance)
• VLSI implementations (highly layered, IBM seems to have a chip recently)
• Consistency of analysis and design (common properties as information processor)
• Neuro-Biological analogy
• Neuron Model:
• Synaptic (input vs. intrinsic bias)
• Adder (summation node)
• Activation function (can be used to represent nonlinear characteristics)
• Threshold function
• Sigmoid class functions (differentiable, such as logistic functions and tanh hyperbolic tangent functions)
• Statistical models of neurons:
• The concept of pseudo-temperature is used to represent thermal fluctuation parameters, not physical temperature
• Rules for the presentation of knowledge:
• Similar inputs in similar categories should usually produce similar representations in the network
• Representation of similarity: Euclidean distance and inner product, opposite to each other to indicate similarity
• For group data, the Mahalanobis distance is used to represent (unknown origin)
• Networks can separate different kinds of input vectors to give a very large difference in expression
• If a feature is important, then the network indicates that the vector will involve a large number of neurons
• Consider an example: When a radar detects a target in a hybrid state, the detection performance is measured by two probability forms:
1. Detection probability, the probability of judging the occurrence of the target when the target exists
2. Probability of false alarm and judgment of the occurrence of target when the target does not exist
• Neyman-pearson criterion: The probability of false alarm is not exceeding the limit of pre-specified value, the detection probability reaches the maximum value.
• This means that there should be a large number of neurons involved in the decision when the real target exists, which guarantees the high accuracy of the decision and the fault tolerance of the wrong neuron.
• If there is a priori information and invariance, it should be attached to the network design, do not need to learn this information and simplify the network design
• This rule causes the network to have a specific structure for the following reasons:
1. The structure of visual and auditory networks of known organisms is particularly
2. Fewer free parameters for a particular network, less training data, faster learning and better generalization performance
3. Ability to speed up specific network information transfer rates (network throughput)
4. Low-cost, small-scale construction of a particular structure network
• How to add a priori information to a neural network:
• There is no effective rule to achieve
• A special process can be implemented:
• Restricting the network structure by using a local connection to the receiving domain
• Limiting the choice of synaptic weights by sharing weights (with good collateral effect, reducing the number of network free parameters)
• Through the above two processes, the resulting local domain is convolution and form, called Convolutional network
• How to establish invariance in neural networks
• Problem leads to:
• Image rotation--does not affect human recognition is the same image
• Due to the Doppler effect, the echo of the active target detected by the radar has a frequency offset--without affecting the monitoring of the activity target.
• A change in the intonation of a person's voice--without affecting the understanding of the meaning of the sentence
• The main task of pattern recognition is to design a classifier that is invariant to these transformations, with the following three techniques:
1. Structural invariance: The design of the structure has taken into account the insensitivity to the transformation, and the disadvantage is that the number of network connections becomes large
2. Training invariance: Different sample training parameters for the same target; disadvantage: It is not guaranteed that the transformation of other types of targets is invariant, and the calculation is more demanding, especially the high dimensional space
3. Invariant feature space: it relies on the ability to extract features that represent the essential information content of the data, and the feature will remain unchanged for the input transformation. The advantages are as follows:
1. The number of features applied to the network is reduced to the desired level
2. Network design Requirements Relaxed
3. Invariance of all target-known transformations is guaranteed

From for notes (Wiz)

Neural Network and machine learning--basic framework Learning