Deep Learning (Bengio) First chapter reading notes

Some Definitions/understandings:

This piece of things or suggest reading, the book explained very clearly, alone may be difficult to understand what is.

Knowledge spectrum : To hard-code the knowledge about the world in a formalized language, computers can automatically use logical reasoning rules to understand declarations through these formal languages.

However, relying on hard-coded knowledge system to face all kinds of difficulties, so people want the AI system has its own ability to acquire knowledge.

Machine learning : The ability of an AI system to acquire knowledge by itself, that is, the ability to extract patterns from raw data.

The performance of a simple machine learning algorithm relies heavily on the representation of a given data, which is not as accurate as it is for the learning effect.

Expression Learning (representation Learning): Use machine learning to discover the expression itself.

The learning representation is often better than a manual design, and requires minimal manual intervention.

variation Factor (factor of Variation): When designing features or learning feature algorithms, our goal is usually to isolate the variables that explain the observed data.

Deep learning allows a computer to construct complex concepts through simpler concepts. (The examples in the comparison book can be understood clearly)

The idea of learning the correct representation of data is a point of view for explaining deep learning.    Another point of view is that depth allows the computer to learn a multi-step computer program, representing that each layer can be thought of as parallel execution of another set of instructions after the computer's memory state. The second point of view is not understood.

distributed Representation (distributed representation) key concepts in the core connection mechanism of this book . The idea is that each input should be represented by many characteristics, and each feature should participate in many representations of possible inputs.

This concept is a clear example: suppose we have a vision system that identifies red, green, or blue cars, trucks, and birds. One way to represent these inputs is to combine nine possible combinations: red trucks, red cars, red birds, green trucks, and so on using separate neurons or hidden cells to activate. This requires nine different neurons, and each neuron must independently learn the concept of color and object identity. One way to improve this is to use a distributed representation that describes the color with three neurons and three neurons that describe the identity of the object. This requires only 6 neurons instead of 9, and the red neurons can learn red from the images of cars, trucks, and birds, not just from a particular category of images.

The diagram shows the relationship between AI, machine learning, presentation learning, deep learning

Two waves of deep learning:

Cybernetics (cybernetics)????

Connection mechanism (connectionism)

Reasons for deep learning success:

1. The amount of data that can be used is increasing.

And because of the advent of the big data age, data is more easily captured.

2, the scale of the model is constantly increasing.

One of the main insights of the connection mechanism is that when many neurons in an animal work together, they become smart.

The small size of the neural network does not solve the problem of high difficulty is certain, than a leech of neurons less than the neural network can not solve complex AI is not surprising.

The scale includes the size of the number of neural nodes and the size of each neural node connection.

As of 2016, a rough rule of thumb is that the supervised deep learning algorithm typically achieves acceptable performance in each class of approximately 5000 labeled samples, and when at least 10 million of the data sets for the sample are used for training, the human performance will be met or exceeded.

Some things:

Rectifier Linear Unit (rectified linear unit)

Imagenet Large-scale visual identity Challenge (ILSVRC)

The story of the Ferret Brain

Deep Learning (Bengio) First chapter reading notes