The measurement and function of information in the 6th chapter of the Beauty of mathematics

1 Information Entropy

The amount of information in a message is directly related to its uncertainty. The amount of information equals the amount of uncertainty.

How to quantify the measurement of information. The information entropy (entropy) is represented by the symbol H, the unit is bit.

The greater the uncertainty of variables, the greater the entropy.

If a book repeats a lot of content, its information is small, redundancy is big.

The redundancy of different languages varies greatly, and the redundancy of Chinese is relatively small in all languages.
2 Function of the information

Information and the elimination of uncertainty are interrelated.

Information is the only way to eliminate system uncertainty (a system is like a black box, introducing information to understand the inner structure of the black box system without obtaining any information)

The nature of Web search is also the process of using information to eliminate uncertainty.

Reasonable use of information, rather than play with the formula and machine learning algorithm, is the key to do a good job search.

Conditional entropy:

, in other words, the uncertainty about X drops when you have more information about Y. In the statistical language model, if Y is regarded as the previous word, it is proved mathematically that the uncertainty of the two-element model is less than the one-element model.

When the above equals sign is set up. The equal sign is set up, the information is added, and the uncertainty is not reduced. If the information we lack has nothing to do with what is being studied, the equal sign is established.

The role of information is to eliminate uncertainty, and the vast problem of natural language processing is to find relevant information.
3 Mutual Information

Shannon proposes a "mutual information" concept as a quantitative measure of two random event "dependencies".

The quantitative measure of the so-called two event correlations is the amount of information provided to eliminate another x uncertainty under the premise of knowing one Y.
4 Relative Entropy

Relative entropy is also used to measure relevance, but it is used to measure the relevance of two of functions that have positive values, unlike the variable's mutual information.

Two identical functions, their relative entropy equals zero.

The greater the relative entropy, the greater the difference of the two functions, whereas the smaller the relative entropy, the smaller the difference of the two functions.

For probability distribution or probability density function, if the value is greater than 0, the relative entropy can measure the difference of two random distributions.