Information Theory Foundation

1. Entropy and information

There is a set of discrete symbol sets {V1,v2,..., VMS}, each with a corresponding probability pi, to measure the randomness (uncertainty or unpredictability) of a particular sequence consisting of this set of symbols, to define the entropy of the discrete distribution: the logarithm of 2, the "bit" of the unit of entropy, and when continuous, the base is e, the unit is " Knight. " If the answer is a question, the probability of each possible answer is 0.5, then the entropy at this point is 1. The formula of entropy can also be written as: H=ε[log (1/p)],p is a random variable, the value p1,p2, ... Pm,log21/p is sometimes called the surprise rate.

The entropy value does not depend on the symbol itself, but on the probabilities of the symbols. For a given m-symbol, when these symbols appear in the same probability, the entropy is maximum (h=log2m). That is, when each symbol appears in the same probability, the most uncertainty about what happens to the next symbol. The probability of only one sign is 1, the other is zero, and the entropy is minimum, which is 0.

The entropy of the continuous condition is defined as:

The mathematical expectation form is H=Ε[LN (1/p)]. In all continuous density functions, if the mean μ and variance σ2 all take known fixed values, the Gaussian distribution causes the entropy to reach the maximum h=0.5+log2 (2π multiplied by σ). If the variance approaches 0, the Gaussian function will approach the Dirac function, at which point the entropy is the least negative infinity. For the distribution of the Dirac function, it is almost certain that each occurrence of the x value is a.

For random variables x and any function f (•), there is H (f (x)) ≤h (x), that is, any processing of the original signal does not increase entropy (information). If f (x) is a function of the constant value, then the entropy is 0. Another important property of discrete distribution entropy: arbitrarily changing event markers does not affect the entropy of this set of symbols, because entropy is only related to the probability of a sign occurrence, not to the symbol itself. However, it is not necessarily true for continuous random variables.

2. Relative entropy

Assume that for the same discrete variable x, there are 2 possible forms of discrete probability distributions p (x) and Q (x). To measure the distance between the two distributions, define the relative entropy (Kullback-leiber distance):

The relative entropy is defined in the continuous case as:

Relative entropy is not a true measure, because DKL is not symmetrical when P and Q are exchanged with each other.

3. Mutual information

Suppose there are two different variable probability distributions, p (x) and Q (x). Mutual information refers to the amount of uncertainty about the other variable after obtaining information about one variable.

Information Theory Foundation