The principle of maximum entropy of first knowledge

I. Overview

In daily life or scientific experiments, a lot of things happen to have a certain randomness, that is, the final result is random, we do not know whether these results obey what the law, we have only a few experimental samples, in this case, How do we make a reasonable inference to the result based on what we have now? The maximum entropy method is one way to solve this problem. The maximum entropy principle is e.t.jaynes in 1957, and its basic idea is as follows: In the knowledge of the unknown part, we should select the probability distribution which conforms to the knowledge and the maximum entropy value. Entropy is defined as the uncertainty of a random variable, and the maximum entropy is the most uncertainty. From this point of view, the maximum entropy principle is essentially to satisfy all known knowledge premise, for the unknown distribution should be the most uncertain or the most random distribution, because only in this way, the final distribution can represent a fairest choice, any subjective choice is essentially for the model added constraints, These conditions are not available according to the information currently known. At present, the maximum entropy principle has been applied greatly in natural language processing. Because the maximum entropy model can combine the information of different sources into a framework, it is very advantageous to solve some complex problems. Therefore, since Della Pietra in 1992 for the first time with the maximum entropy principle of language processing, more and more scholars are using the maximum entropy principle to deal with text classification, phrase recognition and part-of-speech tagging problems, and achieved a relatively satisfactory effect.

The basic idea of the maximum entropy model is that when we build a predictive model for a random event, the model should satisfy all the known feature information obtained from the training sample, and do not make any subjective assumptions about other unknown information, in which case the conditional probability distribution is the most uniform and the risk of prediction is minimal. At this time, the prediction model satisfies the maximum entropy value under the constraint condition, so this model is called the "Maximum model". Life said not to put all the eggs in the same basket, is the maximum entropy of the idea of a very simple statement, when we encounter uncertainty events, we should try to preserve the various possibilities, this principle is in line with the simple desirability.

second, conditional maximum entropy model

Since many of the maximum entropy models are applied in the field of language processing, we use the processing language example to illustrate the maximum entropy model. In fact, the ultimate task of the maximum entropy model is probability estimation, that is, estimating the target class Y, the probability of the occurrence in instance x or under the condition X, that is, the value of P (y|x). The set of x is represented as y in the set of Y. In addition, we use p to represent the set of all conditional probability distributions, and P (y|x) is just one element in P. In the specific language processing, the generation of a Y is to be affected by the X, that is, assuming a random input of a "eat" word, then its context is definitely distributed in the "eat" word around the characters, that is, Y is definitely a set {verb, noun} one. Then the maximum entropy method is based on a number of specific statistical methods, based on the appearance of x, y characteristics, reliable estimation of P (y|x) this probabilistic model. In fact, the maximum entropy method is not limited to the conditional maximum entropy when it is used, initially it is the value of P (x), then it gradually evolves into an estimated p (y|x), and the difference is that P (x, y) or P (x.) is usually the joint maximum entropy model. In the language processing including the later use of the content-based processing is the use of conditional maximum entropy model, hereby explained.

Iii. Introduction to the model

Eat "Around the word is instance X, part of speech is Y." The x here is actually a limitation of the contextual information to the words surrounding the "eating". and y in the actual situation of the possibility is very large, it may be "eating and drinking", "Eat apples", "delicious" and so on. Thus, we can see that the instance space is very large. From the training sample set, we can get like 1 1 2 2 n n (x, y), (x, y),..., (x, y) z major empirical probability distributions, with the resulting logarithm divided by the total logarithm of the probability, that is, the maximum entropy model first through these training samples to train the model, and then take into account the probability distribution of P The application system in the model framework is to estimate the probability distribution P (y|x ') of instance X ' by using a well-trained model, in which 1<=y<=y. Finally, the final recommendation results are formed by the large-to-small arrangement of probabilities.

Iv. Collaborative Filtering

In a general e-commerce system, users tend to score only a few items, and the entire user set is associated with the project collection as a very sparse scoring matrix. In the method of calculating similarity, the relevant scoring information between user or project is very insufficient, reflecting two users or the total score of the project is very limited, so the similarity calculation result is very unreliable, and the actual calculated similarity value is not too high, losing the meaning of the linear correlation itself. Therefore, the recommendation method of selecting nearest neighbor based on similarity is severely negatively affected by the data sparse situation.

v. References

[1] Research on solving the sparsity problem of collaborative filtering based on maximum entropy mode mo

[2] Maximum entropy recommendation algorithm for merging user attributes and Interests

The principle of maximum entropy of first knowledge