Take a look at the data smoothing algorithm

In natural language processing, the probability estimation of the occurrence of a word sequence (sentence) is often calculated. We know that when the algorithm is trained, the corpus cannot contain all the sequences that may appear.

Therefore, in order to prevent the probability of the new sequence not appearing in the training sample is zero, we have invented a lot of algorithms to improve the probability of estimating the new sequence, that is, the data smoothing algorithm.

Laplace Law

The simplest algorithm is the Laplace rule, the idea is very simple, statistical test data set of the elements in the training data set the number of times, the initial value of the counter is not set to 0, but set to 1. This way, even if the element does not appear in the training set, its occurrence count is at least 1. Therefore, the probability estimate of its occurrence will not be zero.

Assume test set  V   An element in the training set  t   appears   r   Times, after Laplace the number of statistics adjusted:

r? =R+1

Of course, this is purely to avoid the 0 probability, and does not solve the problem of effectively predicting the instances that have not been seen. So the Laplace rule is just a very rudimentary technique, a little too small for pediatrics.

Good-turing Estimate

An obvious problem with the

Laplace method is that  < Span id= "mathjax-span-19" class= "math" > ∑r≠∑ r   The Good-turning method considers this to be a major flaw that needs to be improved. In fact, I think this is really not important, as long as we can reasonably estimate the probability of new instances not seen, the total number of statistics change what?  
 
Good-turing The revised formula is really ingenious, and it takes a correction factor behind the Laplace rule to ensure that the total number of times remains the same. This to show off a dazzle or no problem:  
 

　　　　　　　　　　　　　　　　　　　　　　

whichRepresents a test setV, there are altogether elements in training set  t   appeared in   times.

Although I find this method useless, it does ensure that the total number of elements in the test set that are present in the training set does not change. That

　　　　　　　　　　　　　　　　　　　　　　

Obviously n1 =n2 . Perhaps this method can not solve the natural language processing problem, but also  < Span id= "mathjax-span-206" class= "math" >nr=0 The   formula also fails, but the idea should still be valuable, perhaps solving other problems can be used.

Absolute discounts and linear discounts

It is estimated that the author of the invention is stimulated by good-turing, that the method is "Maxi", and that a large number of statistics are given in part to give a smaller number of statistics, reducing the gap between rich and poor. But this method uses a very skillful formula to conceal its essence.

Instead of timid "maxi", there is a naked way, so there is an absolute discount and a linear discount method.

The problem is that "maxi" is not our goal, and what we need is the ability to judge the probability of a sentence that has never been seen in a corpus. How much do you need to be "robbed" to get the right judgment? How much is "Ji"? The question of absolute discount and linear discount is not answered. Therefore, regardless of the good-turing method, or the two discount methods, essentially did not jump out of the Laplace rule of thought.

Witten-bell algorithm

Witten-bell algorithm finally jumped out from the Laplace algorithm, with a qualitative breakthrough. The basic idea of this method is that if an instance in the test does not appear in the training corpus, then he is a new thing, that is, he appears for the first time. Then you can use the probability of seeing a new instance (that is, the first occurrence) in the corpus instead of the probability of an instance not appearing.

Suppose the number of words in the corpus appears in the following table:

　　　　　　　　　　　　

So, we can use approximate representation of the probability of seeing new words in the corpus.

I can't say how much of this is true, but it provides at least a plausible reason than the "maxi" approach.

Take a look at the data smoothing algorithm