Model Pruning for input methods (entropy-based pruning)

Prunning, pruning, as the name suggests, is to remove those that are not important.
Theoretically, the effect achieved by pruning is the maximum similarity between Q after pruning and P before pruning. There are two algorithms entroy-based and rank-based.
For the model, relative entropy is used to portray D (q | p) to minimize the Entropy Difference between the two models, that is, entropy-based. If rank (p | q) is used to describe, the rank difference of the entire model is minimized (mainly for the left word of the same bigram pair), that is, rank-based.
For smoothing, we use Katz smoothing. Entropy-based computing routines: 1. calculate the deleted model parameters for each piece of data. For Katz, re-calculate Alpha. Based on the new/old Alpha, check the change of the model entropy. For specific values, refer to the formula in this paper.


Parameter calculation:


The real difficulty lies in how to compute such information in a distributed manner. Each time a piece of data is deleted, it is obviously not feasible. It cannot be viewed through the formula after it is deleted. The focus here is how to update Alpha and how to iterate after Alpha is determined.
Rank-based algorithms were also tried at the time. They felt similar to entropy-based, and neither of them achieved the expected results, so they were not studied in depth.
The above algorithm is a theoretical algorithm used to depict the distance between two models. It is not suitable for the input method. Generally, the size of the original model of the input method may be 3 billion, and the size of a local model is about. Basically, 90% of the data needs to be cut off, and the final model is quite different from the original model. Entropy-based pruning is generally expected to be slightly different from the original model after being cut. Based on experience, it is good to cut about 20% of the data.
So in the true sense, pruning should be done based on the characteristics of the input method.