Python unsupervised word extraction seo how to quickly correct participle

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

How fast correct participle, for SEO, is to extract tags aggregation, information related good helper.

At present, many word segmentation tools are based on one-yuan word segmentation, need thesaurus to assist.

Through the study of the first chapter of Google Blackboard, how to use the statistical model for participle.

This method considers 3 dimensions

Cohesion: Two consecutive occurrences of the probability is not the degree of independence. For example, "on" The probability of appearing is 1x10^-5, "bed" the probability is 1x10^-10, if the two words of low cohesion, then "go to bed" the probability should and 1x10^-15 close, but in fact "bed" the probability of appearing in the 1x10^-11 of the second side, is much higher than the product of independent probabilities. So we can think that "sex" is a word.

Neighbours word set entropy: The amount of information on the left side of a word, such as "slap", can only be used for "slapping", "slap", "clappy", on the contrary, "past" the word, the front can be used to "walk past", "Run Past", "Climb Over", "hit the Past", "Mix", "Sleep Over", "Dead Past", " Fly past "and so on, information entropy is very high.

Right adjacent word set entropy: The amount of information on the right side of the word, ibid.

Here is a demo using Python (go from: http://www.webinfoextract.com/forum.php?mod=viewthread&tid=20)? 1 2 3 4 5 6 7 8 9 #!/bin/sh python/splitstr