Use hyperspace analogue to language (HAL) for short text Extension

Source: Internet
Author: User

Hal is what I do not explain, specific can refer to: http://www.zhan5zhan.com/post/6.html

1. What is short text

Forums, blogs, microblogs, chat records, and Q & A can all be considered short articles. Although blogs and forums have many long texts, they are a minority.

2. Difficulties in short text

1) Non-standard and colloquial. For example, various abbreviations and typos.

2) Lack of context. In professional forums, it is hard to understand all the special terms. For example, the "Monk" in Diablo 3 refers to a role, and "Mami Love" is a baby drug.

3. Solution: Supplement context and background knowledge

Abbreviations, mistakes, special characters, and isolated words must be placed in a complete context to be understood. How to construct a bag of words to supplement the short text is the key to the problem.

4. Hal Method

Hal finds out that words in the word matrix complement each other with a large number of co-occurrence times between words. The above link provides an example.

5. Phal Method

The supplement to Hal determines whether the co-occurrence word can constitute an "interpretation" Relationship to the original word. There are two factors: closer and closer to the original word, the closer the relationship is, the more times the two appear together, the stronger the relationship.

Therefore, Phal increases the co-occurrence probability and co-occurrence distance compared with Hal.

S (WI | W) = P (WI | W)/L (WI | W)

This is the co-occurrence formula. The larger the probability and the shorter the distance indicate the closer the relationship between the two words.

6. The following are some interesting examples.

  • Xiangyue 12 hotel | 0.149390 | 0.30 | 2.0 Beijing | 0.041757 | 0.13 | 3.1 Beijing | 0.027999 | 0.09 | 3.3 | 0.014967 | 0.05 | 3.7 | 0.014967 | 0.05 | 3.7 | 0.010671 | floor | | 0.02 | 2.0 reservation | 0.010540 | 0.03 | 3.2 Price | 0.006499 | 0.03 | 4.2 accommodation | 0.006499 | 0.03 | 4.2 price | 0.006499 | 0.03 | 4.2 |
    Hotel | 0.003430 | 0.02 | 5.3 | 0.002217 | 0.01 | 5.5
  • Blood ridge sniper 7 plot | 0.071429 | 0.14 | 2.0 | 0.047619 | 0.14 | 3.0 | 0.047619 | 0.14 | 3.0 | 0.047619 | 0.14 | 3.0 | 0.035714 | 0.14 | 4.0 | starring | 0.028571 | 0.14 | 5.0 Shi tailong | 0.023810 | 0.14 | 6. 0
  • Da Lian 2 Doctor | 0.250000 | 0.50 | 2.0 Suzhou | 0.250000 | 0.50 | 2.0 |
  • Journal of forest diseases and insects of China | 0.250000 | 0.50 | 2.0 | 0.166667 | 0.50 | 3.0
  • Nameen 1 home textile | 0.500000 | 1.00 | 2.0
  • Haidian road 3 Chen Shufen | 0.111111 | 0.33 | 3.0 | 0.083333 Chinese Medicine Clinic | 0.33 | 4.0 | 0.066667 | 0.33 migrated | 5.0 |

For example, if you are not familiar with NAMAs, you certainly don't know what it is, but the "home textiles" mentioned later shows that it is a home textile brand. For example, if you don't know, you may think it is a game, a movie, a TV series, or a novel. In fact, he is a movie starring Shi tailong, it is often downloaded by thunder.

7. What is the purpose? OK, this is very useful. It can be used to expand short articles and then for classification, clustering, recommendation system, similarity calculation, semantic understanding, sentiment analysis, public opinion and public opinion, anti-spam .....

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.