Calculate document Similarity)

From: http://blog.chinaunix.net/uid-26548237-id-3541783.html

1. vector space model
Vector space model, as a vector identifier, is an algebraic model used to represent text files. It is used for information filtering, information retrieval, indexing, and related rules.
Documents and problems are expressed by vectors.

Each dimension is equivalent to an independent phrase. If this term appears in the document, its value in the vector is not zero. There are already many different methods to calculate these values. These values are called (phrases) weights. One of the well-known algorithms is the tf_idf weight. We define phrases based on applications. A Typical phrase is a single word, keyword, or long phrase. If a word is selected as a phrase, the dimension of the vector is the number of words that appear in the vocabulary. Vector operations can compare various documents through queries.

Based on the assumption of the document similarity theory, compare each document vector with the original query vector (the two vectors are of the same type . In fact, it is easier to calculate the cosine ratio of the angle between vectors to directly calculate the angle.

D 2 * Q is the document vector (I .e., D 2 ) And query vector (Q in ). The denominator is the modulus of two vectors. The model of a vector is calculated using the following formula:

Because all vectors in this model are strictly non-negative, if the cosine is zero, it indicates that the query vector and the document vector are orthogonal, that is, they do not match (in other words, the token is not found in the document. ), That is, the similarity between the two documents is 0%.

The following is an example of a TF-IDF weight.


Advantages:
Compared with the standard Boolean model, the vector space model has the following advantages:
1. Simple Model Based on Linear Algebra;
2. The weight of a phrase is not binary;
3. Continuous similarity between documents and indexes can be calculated;
4. allow files to be sorted based on possible relevance;
5. Local matching is allowed;

Limitations:
1. It is not applicable to long files because its similarity value is not ideal;
2. The search phrase must exactly match the phrase in the file;
3. files with the same context but different phrases cannot be associated due to poor semantic sensitivity;
4. the sequence in which phrases appear in the document cannot be expressed in the middle of the vector;
5. It is assumed that phrases are statistically independent;
6. the weights are obtained intuitively but not formally enough;

2. Use of Vector Space Models
The following describes how to use the vector space model to calculate the file similarity. The cosine value cosine described above is implemented as an example.
The weights in the implementation directly use the frequency of word appearance. In addition, the English similarity is compared here.

2. # Include<Iostream>
   
4. # Include<Map>
6. # Include<Sys/Stat.H>
8. # Include<Cmath>
10. Using namespace std;
14. # DefineError -1
16. # Define OK 0
18. # Define debug
22. //Used to remove irrelevant words in text
24. //ConstChar delim[] = ".,:;'/\"+I-_(){}[]<>*&^%$#@!?~/|\\=1234567890\T\N";
    
26. Const char delim [] =".,:;'`/\"+-_ () {} [] <> * & ^ % $ #@!?~ /| \\= 1234567890 \ t \ n";
30. Char*Strtolower(Char*Word)
32. {
34. Char*S;
38. For(S=Word; *S;S++)
40. {
42. *S=Tolower(*S);
44. }
46. Return word;
48. }
52. IntReadfile(Char*Text_name,Map<String, Int> &Word_count)
54. {
56. Char*Str;
58. Char*Word;
60. Char*File;
62. Struct stat sb;
66. File*Fp=Fopen(Text_name, "R");
70. If(Fp== Null)
72. {
74. ReturnError;
76. }
80. If(Stat(Text_name, &Sb))
82. {
84. ReturnError;
86. }
90. File= (Char*)Malloc(Sb.St_size);
92. If(File== Null)
94. {
96. Fclose(Fp);
98. ReturnError;
100. }
102. Fread(File,Sizeof(Char),Sb.St_size,Fp);
106. Word=Strtok(File,Delim);
110. While(Word!= Null)
112. {
114. //Delete the length of Word<=1
116. If(Strlen(Word) <=1)
118. {
120. Word=Strtok(Null,Delim);
122. Continue;
124. }
128. Str=Strtolower(Strdup(Word));
130. StringTMP=Str;
132. Word_count[TMP]++;
136. Word=Strtok(Null,Delim);
138. }
140. }
144. IntMain(IntArgc,Char**Argv)
146. {
148. Char*Text_name_one= "./Big.txt";
150. //Char*Text_name_one= "./1.txt";
152. Char*Text_name_two= "./Big.txt";
154. //Char*Text_name_two= "./2.txt";
158. Map<String, Int>Word_count_one;
160. Map<String, Int>Word_count_two;
164. Double multi_one=0.0;
166. Double multi_two=0.0;
168. Double multi_third=0.0;
172. If(Readfile(Text_name_one,Word_count_one) == Error)
174. {
176. Cout<< "Readfile () error ." <<Endl;
178. ReturnError;
180. }
184. # Ifdef debug
186. Map<String, Int>::Iterator map_first=Word_count_one.Begin();
188. For( ;Map_first!=Word_count_one.End();Map_first++)
190. {
192. Cout<<Map_first->First<< "" <<Map_first->Second <<Endl;
194. }
196. # Endif
200. If(Readfile(Text_name_two,Word_count_two) == Error)
202. {
204. Cout<< "Readfile () error ." <<Endl;
206. ReturnError;
208. }
212. # Ifdef debug
214. Map<String, Int>::Iterator map_second=Word_count_two.Begin();
216. For( ;Map_second!=Word_count_two.End();Map_second++)
218. {
220. Cout<<Map_second->First<< "" <<Map_second->Second <<Endl;
222. }
224. # Endif
228. Map<String, Int>::Iterator map_one=Word_count_one.Begin();
230. Map<String, Int>::Iterator map_tmp;
232. For( ;Map_one!=Word_count_one.End();Map_one++)
234. {
236. Map_tmp=Word_count_two.Find(Map_one->First);
238. If(Map_tmp==Word_count_two.End())
240. {
242. Multi_two+=Map_one->Second *Map_one->Second;
244. Continue;
246. }
248. Multi_one+=Map_one->Second *Map_tmp->Second;
250. Multi_two+=Map_one->Second *Map_one->Second;
252. Multi_third+=Map_tmp->Second *Map_tmp->Second;
254. Word_count_two.Erase(Map_one->First);//Delete
256. }
260. //Check whether there are still elements in 2
262. For(Map_tmp=Word_count_two.Begin();Map_tmp!=Word_count_two.End();Map_tmp++)
264. {
266. Multi_third+=Map_tmp->Second *Map_tmp->Second;
268. }
270. Multi_two=SQRT(Multi_two);
272. Multi_third=SQRT(Multi_third);
276. Double result=Multi_one/ (Multi_two*Multi_third);
278. Cout<< "Similarity :" <<Result*100<< "%" <<Endl;
280. Return 0;
282. }

Perform the following tests.
First, to detect two identical English text, text links for http://norvig.com/big.txt

The word statistics in the text show that the similarity between the two identical texts is 100%.

2. Text 1 content:... This is one! Content of text 2: () ...... This is Two

The running result is the same as the actual manual calculation result. The similarity between the two texts is 66.6667%.

The above is just a simple calculation of the similarity between two English texts. It is only calculated at the entry level and does not involve semantics. Therefore, it is relatively simple.
I am very interested in this aspect and will continue to learn other related content.

Introduction to theoretical knowledge: http://zh.wikipedia.org/wiki/%E5%90%91%E9%87%8F%E7%A9%BA%E9%96%93%E6%A8%A1%E5%9E%8B