How to calculate the similarity of two documents (ii)

Note: Fully tested and with full code:

#-*-coding:cp936-*-from Gensim import corpora, models, Similaritiesimport logginglogging.basicconfig (format= '% (ascti Me) S:% (levelname) s:% (message) s ', level=logging.info) documents = ["Shipment of gold damaged in a fire", "Delivery of Si Lver arrived in a silver truck ', ' Shipment of gold arrived in a truck '] texts = [[Word for Word in document.lower (). Split () ] for document in Documents]print "texts" print Texts dictionary = corpora. Dictionary (texts) print "Dictionary" Print Dictionary print "dictionary.token2id" print dictionary.token2id corpus = [ Dictionary.doc2bow (text) for text in Texts]print "Corpus" Print CORPUSTFIDF = models. Tfidfmodel (corpus) CORPUS_TFIDF = tfidf[corpus]print "Doc" for Doc in Corpus_tfidf:print Doc ###### #输出代表文档中每个单词对于该文档的代表性 , that is, if the word appears only in the document, the more powerful the word represents the document, and the more likely it is to be representative if it is repeated multiple times in that article. # doc# [(0, 0.67211468809878627), (1, -0.54880682119355984)]### #这时doc1, and the value of topic0 is larger, so closer to the Topic0 theme # [(0, 0.44124825208697827), (1, 0.83594920480338997)]#### #这时doc2, and Topic1 's value moreLarge, so closer to the Topic1 theme # [(0, 0.80401378963792725)]############################## #这是doc3, only the result of comparison with TOPIC0 (since the other result is 0), So closer to the Topic0 theme print "Tfidf.dfs" Print Tfidf.dfs # # # # # #同idfs, also a dictionary, each key's value represents how many documents the word has ever appeared in print "Tfidf.idfs" Print Tfidf.idfs # # # # # # # # # # # # # # # # # # # #共是个数据的字典, each data value represents the size of the word for the document, i.e. if the word appears in all articles, the description has no representation, the value is 0, and if the word appears in less articles, Represents the word for this document has a stronger representative LSI = models. Lsimodel (CORPUS_TFIDF, Id2word=dictionary, num_topics=2) lsi.print_topics (2) Corpus_lsi = Lsi[corpus_tfidf]print " Corpus_lsi "for doc in corpus_lsi:###### #输出多个文档的文档内容和各主题的匹配性大小关系 print Doc ###### #输出为该文档的内容和各个主题的匹配性大小关系 # # # # # #tes t# Corpus_only_lsi = lsi[corpus]# print "Corpus_only_lsi" # for doc in corpus_only_lsi:# print doc# LDA = models. Ldamodel (CORPUS_TFIDF, Id2word=dictionary, num_topics=2) # print "Lda.print_topic (i)" # for I in range (0, 2): # Print Lda . Print_topic (i) # # Corpus_lda = lda[corpus_tfidf]# print "Corpus_lda" # for doc in corpus_lda:# print doc index = SI Milarities. Matrixsimilarity (lsi[CORPUS]) query = "Gold Silver Truck" Query_bow = Dictionary.doc2bow (Query.lower (). Split ()) print "Query_bow" Print Query_ Bow ##### #把该查询文档 (Word set) changed to (Word bag model) that is: Dictionary format, key is a word, and value is the number of occurrences of the word in the document. Query_lsi = lsi[query_bow]print "Query_lsi" Print Query_lsi # # # # # # # # # # # # # #把该查询文档和两个类别相比较 to its similarity sims = index[query_lsi]print "List (Enumerate (Sims)) "Print List (enumerate (Sims)) # # # # #输出为该查询文档和corpus中三个输入文档的相似性sort_sims = sorted (Enumerate (Sims), KEY=LAMBDA item:-item[1]) print "Sort_sims" Print sort_sims ######## #对sims的排序

We have introduced some background knowledge and Gensim, I believe many students have already tried. This section will start with the most basic installation of Gensim, and then give a very simple example of how to use Gensim, and the next section on its application to the curriculum map.

Second, the installation and use of Gensim

1. Installation
Gensim relies on NumPy and scipy, the two big Python scientific computing toolkits, a simple installation method is pip install, but domestic because of the network often fail. So I downloaded the Gensim source code package installed. Gensim's official Installation page provides a detailed list of compatible Python and NumPy, the version number of the SciPy, and the installation steps, which interested students can refer to directly. Below I just explain the installation under Ubuntu and Mac OS:

1) My VPS is 64-bit Ubuntu 12.04, so install NumPy and scipy relatively simple "sudo apt-get install python-numpy python-scipy", then unzip Gensim installation package, direct "sudo Python setup.py install "can;

2) My Ben is MacBook Pro, installed on Mac OS NumPy and SCIPY source package waste a bit of trouble, especially the latter, has been suggesting that Fortran related things no, Google a bit, Found a lot of people on the Mac installed scipy when all encountered this problem, finally through the homebrew installed Gfortran to fix: "Brew install Gfortran", and then still "sudo python setpy.py Install" NumPy and scipy can be;

2. Use
Gensim Official tutorial is very detailed, English OK students can directly refer to. I'll follow my own example to illustrate how to use Gensim, an example that differs from the official Gensim example, which can be added as a supplement. The previous section mentions a document: latent Semantic indexing (LSI) a Fast track Tutorial, the source of this example is the 3-sentence doc mentioned in this document. Let's start by opening python at the command line and doing some preparatory work:

>>> from Gensim import corpora, models, similarities
>>> Import Logging
>>> logging.basicconfig (format= '% (asctime) s:% (levelname) s:% (message) s ', Level=logging.info)

The example in the above document is then entered as a document and is represented in Python as a list of documents:

>>> documents = ["Shipment of gold damaged in a fire",
... "Delivery of silver arrived in a silver truck",
... "Shipment of Gold arrived in a truck"]

Under normal circumstances, the English text needs to do some preprocessing work, such as to stop the word, tokenize,stemming the text and filter out the low-frequency words, but in order to illustrate the problem, but also to this "LSI Fast track Tutorial" Consistent, The following preprocessing is only to lowercase the English words:

>>> texts = [[Word for Word in document.lower (). "Split ()] for document in documents]
>>> Print Texts
[' Shipment ', ' of ', ' Gold ', ' damaged ', ' in ', ' a ', ' fire '], [' Delivery ', ' the ', ' Silver ', ' arrived ', ' in ', ' a ', ' silver ', ' t Ruck '], [' Shipment ', ' of ', ' Gold ', ' arrived ', ' in ', ' a ', ' truck ']

We can use these documents to extract a "word bag (bag-of-words)" That maps the token of a document to an ID:

>>> dictionary = corpora. Dictionary (texts)
>>> Print Dictionary
Dictionary (Unique tokens)
>>> Print Dictionary.token2id
{' A ': 0, ' damaged ': 1, ' Gold ': 3, ' Fire ': 2, ' of ': 5, ' delivery ': 8, ' arrived ': 7, ' Shipment ': 6, ' in ': 4, ' Truck ': ten, ' s Ilver ': 9}

You can then convert the document represented by the string to the document vector represented by the ID:

>>> corpus = [Dictionary.doc2bow (text) for text in texts]
>>> Print Corpus
[[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), [(0, 1), (4, 1), (5, 1), (7, 1), (8, 1), (9, 2), (10, 1)], [(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (10, 1)]

For example (9,2) this element represents 2 occurrences of the word "silver" with an ID of 9 in the second document.

With this information, we can calculate a TF-IDF "model" based on these "training documents":

>>> TFIDF = models. Tfidfmodel (Corpus)
2013-05-27 18:58:15,831:info:collecting Document Frequencies
2013-05-27 18:58:15,881:info:progress:processing Document #0
2013-05-27 18:58:15,881:info:calculating IDF weights for 3 documents and one features (matrix Non-zeros)

Based on this TF-IDF model, we can represent a document vector with the word frequency representation as a document vector represented by a TF-IDF value:

>>> CORPUS_TFIDF = Tfidf[corpus]
>>> for Doc in CORPUS_TFIDF:
.. print doc
...
[(1, 0.6633689723434505), (2, 0.6633689723434505), (3, 0.2448297500958463), (6, 0.2448297500958463)]
[(7, 0.16073253746956623), (8, 0.4355066251613605), (9, 0.871013250322721), (10, 0.16073253746956623)]
[(3, 0.5), (6, 0.5), (7, 0.5), (10, 0.5)]

Find some tokens that seem to be missing, let's print out the information in the TFIDF model:

>>> Print Tfidf.dfs
{0:3, 1:1, 2:1, 3:2, 4:3, 5:3, 6:2, 7:2, 8:1, 9:1, 10:2}
>>> Print Tfidf.idfs
{0:0.0, 1:1.58496250072116, 2:1.58496250072116, 3:0.5849625007211562, 4:0.0, 5:0.0, 6:0.5849625007211562, 7:0. 5849625007211562, 8:1.58496250072116, 9:1.58496250072116, 10:0.5849625007211562}

We found that the number of documents with IDs 0, 4, 5, 3 words (DF) was 3, and the total number of documents was 3, so IDF was calculated as 0, and it appeared that Gensim did not add 1 to the numerator, making a smooth one. But we also found that the 3 words of a, in, of the preposition, can be used as a stop word during preprocessing, which also shows the effectiveness of TF-IDF from another aspect.

With the TF-IDF value representation of the document vector, we can train a LSI model, similar to the example in latent Semantic indexing (LSI) A Fast Track tutorial, we set the topic number to 2:

>>> LSI = models. Lsimodel (CORPUS_TFIDF, Id2word=dictionary, num_topics=2)
>>> Lsi.print_topics (2)
2013-05-27 19:15:26,467:info:topic #0 (1.137): 0.438* "Gold" + 0.438* "shipment" + 0.366* "Truck" + 0.366* "arrived" + 0.34 5* "Damaged" + 0.345* "fire" + 0.297* "Silver" + 0.149* "delivery" + 0.000* "in" + 0.000* "a"
2013-05-27 19:15:26,468:info:topic #1 (1.000): 0.728* "Silver" + 0.364* "delivery" + -0.364* "fire" + -0.364* "damaged" + 0 .134* "Truck" + 0.134* "arrived" + -0.134* "shipment" + -0.134* "Gold" + -0.000* "a" + -0.000* "in"

The physical meaning of LSI is not very well explained, but the core meaning is to decompose the Matrix SVD, which is composed of the training document vectors, and make a approximate SVD decomposition of rank 2, which can refer to the English tutorail. With this LSI model, we can map the document to a two-dimensional topic space:

>>> Corpus_lsi = LSI[CORPUS_TFIDF]
>>> for Doc in Corpus_lsi:
.. print doc
...
[(0, 0.67211468809878649), (1,-0.54880682119355917)]
[(0, 0.44124825208697727), (1, 0.83594920480339041)]
[(0, 0.80401378963792647)]

As you can see, documents 1,3 and TOPIC1 are more relevant, and documents 2 and TOPIC2 are more relevant;

We can also run a LDA model:

>>> LDA = models. Ldamodel (COPURS_TFIDF, Id2word=dictionary, num_topics=2)
>>> Lda.print_topics (2)
2013-05-27 19:44:40,026:info:topic #0:0.119*silver + 0.107*shipment + 0.104*truck + 0.103*gold + 0.102*fire + 0.101*a rrived + 0.097*damaged + 0.085*delivery + 0.061*of + 0.061*in
2013-05-27 19:44:40,026:info:topic #1:0.110*gold + 0.109*silver + 0.105*shipment + 0.105*damaged + 0.101*arrived + 0. 101*fire + 0.098*truck + 0.090*delivery + 0.061*of + 0.061*in

Each topic in the LDA model has a probabilistic meaning, with a sum of 1, the larger the weight, the more explicit the physical meaning, but in turn the LDA model of the 2 subjects of the three document training is too average and unconvincing.

Well, we go back to the LSI model, with the LSI model, how can we calculate the direct Acacia degree of a document, or change the angle, given a query, how to find the most relevant document? First, of course, the index was built:

>>> index = similarities. Matrixsimilarity (Lsi[corpus])
2013-05-27 19:50:30,282:info:scanning Corpus to determine the number of features
2013-05-27 19:50:30,282:info:creating matrix for 3 documents and 2 features

Or the query in this English tutorial, for example: Gold Silver truck. To quantify it first:

>>> query = "Gold Silver Truck"
>>> Query_bow = Dictionary.doc2bow (Query.lower (). Split ())
>>> Print Query_bow
[(3, 1), (9, 1), (10, 1)]

Then use the previously trained LSI model to map it to the two-dimensional topic space:

>>> Query_lsi = Lsi[query_bow]
>>> Print Query_lsi
[(0, 1.1012835748628467), (1, 0.72812283398049593)]

The last is to calculate the cosine similarity of doc in the index:

>>> Sims = Index[query_lsi]
>>> Print List (enumerate (Sims))
[(0, 0.40757114), (1, 0.93163693), (2, 0.83416492)]

Of course, we can also sort by similarity:

>>> sort_sims = sorted (Enumerate (Sims), KEY=LAMBDA item:-item[1])
>>> Print Sort_sims
[(1, 0.93163693), (2, 0.83416492), (0, 0.40757114)]

Note: This is calculated by LSI and is similar to the results of LDA calculations:

<span style= "color: #CC0000;" >corpus_lsi = lsi[corpus_tfidf]>>> for doc in Corpus_lsi: ... print doc above this program and context is not related, remove the feeling read the full text more fluent. By using LDA to find similar LDA = models. Ldamodel (CORPUS_TFIDF, Id2word=dictionary, num_topics=2) lda.print_topics (2) for Doc in Lda[corpus]:p rint Docquery_lda = Lda[query_bow]index = similarities. Matrixsimilarity (Lda[corpus]) Sims = Index[query_lda]print List (enumerate (Sims)) #[(0, 0.37399071), (1, 0.9899627), (2, 0.9950707) The results of]# #可以看出来用lsi and LDA models are consistent </span>

As you can see, the result of this query is DOC2 > Doc3 > Doc1, which is consistent with fast tutorial, although there are some differences in the numbers:

Well, this is the end of the example, and the next section will focus on how to calculate the theme similarity between courses on the course map based on Gensim, and consider some of the improvements, including the use of the Natural Language Processing Toolkit NLTK in English and the larger Wikipedia corpus to look at the effect.

How to calculate the similarity of two documents (ii)