Latent semantic analysis Latent semantic analytical note (LSA) principle and code

Article reference: http://blog.sina.com.cn/s/blog_62a9902f0101cjl3.html

Latent Semantic Analysis (LSA), also known as latent Semantic indexing (LSI), is an understanding of the potential meanings and concepts found in these documents by analyzing documents.

If each word represents only one concept. And each concept is described only by one word. LSA will be easy( from Word to concept there is a simple mapping relationship)

Unfortunately, the problem is not so simple. Because there are different words to denote the same meaning (synonym). A word means more than one meaning, and all such two semantics (ambiguities) confuse the notion that sometimes people are very difficult to understand.

For example, the term "bank", which appears along with mortgages, loans and interest rates, often represents financial institutions. However, with bait, throw, fish appear together often indicates the river bank.

Working principle of latent semantic analysis

Latent semantic analysis (latent semanticanalysis) originates from the question: How to find the relevant document from the search query. When we try to find the relevant text through the comparative word. There are hard-to-solve limitations, that is, in search we actually want to go to the comparison is not the word, but hidden in the meaning and concept behind the word. Latent semantic analysis attempts to solve the problem by mapping words and documents to a ' conceptual ' space and comparing them within this space (note: a dimensionality reduction technique).

When the author of a document writes, there is a broad choice of words.

Different authors have different preferences for the choice of words, which can lead to confusion of concepts.

This random selection of words introduces noise in the word-concept relationship.

LSA filters out some of this noise. You can also find the smallest collection of concepts from all documents (why is it minimal?). ）。

In order for this problem to be solved better, LSA introduces some important simplifications:

1. The document is represented as "a bunch of words (bags of words)", so the position of the word in the document is not important, only the number of occurrences of a word.

2. Concepts are expressed as a pattern of some of the words that often come together today. For example, "leash" (leash), "treat", "Obey" (obedience) are often present in the document about training dogs.

3. Words are thought to have only one meaning. This obviously has a counter-example (bank indicates a bank or financial institution), but this can make the problem easier. (How can this simplification be flawed?) ）

Next, look at a sample of the LSA. Next part:

a simple sample

A sample sample. I searched for "investing" (investment) on Amazon.com and took the title of Top10 search results. One of them was discarded because it contained only one index word (indexword) and other headings as well.

An index Word can be any word that satisfies the following conditions:

1. Appears in 2 or more than 2 headings and

2. Not a particularly common word such as "and", "the" (Stop word-stopword).

Such words are not included because they do not exist in their own meaning.

In this example, we take out the following inactive words: "and", "edition", "for", "in", "little", "of", "the", "to".

Here are the 9 headings, which are underlined by the index words (non-stop words that appear in 2 or more than 2 headings):

1. The neatest Little Guide-to-Stock Market Investing

2. Investing for Dummies, 4th Edition

3. The Little Book of Common SenseInvesting: The Onlyway-Guarantee Your Fair Share of theStock market Returns

4. The Little Book ofValue investing

5. valueinvesting: from Graham to Buffett and Beyond

6. Richdad ' s Guide toInvesting: What theRich Invest in,that The Poor and the middle Class do No T!

7. investing in Real Estate, 5th Edition

8. stockinvesting forDummies

9. Richdad ' s advisors:the ABC's ofReal Estate investing: Thesecrets of finding Hidden profits most INVe Stors Miss

In this example, the LSA is applied, and we can draw the position of the word and the title in the XY axis diagram (only 2 dimensions). and identify the cluster of headings. The blue circle represents 9 headings, and the red squares represent 11 index words.

We can not only draw the title of the cluster. And because indexed words can be drawn together in the title. We are also able to label these clusters.

For example, a blue cluster. Including the T7 and T9. It's about the Realestate (real estate), the green cluster. Includes titles T2,t4,t5 and T8. It's about valueinvesting (value investing), and finally the red cluster. Includes the title T1 and T3, which is about stockmarket (stock market). Title T6 is an outlier (outlier)

The first step of the LSA is to create a matrix of words to the title (document). In this matrix, each index word occupies a row, and each title occupies a column. Each unit (cell) includes the number of times the word is in that title. For example, the word "book" Out of the T3 now. Out of the T4 now, and "investing" appeared once in all the headings. In general, the Matrix in the LSA is very large and sparse (most of the cells are 0). This is because each title or document generally contains only a small part of the entire vocabulary. More complex LSA algorithms use this sparsity to improve spatial and temporal complexity.

In this post. We use Python code to implement all of the LSA steps. We'll cover all the code. Python code can be down here (see above). You need to install both the NumPy and scipy libraries.

NumPy is a numerical class for Python, using the zeros (initialization matrix). Scipy.linalg in the library of this linear algebra. We introduce the SVD function, which is the core of the LSA, which makes the mysterious value decomposition.

1. from NumPy Import Zeros
2. from Scipy.linalg Import SVD

Stopwords is a stop word, ignorechars is useless punctuation.

2. Titles =
4. [
6. "The neatest Little Guide to the Stock marketinvesting" ,
8. "Investing for Dummies, 4thEdition" ,
10. "The Little Book of Common senseinvesting:the only the Guarantee YourFair Share of stockmarket Returns" ,
12. "The Little Book of Valueinvesting" ,
14. "Value investing:from Graham to Buffettand Beyond" ,
16. "Rich Dad's Guide Toinvesting:what the Rich Invest in, that's the Poor and the Middleclass do not!" ,
18. "Investing in Real Estate, 5thEdition" ,
20. "Stock Investing fordummies" ,
22. "Rich Dad ' s advisors:the ABC's of Realestate investing:the secrets of finding Hidden profits mostinvestors Miss"
24. ]
26. stopwords = ['and ',' Edition ',' for ', ' in ' , ' Little ' , ' of ' , ' the ' , ' to ' ]
28. Ignorechars = "' "',: '! '

This defines an LSA class that contains its initialization process wdict is a dictionary. DCount is used to record document numbers.

1. class LSA (object):
2. def __init__ (self, stopwords, ignorechars):
3. Self . Stopwords =stopwords
4. Self . Ignorechars =ignorechars
5. Self . Wdict ={}
6. Self . DCount = 0

The function is to break down the document into words and filter out the inactive words and punctuation, and the rest of the words will fill in the wdict with the document numbers that appear, for example. The word book appears in titles 3 and 4 now, then we have self.wdict[' book ']= [3, 4].

The equivalent of building a inverted platoon.

1. def Parse (self, doc):
2. words = Doc.split (); for W inwords:
3. w = w.lower (). Translate (None, self. Ignorechars)
4. if W inch Self . Stopwords:
5. Continue
6. elif W inch Self . Wdict:
7. Self . Wdict[w].append (selfdcount)
8. Else :
9. Self . wdict[w] =[self. DCount]
10. Self . dcount+= 1

After all the documents have been parsed, all occurrences of the word (i.e. the keys of the dictionary) are taken out and sorted. Create a matrix. The number of rows is the number of words, and the number of columns is the number of documents. At last. The values of the corresponding matrix cells are counted for all the words and documents.

1. def Build (self):
2. Self . Keys =[k for kin self. Wdict.keys ()if Len (selfwdict[k]) >1]
3. Self . Keys.sort ()
4. Self . A =zeros ([Len (self. Keys),self. DCount])
5. for I, K inch Enumerate (self. Keys):
6. for D inch Self . Wdict[k]:
7. Self . a[i,d]+= 1

Print the matrix and draw the graph.

def PrintA (self):
Print self. A

#神秘值分解矩阵为u, S,VT
U,S,VT = SVD (self. A

Print "" \ r "" "
Print U
Print "" \ r "" "
Print S
Print "" \ r "" "
Print VT
Print "" \ r "" "

#绘图的标题和x轴和y轴的维度名称

Plt.title ("LSI")
Plt.xlabel (U ' dimention2 ')
Plt.ylabel (U ' dimention3 ')

#画文档标题
titles = [' T1 ', ' T2 ', ' T3 ', ' T4 ', ' T5 ', ' T6 ', ' T7 ', ' T8 ', ' T9 ']
Vdemention2 = vt[1]
Vdemention3 = vt[2]
For j in Range (Len (vdemention2)):
Text (Vdemention2[j],vdemention3[j],titles[j])
Plot (Vdemention2, Vdemention3, '. ')

#画词语
UT = u.t
Demention2 = ut[1]
Demention3 = ut[2]
For I in range (len (demention2)):
Text (Demention2[i],demention3[i],self.keys[i])
Plot (Demention2, Demention3, '. ')

Program Entry:

Mylsa = LSA (stopwords, Ignorechars)

For T in titles:

Mylsa.parse (t)

Mylsa.build ()

Mylsa.printa ()

The code is executed using Winpython's own Spyder. The following sections are source code:

#-*-Coding:utf-8-*-"" "Created on Wed June one 17:02:39 2014@author:modified by Zhouxu,add plot" "" from NumPy import Zero Simport NumPy as Npfrom scipy.linalg import svdtitles =["The neatest Little Guide to Stock Market investing", "INVe Sting for Dummies, 4th Edition ', ' The Little Book of Common Sense Investing:the ' only ' to ' Guarantee Your Fair Share Of the Stock market Returns ', ' the Little Book of Value Investing ', "value Investing:from Graham to Buffett and Beyond" , "Rich Dad's Guide to investing:what the rich Invest in, that the Poor and the middle Class do not!", "Investing I n Real Estate, 5th Edition "," Stock Investing For Dummies "," Rich Dad's advisors:the ABC ' s of Real Estate Investing : The secrets of finding Hidden profits most investors Miss "]stopwords = [' and ', ' edition ', ' for ', ' in ', ' little ', ' of ', ' the ', ' to ']ignorechars = ' ',: '! ' Class LSA (object): Def __init__ (self, Stopwords, ignorechars): Self.stopwords = Stopwords Self.ignoRechars = Ignorechars self.wdict = {} self.dcount = 0 def parse (self, doc): words = Doc.split ();            For W in words: #print self.dcount w = w.lower (). Translate (None, self.ignorechars) If w in Self.stopwords:continue elif W in Self.wdict:self.wdict[w].append (sel        F.dcount) Else:self.wdict[w] = [Self.dcount] Self.dcount + 1 def build (self):        Self.keys = [k for k in Self.wdict.keys () If Len (Self.wdict[k]) > 1] self.keys.sort () print Self.keys Self.                A = Zeros ([Len (Self.keys), Self.dcount]) for I, K in Enumerate (Self.keys): For D in Self.wdict[k]: Self. A[I,D] + = 1 def printA (self): print self. A U,S,VT = SVD (self.                A) print "" \ r "" "Print U print" "" \ r "" "Print s print" "\ r" ""      Print VT print "" \ r "" "            Plt.title ("LSA") Plt.xlabel (U ' dimention2 ') plt.ylabel (U ' dimention3 ') titles = [' T1 ', ' T2 ', ' T3 ', ' T4 ', ' T5 ', ' T6 ', ' T7 ', ' T8 ', ' T9 '] vdemention2 = vt[1] Vdemention3 = vt[2] for j in Range (Len (vdemen                        Tion2): Text (Vdemention2[j],vdemention3[j],titles[j]) plot (Vdemention2, Vdemention3, '. ')            UT = u.t demention2 = ut[1] Demention3 = ut[2] for i in range (len (demention2)):                        Text (Demention2[i],demention3[i],self.keys[i]) plot (Demention2, Demention3, '. ') Mylsa = LSA (Stopwords, ignorechars) for T in Titles:mylsa.parse (t) mylsa.build () Mylsa.printa ()

Program execution Results:

Watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvym9imda3/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/dissolve/70 /gravity/center ">

Copyright notice: This article Bo Master original articles, blogs, without consent may not be reproduced.

Latent semantic analysis Latent semantic analytical note (LSA) principle and code