Python_LDA implementation method and python_lda implementation

Source: Internet
Author: User

Python_LDA implementation method and python_lda implementation

LDA (Latent Dirichlet allocation) model is a common and widely used probabilistic topic model. Its implementation is generally implemented through Variational inference and the glassamping. When proposing the LDA model, the author provides the C source code of its variational Reasoning (the C ++ adapted class will be added later). Here, the LDA class and implementation of Python-based third-party module rewriting are provided.

# Coding: utf-8import numpy as npimport ldaimport lda. datasetsimport jiebaimport codecsclass LDA_v20161130 (): def _ init _ (self, topics = 2): self. n_topic = topics self. corpus = None self. vocab = None self. ppCountMatrix = None self. stop_words = [U', ', U '. ', U',', U' (', U')', U' · ', U '! ', U'', U':', U' "', U'"', U' \ n'] self. model = None def loadCorpusFromFile (self, fn): # Chinese Word Segmentation f = open (fn, 'R') text = f. readlines () text = r ''. join (text) seg_generator = jieba. cut (text) seg_list = [I for I in seg_generator if I not in self. stop_words] seg_list = r ''. join (seg_list) # Cut all words that appear into the dictionary seglist = seg_list.split ("") self. vocab = [] for word in seglist: if (word! = U'' and word not in self. vocab): self. vocab. append (word) CountMatrix = [] f. seek (0, 0) # count the Word Frequency in each document for line in f: # set zero count = np. zeros (len (self. vocab), dtype = np.int) text = line. strip () # However, you must first split seg_generator = jieba. cut (text) seg_list = [I for I in seg_generator if I not in self. stop_words] seg_list = r ''. join (seg_list) seglist = seg_list.split ("") # query the word frequency in the dictionary for word in seglist: if w Ord in self. vocab: count [self. vocab. index (word)] + = 1 CountMatrix. append (count) f. close () # self. ppCountMatrix = (len (CountMatrix), len (self. vocab) self. ppCountMatrix = np. array (CountMatrix) print "load corpus from % s success! "% Fn def setStopWords (self, word_list): self. stop_words = word_list def fitModel (self, n_iter = 1500, _ alpha = 0.1, _ eta = 0.01): self. model = lda. LDA (n_topics = self. n_topic, n_iter = n_iter, alpha = _ alpha, eta = _ eta, random_state = 1) self. model. fit (self. ppCountMatrix) def printTopic_Word (self, n_top_word = 8): for I, topic_dist in enumerate (self. model. topic_word _): topic_words = np. array (self. vocab) [np. argsort (topic_dist)] [:-(n_top_word + 1):-1] print "Topic:", I, "\ t", for word in topic_words: print word, print def printDoc_Topic (self): for I in range (len (self. ppCountMatrix): print ("Doc % d :( (top topic: % s) topic distribution: % s)" % (I, self.model.doc_topic_polici#.argmax(,,self.model.doc _ topic _ [I]) def printVocabulary (self): print "vocabulary:" for word in self. vocab: print word, print def saveVocabulary (self, fn): f = codecs. open (fn, 'w', 'utf-8') for word in self. vocab: f. write ("% s \ n" % word) f. close () def saveTopic_Words (self, fn, n_top_word =-1): if n_top_word =-1: n_top_word = len (self. vocab) f = codecs. open (fn, 'w', 'utf-8') for I, topic_dist in enumerate (self. model. topic_word _): topic_words = np. array (self. vocab) [np. argsort (topic_dist)] [:-(n_top_word + 1):-1] f. write ("Topic: % d \ t" % I) for word in topic_words: f. write ("% s" % word) f. write ("\ n") f. close () def saveDoc_Topic (self, fn): f = codecs. open (fn, 'w', 'utf-8') for I in range (len (self. ppCountMatrix): f. write ("Doc % d :( (top topic: % s) topic distribution: % s) \ n" % (I, self.model.doc _ topic _ [I]. argmax (), self.model.doc _ topic _ [I]) f. close ()

Algorithm Implementation demo:

For example, capture the BBC trump election news as a corpus and enter the following code:

if __name__=="__main__":  _lda = LDA_v20161130(topics=20)  stop = [u'!', u'@', u'#', u',',u'.',u'/',u';',u' ',u'[',u']',u'$',u'%',u'^',u'&',u'*',u'(',u')',      u'"',u':',u'<',u'>',u'?',u'{',u'}',u'=',u'+',u'_',u'-',u'''''']  _lda.setStopWords(stop)  _lda.loadCorpusFromFile(u'C:\\Users\Administrator\Desktop\\BBC.txt')  _lda.fitModel(n_iter=1500)  _lda.printTopic_Word(n_top_word=10)  _lda.printDoc_Topic()  _lda.saveVocabulary(u'C:\\Users\Administrator\Desktop\\vocab.txt')  _lda.saveTopic_Words(u'C:\\Users\Administrator\Desktop\\topic_word.txt')  _lda.saveDoc_Topic(u'C:\\Users\Administrator\Desktop\\doc_topic.txt')

Because all the corpus is in English, stop_words here are all set as English symbols, 20 theme settings, and 1500 iterations. The results show that there are 148 documents, 1347 words in the dictionary, 4174 words in total, and 17 s running on the i3 computer.
The output of Topic_words is as follows:

Topic: 0
To will and of he be trumps the what policy
Topic: 1 he wowould in said not no with mr this
Topic: 2 for or can some whether have change health obamacare insurance
Topic: 3 the to that president as of us also first all
Topic: 4 trump to when with now were republican mr office presidential
Topic: 5 the his trump from uk who president to american house
Topic: 6 a to that was it by issue vote while marriage
Topic: 7 the to of an are they which by cocould from
Topic: 8 of the states one votes planned won two new clinton
Topic: 9 in us a use for obama law entry new interview
Topic: 10 and on immigration has that there website vetting action given

The output of Doc_Topic is as follows:

Doc 0 :( (top topic: 4) topic distribution: [0.02972973 0.0027027 0.0027027 0.16486486 0.32702703 0.19189189
0.0027027 0.0027027 0.02972973 0.0027027 0.02972973 0.0027027
0.0027027 0.0027027 0.02972973 0.0027027 0.02972973 0.0027027
0.13783784])
Doc 1 :( (top topic: 18) topic distribution: [0.21 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.11 0.01 0.01 0.01
0.01 0.01 0.01 0.01 0.01 0.01 0.31 0.21])
Doc 2 :( (top topic: 18) topic distribution: [0.02075472 0.00188679 0.03962264 0.00188679 0.00188679 0.00188679
0.00188679 0.15283019 0.00188679 0.02075472 0.00188679 0.24716981
0.00188679 0.07735849 0.00188679 0.00188679 0.00188679 0.00188679
0.41698113])

Of course, for English Corpus, You need to exclude most of the virtual words and commonly used meaningless words, such as it, this, there, that... in actual operations, you need to set parameters reasonably.

An attempt to change the Chinese corpus was made to use Xi's report on the death of Mr. Castel and the resignation of Park Geun.

Topic: 0
The comrades and the people of the Cuban
Topic: 1 at Park Geun, the President told us that
Doc 0 :( (top topic: 0) topic distribution: [0.91714123 0.08285877])
Doc 1 :( (top topic: 1) topic distribution: [0.09200666 0.90799334])

There are still some virtual words, such as "of", "and", "," and "," and "on", but roughly speaking, the topics of the two news are obviously distributed, the effect is good.

Summary

The above is all the details about the Python_LDA implementation method in this article. I hope it will be helpful to you. If you are interested, please refer to this site: python + mongodb data capturing details, creating a binary tree for Python exploration, modifying the Python search path for Python exploration, etc. If you have any questions, please feel free to leave a message, you are welcome to discuss with us. Thank you for your support!

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.