Python_LDA實現方法詳解,python_lda實現詳解

來源:互聯網
上載者:User

Python_LDA實現方法詳解,python_lda實現詳解

LDA(Latent Dirichlet allocation)模型是一種常用而用途廣泛地機率主題模型。其實現一般通過Variational inference和Gibbs Samping實現。作者在提出LDA模型時給出了其變分推理的C源碼(後續貼出C++改編的類),這裡貼出基於Python的第三方模組改寫的LDA類及實現。

#coding:utf-8import numpy as npimport ldaimport lda.datasetsimport jiebaimport codecsclass LDA_v20161130():  def __init__(self, topics=2):    self.n_topic = topics    self.corpus = None    self.vocab = None    self.ppCountMatrix = None    self.stop_words = [u',', u'。', u'、', u'(', u')', u'·', u'!', u' ', u':', u'“', u'”', u'\n']    self.model = None  def loadCorpusFromFile(self, fn):    # 中文分詞    f = open(fn, 'r')    text = f.readlines()    text = r' '.join(text)    seg_generator = jieba.cut(text)    seg_list = [i for i in seg_generator if i not in self.stop_words]    seg_list = r' '.join(seg_list)    # 切割統計所有出現的詞納入詞典    seglist = seg_list.split(" ")    self.vocab = []    for word in seglist:      if (word != u' ' and word not in self.vocab):        self.vocab.append(word)    CountMatrix = []    f.seek(0, 0)    # 統計每個文檔中出現的詞頻    for line in f:      # 置零      count = np.zeros(len(self.vocab),dtype=np.int)      text = line.strip()      # 但還是要先分詞      seg_generator = jieba.cut(text)      seg_list = [i for i in seg_generator if i not in self.stop_words]      seg_list = r' '.join(seg_list)      seglist = seg_list.split(" ")      # 查詢詞典中的詞出現的詞頻      for word in seglist:        if word in self.vocab:          count[self.vocab.index(word)] += 1      CountMatrix.append(count)    f.close()    #self.ppCountMatrix = (len(CountMatrix), len(self.vocab))    self.ppCountMatrix = np.array(CountMatrix)    print "load corpus from %s success!"%fn  def setStopWords(self, word_list):    self.stop_words = word_list  def fitModel(self, n_iter = 1500, _alpha = 0.1, _eta = 0.01):    self.model = lda.LDA(n_topics=self.n_topic, n_iter=n_iter, alpha=_alpha, eta= _eta, random_state= 1)    self.model.fit(self.ppCountMatrix)  def printTopic_Word(self, n_top_word = 8):    for i, topic_dist in enumerate(self.model.topic_word_):      topic_words = np.array(self.vocab)[np.argsort(topic_dist)][:-(n_top_word + 1):-1]      print "Topic:",i,"\t",      for word in topic_words:        print word,      print  def printDoc_Topic(self):    for i in range(len(self.ppCountMatrix)):      print ("Doc %d:((top topic:%s) topic distribution:%s)"%(i, self.model.doc_topic_[i].argmax(),self.model.doc_topic_[i]))  def printVocabulary(self):    print "vocabulary:"    for word in self.vocab:      print word,    print  def saveVocabulary(self, fn):    f = codecs.open(fn, 'w', 'utf-8')    for word in self.vocab:      f.write("%s\n"%word)    f.close()  def saveTopic_Words(self, fn, n_top_word = -1):    if n_top_word==-1:      n_top_word = len(self.vocab)    f = codecs.open(fn, 'w', 'utf-8')    for i, topic_dist in enumerate(self.model.topic_word_):      topic_words = np.array(self.vocab)[np.argsort(topic_dist)][:-(n_top_word + 1):-1]      f.write( "Topic:%d\t"%i)      for word in topic_words:        f.write("%s "%word)      f.write("\n")    f.close()  def saveDoc_Topic(self, fn):    f = codecs.open(fn, 'w', 'utf-8')    for i in range(len(self.ppCountMatrix)):      f.write("Doc %d:((top topic:%s) topic distribution:%s)\n" % (i, self.model.doc_topic_[i].argmax(), self.model.doc_topic_[i]))    f.close()

演算法實現demo:

例如,抓取BBC川普當選的新聞作為語料,輸入以下代碼:

if __name__=="__main__":  _lda = LDA_v20161130(topics=20)  stop = [u'!', u'@', u'#', u',',u'.',u'/',u';',u' ',u'[',u']',u'$',u'%',u'^',u'&',u'*',u'(',u')',      u'"',u':',u'<',u'>',u'?',u'{',u'}',u'=',u'+',u'_',u'-',u'''''']  _lda.setStopWords(stop)  _lda.loadCorpusFromFile(u'C:\\Users\Administrator\Desktop\\BBC.txt')  _lda.fitModel(n_iter=1500)  _lda.printTopic_Word(n_top_word=10)  _lda.printDoc_Topic()  _lda.saveVocabulary(u'C:\\Users\Administrator\Desktop\\vocab.txt')  _lda.saveTopic_Words(u'C:\\Users\Administrator\Desktop\\topic_word.txt')  _lda.saveDoc_Topic(u'C:\\Users\Administrator\Desktop\\doc_topic.txt')

因為語料全部為英文,因此這裡的stop_words全部設定為英文符號,主題設定20個,迭代1500次。結果顯示,文檔148篇,詞典1347詞,總詞數4174,在i3的電腦上運行17s。
Topic_words部分輸出如下:

Topic: 0
to will and of he be trumps the what policy
Topic: 1 he would in said not no with mr this but
Topic: 2 for or can some whether have change health obamacare insurance
Topic: 3 the to that president as of us also first all
Topic: 4 trump to when with now were republican mr office presidential
Topic: 5 the his trump from uk who president to american house
Topic: 6 a to that was it by issue vote while marriage
Topic: 7 the to of an are they which by could from
Topic: 8 of the states one votes planned won two new clinton
Topic: 9 in us a use for obama law entry new interview
Topic: 10 and on immigration has that there website vetting action given

Doc_Topic部分輸出如下:

Doc 0:((top topic:4) topic distribution:[ 0.02972973 0.0027027 0.0027027 0.16486486 0.32702703 0.19189189
0.0027027 0.0027027 0.02972973 0.0027027 0.02972973 0.0027027
0.0027027 0.0027027 0.02972973 0.0027027 0.02972973 0.0027027
0.13783784 0.0027027 ])
Doc 1:((top topic:18) topic distribution:[ 0.21 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.11 0.01 0.01 0.01
0.01 0.01 0.01 0.01 0.01 0.01 0.31 0.21])
Doc 2:((top topic:18) topic distribution:[ 0.02075472 0.00188679 0.03962264 0.00188679 0.00188679 0.00188679
0.00188679 0.15283019 0.00188679 0.02075472 0.00188679 0.24716981
0.00188679 0.07735849 0.00188679 0.00188679 0.00188679 0.00188679
0.41698113 0.00188679])

當然,對於英文語料,需要排除大部分的虛詞以及常用無意義詞,例如it, this, there, that...在實際操作中,需要合理地設定參數。

換中文語料嘗試,採用習大大就卡斯特羅逝世發表的弔唁文章和樸槿惠辭職的新聞。

Topic: 0
的 同志 和 人民 卡斯特羅 菲德爾 古巴 他 了 我
Topic: 1 在 樸槿惠 向 表示 總統 對 將 的 月 國民
Doc 0:((top topic:0) topic distribution:[ 0.91714123 0.08285877])
Doc 1:((top topic:1) topic distribution:[ 0.09200666 0.90799334])

還是存在一些虛詞,例如“的”,“和”,“了”,“對”等詞的幹擾,但是大致來說,兩則新聞的主題分布很明顯,效果還不賴。

總結

以上就是本文關於Python_LDA實現方法詳解的全部內容,希望對大家有所協助。感興趣的朋友可以繼續參閱本站:python+mongodb資料抓取詳細介紹、Python探索之建立二叉樹、Python探索之修改Python搜尋路徑等,有什麼問題可以隨時留言,歡迎大家一起交流討論。感謝朋友們對本站的支援!

聯繫我們

該頁面正文內容均來源於網絡整理,並不代表阿里雲官方的觀點,該頁面所提到的產品和服務也與阿里云無關,如果該頁面內容對您造成了困擾,歡迎寫郵件給我們,收到郵件我們將在5個工作日內處理。

如果您發現本社區中有涉嫌抄襲的內容,歡迎發送郵件至: info-contact@alibabacloud.com 進行舉報並提供相關證據,工作人員會在 5 個工作天內聯絡您,一經查實,本站將立刻刪除涉嫌侵權內容。

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.