python產生職業要求詞雲

標籤：port   index   creat   logs   plt   jobs   img   image   dex   

接著上篇的說的，爬取了大資料相關的職位資訊，http://www.17bigdata.com/jobs/。

# -*- coding: utf-8 -*-"""Created on Thu Aug 10 07:57:56 2017@author: lenovo"""from wordcloud import WordCloudimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport jiebadef cloud(root,name,stopwords):    filepath = root +‘\\‘ + name    f = open(filepath,‘r‘,encoding=‘utf-8‘)    txt = f.read()    f.close()    cut = jieba.cut(txt)    words = []    for i in cut:        words.append(i)    df = pd.DataFrame({‘words‘:words})    s= df.groupby(df[‘words‘])[‘words‘].agg([(‘size‘,np.size)]).sort_values(by=‘size‘,ascending=False)    s = s[~s.index.isin(stopwords[‘stopword‘])].to_dict()    wordcloud = WordCloud(font_path =r‘E:\Python\machine learning\simhei.ttf‘,background_color=‘black‘)    wordcloud.fit_words(s[‘size‘])    plt.imshow(wordcloud)    pngfile = root +‘\\‘ + name.split(‘.‘)[0] + ‘.png‘    wordcloud.to_file(pngfile)    import os jieba.load_userdict(r‘E:\Python\machine learning\NLPstopwords.txt‘)stopwords = pd.read_csv(r‘E:\Python\machine learning\StopwordsCN.txt‘,encoding=‘utf-8‘,index_col=False)for root,dirs,file in os.walk(r‘E:\職位資訊‘):    for name in file:        if name.split(‘.‘)[-1]==‘txt‘:            print(name)            cloud(root,name,stopwords)        

詞云：

可以看出有些雜訊詞沒能被去除，比如相關、以上學曆等無效詞彙。本想通過DF判斷停用詞，但是我爬的時候沒顧及到這個問題，外加本身記錄數也不高，就沒再找職位資訊的停用詞。當然也可看出演算法和經驗是很重要的。加油

python產生職業要求詞雲