java實現文本詞頻統計

標籤：java實現   key   get   hashmap   builder   過濾   stream   stringbu   shm   

File f=new File(path);
Map<String,Integer>map=new HashMap<>();
Version matchVersion = Version.LUCENE_31;
Analyzer analyzer = new StopAnalyzer(matchVersion);
BufferedReader br = new BufferedReader(new FileReader(f));//讀取檔案

TokenStream ts = analyzer.tokenStream(null, br);//用analyzer分詞，得到token流
ts = new PorterStemFilter(ts);//過濾器提取詞幹
CharTermAttribute ca = ts.addAttribute(CharTermAttribute.class);//ca儲存了ts的文本資訊
ts.reset();//必須的
while(ts.incrementToken()){
String term = ca.toString();
if(!map.keySet().contains(term)){
map.put(term, 1);
}else
{
map.put(term, map.get(term)+1);
}
}
ts.end();
ts.close();
analyzer.close();
br.close();

StringBuilder sb=new StringBuilder();
File gh=new File(path);
for(String key:map.keySet()){
sb.append(key+" "+map.get(key)+"\r\n");
}
BufferedWriter bw=new BufferedWriter(new FileWriter(gh));
bw.write(sb.toString());
bw.flush();
bw.close();

java實現文本詞頻統計