hadoop-statistics The number of names appearing in a dream of red mansions

 PackageOrg.bigdata508.util;ImportJava.io.BufferedReader;ImportJava.io.File;Importjava.io.FileNotFoundException;ImportJava.io.FileReader;Importjava.io.IOException;ImportJava.io.StringReader;Importjava.util.ArrayList;ImportJava.util.HashSet;Importjava.util.List;ImportJava.util.Set;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;Importorg.apache.hadoop.io.IntWritable;Importorg.apache.hadoop.io.LongWritable;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.Mapper;ImportOrg.apache.hadoop.mapreduce.Reducer;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.lucene.analysis.TokenStream;ImportOrg.apache.lucene.analysis.tokenattributes.CharTermAttribute;ImportORG.BIGDATA.UTIL.HADOOPCFG;ImportOrg.wltea.analyzer.lucene.IKAnalyzer; Public classdreamofredmansion { Public Staticset<string> dic =NewHashset<>(); Static{String ProjectPath= Dreamofredmansion.class. GetResource ("/"). GetFile (). toString (); Try{bufferedreader br=NewBufferedReader (NewFileReader (NewFile (ProjectPath+ File.separator + "\\DreamOfRed.txt"))); String Line=NULL;  while((Line=br.readline ())! =NULL) { line=line.replaceall ("\\s+", "" ");            Dic.add (line);        } br.close (); } Catch(FileNotFoundException e) {e.printstacktrace (); } Catch(IOException e) {e.printstacktrace (); }    }    Private StaticList<string>Parse (String text) {List<String> words =NewArraylist<string>(); //To create a ikanalyzer Chinese Word objectIkanalyzer Analyzer =NewIkanalyzer (); //using intelligent participleAnalyzer.setusesmart (true); //participleStringReader reader =NewStringReader (text); Try{tokenstream TS= Analyzer.tokenstream ("Content", reader); Chartermattribute STRs= Ts.getattribute (Chartermattribute.class); Ts.addattribute (Chartermattribute.class); //traversing word breaker data             while(Ts.incrementtoken ()) {if(Dic.contains (strs.tostring ())) {Words.add (strs.tostring ()); }            }        } Catch(IOException e) {e.printstacktrace ();        } reader.close (); returnwords; }        Private Static classDreamofredmansionmapperextendsmapper<longwritable, text, text, longwritable>{@Overrideprotected voidmap (longwritable key, Text value, context context)throwsIOException, interruptedexception {List<String> names =Parse (value.tostring ());  for(String name:names) {context.write (NewText (name),NewLongwritable (1)); }        }    }    Private Static classDreamofredmansionreducerextendsReducer<text, Longwritable, Text, longwritable>{@Overrideprotected voidReduce (Text key, iterable<longwritable>values, context context)throwsIOException, interruptedexception {Long sum= 0L;  for(longwritable value:values) {sum= Sum +Value.get (); } context.write (NewText (key+ ","),Newlongwritable (sum)); }    }     Public Static classDreamofredmansionsortmapperextendsMapper<longwritable, Text, longwritable, text>{@Overrideprotected voidMap (longwritable key, text value, mapper<longwritable, text, longwritable, text>. Context context)throwsIOException, interruptedexception {longwritable data=NewLongwritable (Integer.parseint (Value.tostring () Split (",") [1].trim ())); Context.write (data,NewText (Value.tostring (). Split (",") [0])); }            }         Public Static classDreamofredmansionsortreduceextendsreducer<longwritable, text, text, longwritable>{@Overrideprotected voidReduce (longwritable key, iterable<text>values, Reducer<longwritable, text, text, Longwritable> Context context)throwsIOException, interruptedexception { for(Text str:values) {context.write (str, key); }        }            }     Public Static voidMain (string[] args)throwsexception{Configuration cfg=hadoopcfg.getinstance (); Job Job=job.getinstance (CFG); Job.setjobname ("Dreamofredmansion"); Job.setjarbyclass (dreamofredmansion.class); //MapperJob.setmapperclass (Dreamofredmansionmapper.class); Job.setmapoutputkeyclass (Text.class); Job.setmapoutputvalueclass (longwritable.class); //ReducerJob.setreducerclass (Dreamofredmansionreducer.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (longwritable.class); Fileinputformat.addinputpath (Job,NewPath ("/dreamofredmansioninput")); Fileoutputformat.setoutputpath (Job,NewPath ("/red_out/")); Job.waitforcompletion (true); Configuration conf=hadoopcfg.getinstance (); Job Job2=job.getinstance (conf); Job2.setjobname ("Dreamofredmansionsort"); Job2.setjarbyclass (dreamofredmansion.class); //SortmapperJob2.setmapperclass (Dreamofredmansionsortmapper.class); Job2.setmapoutputkeyclass (longwritable.class); Job2.setmapoutputvalueclass (Text.class); //SortreducerJob2.setreducerclass (dreamofredmansionsortreduce.class); Job2.setoutputkeyclass (Text.class); Job2.setoutputvalueclass (longwritable.class); Fileinputformat.addinputpath (JOB2,NewPath ("/red_out")); Fileoutputformat.setoutputpath (JOB2,NewPath ("/red_out_sort/")); Job2.waitforcompletion (true); }}

The DIC dictionary gets the matching name:

DreamOfRed.txt

Lin Daiyu xue Bao Chai Jia Yuanchun Jia Chun Jia Tanchun Jia spring Li Yumiong Yu Yun Wang Xifeng Jia Chi Qin keqing qing wen musk deer moon duck snow geese purple cuckoo blue mark flat son incense Ling Jin Yu Chess holding piano lai big Coke king good bao Zhou Ruilin Baoyong Wu Gui colleague Deng Good time Ashoka son Yu Xinqing son Xing son lung son of son son of child life son son child small house Li Xing Jade chopstick son Jia Jing jia pardon Jia Zheng Jia Bao Yu Jia Lian Jia Jian baoyu Jayong Ocalan Jia Yin Jia Yuan Qi Guanfang Lotus Root official Medicine officer Yugenbao Official Age official eggplant Aiguen Bean Officer Kwai official Miaoyu Intelligent Wisdom Tong Zhi good round letter big color empty color colorful Choi Feng Choi Phoenix Colorful color bright clouds Zhangbaoyu Siborchen xue pan Shirewang lady Wang Ziteng US Erjie jie Jaminjarejadah Jadaihua Jadaisiu Cheng Zhixing nerd Zhang Hua Feng Yu Zhang Brother tea smoke sweep red hoe Medicine companion crane Small Magpie Small red cicada Liu grandmother Madaubo song mother Zhang Mom Qin Yu Shing Liu Xiang Lotus Dongping Wang Ujin Leng xingshan zi Fang load right Xia Bingzhong Zhou eunuch Chu Shian holding piano chess The picturesque pearl Amber Glass Jade Two wood two girl Busi Buren Bu Picturesque picturesque of uncle picturesque of the master big Elder sister son million son million virtual mountain son wild small red small Chardonnay small small cicada small small baby small little OIOS small little son small child small baby little dove small Good luck son Khumanning loyal prince Wei Joran female Nijo first Parma Shangmaqui Madaubo Fung Wang Ren Wang's Princes Wang Chengwing Wangzhong Wang Rong Wang Xin-wang affixed King King Doctor Wang aunt Wang Uncle Wang Zisheng Prince Teng King cure too much place tune King Grandma Wang Nanny King Master Wang Zuomei Wang Wang June effect Wang Hising and Rong Wang Jiren Wang Qing son king son Wang Guole king of the King's short legs Wang Shanbao Wangdosswan mother Wang Cheng father Wang Xing wife Wang Xin home Wang Ziteng Mrs. Wang live daughter-in-law Wang Shanbao home of the cloud son Yun Guangyun master five son five sister-in-law Hong Chun Concubine Overlord Emperor us Erjie Jie, especially mother Juloan People Niuqing New Jizong Mao Ugen Long mansion official Changan Garrison Changan Mansion Changan Garrison of the son revenge Fengfeng sister Fung Brother Chicken Girl Chicken Brothers Phoenix spicy culture civilian Wen Qingwen wonderful live apricot Fang introduction Spring Sorrow Gold Doubles Shoudri Kongjizong book official water-soluble Narcissus temple gu zi Jade Guan yu gui jade love Jade gui son Yusing Yu Yu son Niang Jade Gui son home jade daughter-in-law jade chopstick daughter-in-law Qiao sister AI officer can Can son Stone Liang Mian stone nerd Shihei son Peace Festival degree Dongping County Wang Tian Mom leaf born leaf mom history public Stine Shi Staijun shi Xiangyun Stine lady Shi lady Shi Xiangyun shi Xiangyun father Shi Xiangyun mother four Sister IV funny Wang No Wang No County Baoyong Princess Baiyu white old wife white old daughter-in-law son Lok good Dukes Waifan Wang Ningguo Public von Von Dong Feng Fat Feng Zi British secretary Chess chess mom Xingsir Shangzhong xing Uncle Mrs. 邢岫 Smoke Xing sister-in-law Shand old three old Zhao old monk old concubine mechanism head old Tian Mama old song Mama Old Zhu Lao Zhang mom old leaf mom old Wang home old Wang Dausiping rajah Xining Dukes Gray-to-adult Bishi sweep red sweep Flower Bean officer with your hi Zhu's wife, more official children, many girls, more muddy insects, Wu General Qing son Qing Guo Gong liusi Liu Liu Ma Liu Big Husband Liu Tie mouth xing son An Guo sebastes Schlegeli Garrison son Yin and Yang Sundashin Sun Shaozu shou son Sousanber Rue FangThe official flower mother Flower from the flower elder sister Flowers Big Elder sisters Yan Master Apricot Slave lee two Li Lee mom Revan Li Xiao Li Gui Li Xiang lowered reed Li Xing Lee young man lee boobs Lee grandma Li Shouzhong Xian son Lee Li Gong cut Li Hui Lee Royal history Lee Aunt Lee aunt Lee 县衙very Lee Young Yang Denrong Yang to rise to the prosperous come to Wang son to raise daughter to the son to Wang home to Wang daughter-in-law to the happy daughter-in-law Wu Liang Wu Gui Wuxingwu aunt Wu Tianhuo Wu, Wu Guire Wu Guifei wife Wu Gui family Wunwendenwuchengen daughter-in-law Wu Gui son-in-law, when you feel the king of the Sons of God, when the mother colleague the spring spring Mom Lengishing Lengishing home of Qin Xiang Shen visiting Shen Mama Song Mother song mom and dad song mothers Forget Inryang son miaoyu Zhang Yizhang sheets Zhang temporary doctor Zhang Zhang Wangii intension Medical Zhang Youshi Zhang Zhang Lao Zhang, such as Zhang Gui Zhang, Judge Zhang Juojin Zhang Brother Zhang Yi advocate home Zhang Live Zhang Daoshi Zhang Dehui Zhang Grandpa Grand Master Sheets Home Chen Yichen also Jun Chenruiwen fall son fall son Niang Green son Ying Lin three Linhai Lin Lin Aunt Lin Zhihoringru Hai Linhong Yulin of filial piety Lin Zhixiao home of Roses flower Board son pillow xia old friend Guaizi hold Qin Wang son Wang Re home of Wang son-in-law Wang daughter-in-law loyal Jing Houzhongyi Pro Wang Zhongshun rajah homage Prince Virtuous Princess Good Wai book Pei Feng Jin kinyoun Venus Kim Yorkin Brother Gold color Jin Wenxiang Jin Yu Golden widow Golden Jin Wenxiang Jin Wenxiang home Jin Wenxiang daughter-in-law gold color wife dog son Zhou Zhou Junzhouri Tuesday, Aunt Zhou Zhou Aunt Week eunuch Zhou Gong week Mammy Week mother Zhou Concubine Zhou Sister Zhou Home Zhou Niang Aunt Week sister-in-law Zhou Yi main Zhou Rui home Zhou Rui daughter-in-law Yeeli Hua Zheng Entrance when Yeeli Hua Zheng home of Zheng Good when daughter-in-law entrance home of Yi Red son Candalian single aunt single hired benevolence fried beans net false bao Bogongbao Jewelry Chai chengzhong hou empty man spring fiber Spring Yan Pearl Jean Big grandmother Jane Mrs. Zhang Glass Seal District Zhao Tiandong Zhaotianliang Zhao eunuch Zhao Yihua Zhao Master Zhao nanny shi Qu Officer Zhao Zhao Vice Minister Zhao Aunt Zhao mother Zhao aunt and grandma Qian Xue Medicine officer Boundless Tuas Rong Hu Hu Tai medical hu Master Hu Junrong Huslay Hudaochang hu Quack Moshino hu Lao name public Nanan Wang Nan-jucheon Wang Nan-an Wang Taifei suppository Liushi Fang Liu Liu Biao willow Mother Liu Willow Home Willow Xiang Lotus Willow Willow Aunt Willow two daughter-in-law Willow family Daughter-in-law pillar Snow Lin ' An, Lin Chang-lin Linan bo chiming pro Chang Bo chiming linan the old Lady Zhao son Zhao let love the autumn grain Autumn Ling autumn Cuschand the Emperor Empress Dowager Hou Xiaokang Hou Xiaoming boryeong Hou Huro degree hate bodhi Lou wish mom God Ying Waiter send Jade man fee Aunt Jiao red jiao apricot jiang bead grass jiang Bead Xian jiang Bead Fairy jiang Flower hole main Qin's Qin Xian qinzong Qin Yuqing Qin Xian wife Qin Xian home so-oak Lotus flower Ying son Ying son Guille home Jiahua Chia Ocalan jiamu Jiazhi Jia Yin Jia Zhi Jafan Jiafang baoyu Jafaja van Jiajin Jia Xing Jia Jia Jia Zhu Jia Jia Jia Jia bacteria Jia Ping Jiawa Jia Jia Chen Jia Lan Jia Jia Jayong Jia Yuan Jia Jia show Jia Juan Jia Jia Jia apply Jia Jiazao Jia Tianxiang Jadaihua Jadaisiu Jadaishan Jadairu Jiacun Zhou Jiahu village Jia Shifei Jain Hu Jiachao sister Jia Shi four funny Jayong daughter Gunjungu mother Jia (Wang Yu) mother since Zhabdrung Shahisha Mrs Summer eunuch Summer granny Xia Jingui Xiashou 夏秉忠 Summer woman round letter Money LQian Chang Kui and the United States Tong Shi official embroidered chicken embroidered orange Luan Lian Two grandmother Warbler Huang, Cao Xueqin Mei Hanlin the son Jiang Jianhui Ling Chau attack people deaf old mother snow geese spring purple burnt silver Sister silver Butterfly kai Luan Yuan Cai Choi son Choi Fung Colorful color cai brother Cai Luan Choi Wives Choi son niang rosy clouds mom long son chi Xia Qi official son amber beautiful son Xi Luan gourd monk Shining Shanyu letter Hanchiqu official black son jingtian hou hoe Medicine wisdom Intelligent Wisdom Son Shong Gucheng test Fu Qiufang Jiao da Guangdong general baking tea Good sister good funny portrayed lame foot among insects real wa Huang Mei wisp son Richuria grandma Shan Congfei zhenshiyin lady Shong Shing Zhangbaoyu Shang Shang son Shang Lady Rong elder brother Rong elder daughter-in-law Rais-Ray-Darais Raichangron Lai sister-in-law Lai two family of the Lackey Grandma Lai big daughter-in-law Chuliang Chu Shian Jiutai age official deformity man silly elder sister silly Uncle Silly elder sister Mama Jin Township Houjin Hou chiming Gianguang Jianziliang Bau Medical abalone two abalone two daughter-in-law crazy Dream Goddess Green Moon Green Mark Jade static false sill outsiders fortune-teller Xiaoxiang concubine Purple Jade cloud cui wisp cui ink Juan Sister Juan Big Grandma Wai Banana under guest Rui Guan Rui Bead Drunk King Ink Rain zhuan son rice Pan You an pan three Bao Indigo xue Father Sherberscheven xue Aunt Hoki son parrot parrot mu Dai dai Liang's tan cloud Xiangyang Hou Lotus Lotus root of the head monk Caperidge Wu June Police magic Fairy Police illusion goddess frown son frown Qing

Novel text: DreamOfRedMansion.txt

hadoop-statistics The number of names appearing in a dream of red mansions