"Big Data series" Under Windows Connect Linux environment development

One, configuration file 1.core-site.xml

<configuration>   <property>     <name>fs.defaultFS</name>     <value>hdfs: // www.node1.com:9000</value>   </property></configuration>

2, Hdfs-site.xml

<configuration>  <property>    <name>dfs.replication</name>    <value>2< /value>  </property></configuration>

3, Yarn-site.xml

<property><name>yarn.resourcemanager.hostname</name><value>www.node1.com</value> </property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_ Shuffle</value></property></configuration>

4, Slaves

Www.node2.comwww.node3.com

Second, establish a local connection three, create MapReduceProject1, file--New-other--mapreduceproject 2, set up test files

Importjava.io.IOException;ImportJava.util.StringTokenizer;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;Importorg.apache.hadoop.io.IntWritable;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.Mapper;ImportOrg.apache.hadoop.mapreduce.Reducer;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser; Public classWordCount { Public Static classTokenizermapperextendsMapper<object, text, text, intwritable>{    Private Final StaticIntwritable one =NewIntwritable (1); PrivateText Word =NewText ();  Public voidmap (Object key, Text value, context context)throwsIOException, interruptedexception {stringtokenizer ITR=NewStringTokenizer (value.tostring ());  while(Itr.hasmoretokens ()) {Word.set (Itr.nexttoken ());    Context.write (Word, one); }    }    }     Public Static classIntsumreducerextendsReducer<text,intwritable,text,intwritable> {    Privateintwritable result =Newintwritable ();  Public voidReduce (Text key, iterable<intwritable>values, context context)throwsIOException, interruptedexception {intsum = 0;  for(intwritable val:values) {sum+=Val.get ();    } result.set (sum);    Context.write (key, result); }    }     Public Static voidMain (string[] args)throwsException {Configuration conf=NewConfiguration (); String[] Otherargs=Newgenericoptionsparser (conf, args). Getremainingargs (); if(Otherargs.length! = 2) {System.err.println (otherargs.length); System.err.println ("Usage:wordcount <in> <out>"); System.exit (2); } System.out.println (otherargs[0]); System.out.println (otherargs[1]); Job Job=NewJob (conf, "word count"); Job.setjarbyclass (WordCount.class); Job.setmapperclass (tokenizermapper.class); Job.setcombinerclass (intsumreducer.class); Job.setreducerclass (intsumreducer.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (intwritable.class); Fileinputformat.addinputpath (Job,NewPath (otherargs[0])); Fileoutputformat.setoutputpath (Job,NewPath (otherargs[1])); System.exit (Job.waitforcompletion (true) ? 0:1); }}

3. Run Configuration

HDFs://WWW.NODE1.COM:9000/USR/WCHDFs://www.node1.com:9000/usr/wc/output

4. Run

5, part-r-00000

Apple    2banana    1cat    1dog    1hadoop    1hadpp    1 Hello     1mapreduce    1    name    1 World 1yarn    2

6, Wc.txt

Hadoop hellohadpp worldapple Dogbanana catmapreduce Nameyarnappleyarn

"Big Data series" Under Windows Connect Linux environment development