Hadoop大資料之第一個程式--WordCount

標籤：

Map類：

 1 package lyc.yushao.hadoop.mr.wc; 2  3 import java.io.IOException; 4  5 import org.apache.hadoop.io.LongWritable; 6 import org.apache.hadoop.io.Text; 7 import org.apache.hadoop.mapreduce.Mapper; 8  9 public class WCMapper extends Mapper<LongWritable, Text, Text, LongWritable> {10 11     @Override12     protected void map(LongWritable key, Text value, Context context)13             throws IOException, InterruptedException {14         // 首先，接收資料15         // accept data16         String line = value.toString();17         // 進行拆分18         // split19         String[] words = line.split(" ");20         // 進行迴圈21         // loop22         for (String w : words) {23             // 發送24             // send25             context.write(new Text(w), new LongWritable(1));26         }27 28     }29 30 }

Reduce類：

 1 package lyc.yushao.hadoop.mr.wc; 2  3 import java.io.IOException; 4  5 import org.apache.hadoop.io.LongWritable; 6 import org.apache.hadoop.io.Text; 7 import org.apache.hadoop.mapreduce.Reducer; 8  9 public class WCReducer extends Reducer<Text, LongWritable, Text, LongWritable> {10 11     @Override12     protected void reduce(Text key, Iterable<LongWritable> values,13             Context context) throws IOException, InterruptedException {14         // 定義一個計數器15         // define a counter16         long counter = 0;17 18         // 接收資料 迴圈19         // accept data and loop20         for (LongWritable i : values) {21             // sum22             counter += i.get();23 24         }25         // send26         context.write(key, new LongWritable(counter));27 28     }29 30 }

 

WordCount類：

 1 package lyc.yushao.hadoop.mr.wc; 2  3 import org.apache.hadoop.conf.Configuration; 4 import org.apache.hadoop.fs.Path; 5 import org.apache.hadoop.io.LongWritable; 6 import org.apache.hadoop.io.Text; 7 import org.apache.hadoop.mapreduce.Job; 8 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 9 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;10 11 public class WordCount {12 13     public static void main(String[] args) throws Exception {14         Configuration conf = new Configuration();15         // 將mapreduce抽象成一個作業16         Job job = Job.getInstance(conf);17 18         // notice!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!19         job.setJarByClass(WordCount.class);20 21         // 將自訂的類組裝起來22 23         // set mapper‘s properties24         job.setMapperClass(WCMapper.class);25 26         job.setMapOutputKeyClass(Text.class);27 28         job.setMapOutputValueClass(LongWritable.class);29 30         // 讀取HDFS資料31         FileInputFormat.setInputPaths(job, new Path("/words.txt"));32 33         // set reducer‘s properties34         job.setReducerClass(WCReducer.class);35         // 輸出到HDFS裡面36         job.setOutputKeyClass(Text.class);37         job.setOutputValueClass(LongWritable.class);38         FileOutputFormat.setOutputPath(job, new Path("/wcout111"));39 40         // 調用job的一些方法來提交41         // submit,but this is not good42         // job.submit();43         job.waitForCompletion(true);44     }45 46 }

 

右鍵：工程名

Export

JAR File

 

 

Finish

命令列中：

hadoop jar /root/mrs.jar 

 

SUCCESS！！

Hadoop大資料之第一個程式--WordCount