"hadoop2.6.0" writing MapReduce in C + +

Hadoop uses Hadoop streaming to implement MapReduce code written in a non-Java language. For a little bit of Java is not going to me, this is really a great news.

Introduction to the official online Hadoop streaming in: http://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/ Hadoop-mapreduce-client-core/hadoopstreaming.html

We use the WordCount example to illustrate that the input file I use is downloaded from the Internet in the seventh Harry Potter English version, named H.txt

Write a map program in C + +, as long as you can read the information from the standard input, and can use the standard output to output <key, value> key value pairs on the line.

For the WordCount word count, the map program is very simple, as long as the output of each word in the back and then output a 1 line, indicating that each word appears 1 times

The Wordcount_map.cpp program is as follows:

#include <iostream>#include<string>using namespacestd;intMainintargcChar**argv) {    stringWord;  while(Cin >>word) {cout<< Word <<"/ T"<<"1"<<Endl; }    return 0;}

Reduce program to be able to read the map output key value pairs, and the key value (word) of the same key value pairs to do the integration, and output integration results

The Wordcount_reduce.cpp program is as follows:

#include <iostream>#include<string>#include<map>using namespacestd;intMainintargcChar**argv) {    stringkey, Num; Map<string,int>count; Map<string,int>:: Iterator it;  while(Cin >> Key >>num) {It=Count.find (key); if(It! =Count.end ()) {It->second++; }        Else{Count.insert (Make_pair (Key,1)); }    }     for(it = Count.begin (); It! = Count.end (); it++) {cout<< It->first <<"/ T"<< It->second <<Endl; }    return 0;}

Compile the two. cpp files into an executable file and place the two executables under the Hadoop root directory

g++-o mapperc wordcount_map.cppg+-o reducec wordcount_reduce.cpp

Uploading pending files H.txt to/user/kzy/input in HDFs

Bin/hdfs dfs-put h.txt  /user/kzy/input

To run Hadoop streaming requires Hadoop-streaming-2.6.0.jar, located in hadoop-2.6.0/share/hadoop/tools/lib/ Hadoop-streaming-2.6.0.jar START I can not run a variety of, is because the new version of the file location and the previous different.

Execute MapReduce, the options inside I don't fully understand, but this will work. Note that the-jobconf in the old version has been renamed-D.

Bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.6. 0. jar  \
  -D mapred.job.name="word count~"  -input/user/kzy/input/- output/user/output/c++_out  -mapper./-reducer./reducec  -file Mapperc  

View the results, sort in-K 2 for the second field to be used as the tab delimiter to sort-n means to sort by numerically-R to show results from large to small 20 rows

Bin/hadoop dfs-cat/user/output/c++_out/* | sort-k 2-n-r|head-20

The results are as follows:

"hadoop2.6.0" writing MapReduce in C + +