Compile makefile for hadoop mapreduce Program

Recently, mapreduce programs based on hadoop need to be integrated into a large framework written in C/C ++. During make, mapreduce applications must be automatically compiled and packaged. The following uses a simple wordcount1 as an example to describe the specific implementation details. Note: The hadoop version is 2.4.0.

The source code contains two files. One is wordcount1.java, which is the logic for implementing word count. The other is counterthread. Java, where the simple number of lines currently processed is calculated and printed. For the code, see Appendix 1. the key to writing makefile is to load all the jar package paths provided by hadoop. As you can see, many documents on the Internet have implemented a script to put all the files under the hadoop directory. it is too troublesome to put the JAR file in a path and compile it. Of course there are some simple methods, but they are all old hadoop versions such as 0.20.

In fact, hadoop provides a command hadoop classpath to obtain the path containing all jar packages. therefore, you only need to use javac-classpath "'hadoop classpath '"*. java, and then use jar-CVF to package the class file. The specific makefile code is as follows:

SRC_DIR = src/mypackage/*.java CLASS_DIR = binTARGET_JAR = WordCountall:$(TARGET_JAR)$(TARGET_JAR): $(SRC_DIR) mkdir -p $(CLASS_DIR)#javac -classpath `$(HADOOP) classpath` -d $(CLASS_DIR) $(SRC_DIR) javac -classpath "`hadoop classpath`" src/mypackage/*.java -d $(CLASS_DIR) -Xlintjar -cvf $(TARGET_JAR).jar -C $(CLASS_DIR) ./ clean: rm -rf $(CLASS_DIR) *.jar

Make:

[Email protected]: wordcount1 $ makemkdir-P binjavac-classpath "'hadoop classpath'" src/mypackage /*. java-D bin-xlintwarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/common/lib/jaxb-api.jar": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/common/lib/activation. jar ": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/common/lib/jsr173_1.0_api.jar": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/common/lib/jaxb1-impl.jar": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/yarn/lib/jaxb-api.jar": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/yarn/lib/activation. jar ": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/yarn/lib/jsr173_1.0_api.jar": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/share/hadoop/yarn/lib/jaxb1-impl.jar": no such file or directorywarning: [path] Bad path element "/home/Lichao/software/hadoop-src/hadoop-2.4.0-src/hadoop-Dist/target/hadoop-2.4.0/contrib/Capacity-schedity /*. jar ": no such file or directorysrc/mypackage/wordcount1.java: 61: Warning: [deprecation] job (configuration, string) in job has been deprecatedjob job = new job (Conf, "wordcount1"); // create a new job ^ 10 warningsjar-CVF wordcount. jar-C bin. /added manifestadding: mypackage/(in = 0) (out = 0) (stored 0%) adding: mypackage/wordcount1.class (in = 1970) (out = 1037) (deflated 47%) adding: mypackage/counterthread. class (in = 1760) (out = 914) (deflated 48%) adding: mypackage/wordcount1 $ intsumreducer. class (in = 1762) (out = 749) (deflated 57%) adding: mypackage/wordcount1 $ tokenizermapper. class (in = 1759) (out = 762) (deflated 56%) adding: log4j. properties (in = 476) (out = 172) (deflated 63%)

Although there is a warning, the result is not affected. After compilation, let's perform a simple test.

Test data: While true; do seq 1 100000> tmpfile; done; Ctrl + c

Then put the data on HDFS, hadoop FS-put tmpfile/data/

Run the mapreduce program: hadoop jar wordcount. Jar mypackage/wordcount1/data/tmpfile/output2.

The effect is as follows:

14/07/15 13:26:01 warn util. nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable14/07/15 13:26:03 info client. rmproxy: connecting to ResourceManager at localhost/127.0.0.1: 803214/07/15 13:26:05 info input. fileinputformat: total input paths to process: 114/07/15 13:26:05 info mapreduce. jobsubmitter: Number of splits: 614/07/15 13:26:06 info mapreduce. jobsubmitter: submitting tokens for job: job_140542597558_000314/07/15 13:26:06 info impl. yarnclientimpl: submitted application application_140542597558_000314/07/15 13:26:06 info mapreduce. job: the URL to track the job: http: // Ubuntu: 8088/Proxy/application_140542597558_0003/14/07/15 13:26:06 info mapreduce. job: running job: job_140542597558_000314/07/15 13:26:20 info mapreduce. job: Job job_140542597558_0003 running in Uber mode: false14/07/15 13:26:20 info mapreduce. job: Map 0% reduce 0%/15 13:26:34 warn mapreduce. counters: group Org. apache. hadoop. mapred. task $ counter is deprecated. use Org. apache. hadoop. mapreduce. taskcounter instead: 014/07/15 13:26:48 info mapreduce. job: Map 2% reduce 0% input rows: 313847414/07/15 13:26:51 info mapreduce. job: Map 5% reduce 0% 14/07/15 13:26:54 info mapreduce. job: Map 6% reduce 0% 14/07/15 13:26:55 info mapreduce. job: Map 8% reduce 0% 14/07/15 13:26:57 info mapreduce. job: Map 9% reduce 0% 14/07/15 13:26:58 info mapreduce. job: Map 11% reduce 0% 14/07/15 13:27:00 info mapreduce. job: Map 12% reduce 0% 14/07/15 13:27:01 info mapreduce. job: Map 13% reduce 0% input rows: 2338359514/07/15 13:27:05 info mapreduce. job: Map 14% reduce 0% input rows: 2338359514/07/15 13:27:23 info mapreduce. job: Map 15% Reduce 0% 14/07/15 13:27:27 info mapreduce. job: Map 16% reduce 0% 14/07/15 13:27:28 info mapreduce. job: Map 18% reduce 0% 14/07/15 13:27:30 info mapreduce. job: Map 19% reduce 0% 14/07/15 13:27:31 info mapreduce. job: Map 21% reduce 0% 14/07/15 13:27:34 info mapreduce. job: Map 24% reduce 0% input rows: 3843030114/07/15 13:27:37 info mapreduce. job: Map 25% reduce 0% 14/07/15 13:27:40 info mapreduce. job: Map 26% reduce 0% input rows: 4282632214/07/15 13:27:57 info mapreduce. job: Map 27% reduce 0% 14/07/15 13:28:00 info mapreduce. job: Map 29% reduce 0% 14/07/15 13:28:02 info mapreduce. job: Map 30% reduce 0% 14/07/15 13:28:03 info mapreduce. job: Map 32% reduce 0% input rows: 5451353114/07/15 13:28:05 info mapreduce. job: Map 33% reduce 0% 14/07/15 13:28:06 info mapreduce. job: Map 34% reduce 0% 14/07/15 13:28:08 info mapreduce. job: Map 35% reduce 0% 14/07/15 13:28:09 info mapreduce. job: Map 36% reduce 0% input rows: 6095908114/07/15 13:28:22 info mapreduce. job: Map 42% reduce 0% 14/07/15 13:28:30 info mapreduce. job: Map 43% reduce 0% 14/07/15 13:28:31 info mapreduce. job: Map 44% reduce 0% 14/07/15 13:28:34 info mapreduce. job: Map 45% reduce 0% 14/07/15 13:28:35 info mapreduce. job: Map 46% reduce 0% input rows: 6993615914/07/15 13:28:37 info mapreduce. job: Map 47% reduce 0% 14/07/15 13:28:38 info mapreduce. job: Map 48% reduce 0% 14/07/15 13:28:41 info mapreduce. job: Map 49% reduce 0% 14/07/15 13:28:44 info mapreduce. job: Map 50% reduce 0% input rows: 7716046114/07/15 13:29:01 info mapreduce. job: Map 51% reduce 0% 14/07/15 13:29:04 info mapreduce. job: Map 52% reduce 0% 14/07/15 13:29:05 info mapreduce. job: Map 53% reduce 0% input rows: 8300037314/07/15 13:29:07 info mapreduce. job: Map 54% reduce 0% 14/07/15 13:29:09 info mapreduce. job: Map 55% reduce 0% 14/07/15 13:29:10 info mapreduce. job: Map 56% reduce 0% 14/07/15 13:29:13 info mapreduce. job: Map 57% reduce 0% 14/07/15 13:29:16 info mapreduce. job: Map 58% reduce 0% input rows: 9336176614/07/15 13:29:32 info mapreduce. job: Map 59% reduce 0% input rows: 9819469614/07/15 13:29:35 info mapreduce. job: Map 60% reduce 0% 14/07/15 13:29:37 info mapreduce. job: Map 61% reduce 0% 14/07/15 13:29:38 info mapreduce. job: Map 62% reduce 0% 14/07/15 13:29:40 info mapreduce. job: Map 63% reduce 0% 14/07/15 13:29:41 info mapreduce. job: Map 64% reduce 0% 14/07/15 13:29:44 info mapreduce. job: Map 65% reduce 0% 14/07/15 13:29:48 info mapreduce. job: Map 66% reduce 0% input rows: 10956218414/07/15 13:30:04 info mapreduce. job: Map 67% reduce 0% input rows: 11336281814/07/15 13:30:06 info mapreduce. job: Map 68% reduce 0% 14/07/15 13:30:08 info mapreduce. job: Map 69% reduce 0% 14/07/15 13:30:10 info mapreduce. job: Map 70% reduce 0% 14/07/15 13:30:12 info mapreduce. job: Map 71% reduce 0% 14/07/15 13:30:15 info mapreduce. job: Map 72% reduce 0% input rows: 12307411914/07/15 13:30:32 info mapreduce. job: Map 76% reduce 0% 14/07/15 13:30:33 info mapreduce. job: Map 80% reduce 0% 14/07/15 13:30:34 info mapreduce. job: Map 83% reduce 0% 14/07/15 13:30:35 info mapreduce. job: Map 84% reduce 0% input rows: 12307411914/07/15 13:30:37 info mapreduce. job: Map 89% reduce 0% 14/07/15 13:30:38 info mapreduce. job: Map 92% reduce 0% 14/07/15 13:30:39 info mapreduce. job: Map 95% reduce 0% 14/07/15 13:30:40 info mapreduce. job: Map 100% reduce 0% input rows: 12307411914/07/15 13:30:53 info mapreduce. job: Map 100% reduce 100% 14/07/15 13:30:53 info mapreduce. job: Job job_140542597558_0003 completed successfully14/07/15 13:30:53 info mapreduce. job: counters: 50 File System countersfile: number of bytes READ = 58256119 file: number of bytes written = 66039749 file: Number of read Operations = 0 file: number of large read Operations = 0 file: Number of write operations = 0 HDFS: number of bytes READ = 724520133 HDFS: number of bytes written = 1088895 HDFS: number of read Operations = 21 HDFS: Number of large read Operations = 0 HDFS: number of write operations = 2job counters killed map tasks = 2 launched map tasks = 8 launched reduce tasks = 1data-local map tasks = 8 total time spent by all maps in occupied slots (MS) = 1528715 total time spent by all reduces in occupied slots (MS) = 17508 total time spent by all MAP tasks (MS) = 1528715 total time spent by all reduce tasks (MS) = 17508 total vcore-seconds taken by all MAP tasks = 1528715 total vcore-seconds taken by all reduce tasks = 17508 total megabyte-seconds taken by all MAP tasks = 1565404160 total megabyte-seconds taken by all reduce tasks = export frameworkmap input records = 123074119map output records = export output bytes = 1216795535map output materialized bytes = 7133406 input split bytes = 594 combine input records = 127374119 combine output records = 4900000 reduce input groups = 100000 reduce shuffle bytes = 7133406 Reduce input records = 600000 reduce output records = 100000 spilled records = 5500000 shuffled maps = 6 failed shuffles = 0 merged map outputs = 6gc time elapsed (MS) = 12761cpu time spent (MS) = 1397060 physical memory (bytes) snapshot = 1797943296 virtual memory (bytes) snapshot = 5082316800 total committed heap usage (bytes) = 1398800384 shuffle errorsbad_id = 0 connection = 0io_error = 0wrong_length = 0wrong_map = 0wrong_reduce = 0 file input format counters bytes READ = 724519539 file output format counters bytes written = 1088895

Appendix 1: wordcount1.java and counterthread. Java code

// Wordcount1.java code

Package mypackage; import Java. io. ioexception; import Java. util. stringtokenizer; import Org. apache. hadoop. conf. configuration; import Org. apache. hadoop. FS. path; import Org. apache. hadoop. io. intwritable; import Org. apache. hadoop. io. text; import Org. apache. hadoop. mapreduce. job; import Org. apache. hadoop. mapreduce. mapper; import Org. apache. hadoop. mapreduce. reducer; import Org. apache. hadoop. mapreduce. lib. input. file Inputformat; import Org. apache. hadoop. mapreduce. lib. output. fileoutputformat; import Org. apache. hadoop. util. genericoptionsparser; public class wordcount1 {public static class tokenizermapper extends mapper <object, text, text, intwritable> {private final static intwritable one = new intwritable (1 ); // create the "int" type variable one. The initial value is 1 private text word = new text (); // create the "string: type variable word, used to receive the input word public void map (Object K Ey, text value, context) throws ioexception, interruptedexception {stringtokenizer itr = new stringtokenizer (value. tostring (); // segment the input text by line while (itr. hasmoretokens () {word. set (itr. nexttoken (); // assign context to word. write (word, one); // pass the key-Value Pair word one} // system. out. println ("read lines:" + context. getcounter ("org. apache. hadoop. mapred. task $ counter "," map_input_records "). getvalue (); // system. ou T. println ("Number of input rows:" + context. getcounters (). findcounter ("org. apache. hadoop. mapred. task $ counter "," map_input_records "). getvalue (); // system. out. println ("Number of input rows:" + context. getcounters (). findcounter ("", "map_input_records "). getvalue () ;}} public static class intsumreducer extends CER <text, intwritable, text, intwritable> {private intwritable result = new intwritable (); // create the integer variable resultpublic void red Uce (Text key, iterable <intwritable> values, context) throws ioexception, interruptedexception {int sum = 0; // create the int variable sum Initial Value 0for (intwritable VAL: values) {sum + = Val. get (); // convert all value classes corresponding to each key} result. set (SUM); // sum is used to input result context. write (Key, result); // pass the key-result pair} public static void main (string [] ARGs) throws exception {configuration conf = new configuration (); // string [] newargs = New string [] {"HDFS: // localhost: 9000/data/tmpfile", "HDFS: // localhost: 9000/data/wc_output "}; string [] otherargs = new genericoptionsparser (Conf, argS ). getremainingargs (); If (otherargs. length! = 2) {system. err. println ("Usage: wordcount <in> <out>"); system. exit (2);} job = new job (Conf, "wordcount1"); // create a new jobjob. setjarbyclass (wordcount1.class); job. setmapperclass (tokenizermapper. class); // set the map class job. setcombinerclass (intsumreducer. class); // sets the combiner class job. setreducerclass (intsumreducer. class); // set the CER class job. setoutputkeyclass (text. class); // output key type job. setoutputvalueclass (intwritable. clas S); // output value type fileinputformat. addinputpath (job, new path (otherargs [0]); // input and output parameters (specified in settings) fileoutputformat. setoutputpath (job, new path (otherargs [1]); counterthread Ct = new counterthread (job); Ct. start (); job. waitforcompletion (true); system. exit (0); // system. exit (job. waitforcompletion (true )? 0: 1 );}}

// Counterthread. Java code

Package mypackage; import Java. lang. *; import Java. io. ioexception; import Java. util. stringtokenizer; import Org. apache. hadoop. conf. configuration; import Org. apache. hadoop. FS. path; import Org. apache. hadoop. io. intwritable; import Org. apache. hadoop. io. text; import Org. apache. hadoop. mapreduce. job; import Org. apache. hadoop. mapreduce. jobstatus; import Org. apache. hadoop. mapreduce. mapper; import Org. apache. hadoop. mapreduce. reducer; import Org. apache. hadoop. mapreduce. lib. input. fileinputformat; import Org. apache. hadoop. mapreduce. lib. output. fileoutputformat; import Org. apache. hadoop. util. genericoptionsparser; public class counterthread extends thread {public counterthread (job) {_ job = job;} public void run () {While (true) {try {thread. sleep (1000*5);} catch (interruptedexception E1) {// todo auto-generated catch blocke1.printstacktrace ();} Try {If (_ job. getstatus (). getstate () = jobstatus. state. running) // continue; system. out. println ("Number of input rows:" + _ job. getcounters (). findcounter ("org. apache. hadoop. mapred. task $ counter "," map_input_records "). getvalue ();} catch (ioexception e) {// todo auto-generated catch blocke. printstacktrace ();} catch (interruptedexception e) {// todo auto-generated catch blocke. printstacktrace () ;}} private job _ job ;}