Hadoop Learning Notes (vii)--HADOOP weather data Run in the authoritative guide

1) HDFs File System Preparation work
A) # Hadoop fs–ls/user/root #查看hdfs文件系统
b) # Hadoop fs-rm/user/root/output02/part-r-00000
c) Delete the document, delete the folder
d) # Hadoop fs-rm–r/user/root/output02
e) # Hadoop fs–mkdir–p INPUT/NCDC
f) Unzip the input file and Hadoop does not recognize the. zip or. rar
g) # Hadoop Fs-put. /input_tmp/* INPUT/NCDC
h) The input file content format corresponding to the same task must be consistent.
2) Write the program
A) Maxtemperature.java

CC Maxtemperature application to find the maximum temperatureinchThe weather dataset//vv maxtemperatureimport org. Apache. Hadoop. FS. Path;import org. Apache. Hadoop. IO. Intwritable;import org. Apache. Hadoop(i). Text;import org. Apache. Hadoop. MapReduce. Job;import org. Apache. Hadoop. MapReduce. Lib. Input. Fileinputformat;import org. Apache. Hadoop. MapReduce. Lib. Output. Fileoutputformat;public class Maxtemperature {public static void main (string[] args) throws Exception {if (args. Length!=2) {System. Err. println("Usage:maxtemperature <input path> <output path>");System. Exit(-1);} Job Job = new Job ();Job. Setjarbyclass(maxtemperature. Class);Job. Setjobname("Max Temperature");Fileinputformat. Addinputpath(Job, New Path (args[0]));Fileoutputformat. Setoutputpath(Job, New Path (args[1]));Job. Setmapperclass(Maxtemperaturemapper. Class);Job. Setreducerclass(Maxtemperaturereducer. Class);Job. Setoutputkeyclass(Text. Class);Job. Setoutputvalueclass(intwritable. Class);System. Exit(Job. WaitForCompletion(true)?0:1);}}//^^ Maxtemperature

b) Maxtemperaturemapper.java

//CC Maxtemperaturemapper Mapper for maximum temperature example//VV MaxtemperaturemapperImport Java.io.ioexception;import Org.apache.hadoop.io.intwritable;import org.apache.hadoop.io.LongWritable; Import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.Mapper; Public  class maxtemperaturemapperextends Mapper<longwritable,  Text, text, intwritable> {    Private Static Final intMISSING =9999; @Override Public voidMap (longwritable key, Text value, Context context) throws IOException, interruptedexception {String line = value.    ToString (); String year = line.substring ( the, +);intAirtemperature;if(Line.charat ( the) ==' + ') {//parseint doesn ' t like leading plus signsAirtemperature = Integer.parseint (line.substring ( the, the)); }Else{airtemperature = Integer.parseint (line.substring ( the, the)); } String quality = Line.substring ( the, the);if(Airtemperature! = MISSING && quality.matches ("[01459]") {Context.write (NewText (year),NewIntwritable (airtemperature)); }  }}//^^ Maxtemperaturemapper

c) Maxtemperaturereducer.java

//CC maxtemperaturereducer Reducer for maximum temperature example//VV MaxtemperaturereducerImport Java.io.ioexception;import Org.apache.hadoop.io.intwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.Reducer; Public  class maxtemperaturereducerextends Reducer<Text,  Intwritable, Text, intwritable> {  @Override Public voidReduce (Text key, iterable<intwritable> values, context context) throws IOException, Interruptedexception {intMaxValue = Integer.min_value; for(Intwritable value:values)    {maxValue = Math.max (MaxValue, Value.get ()); } context.write (Key,NewIntwritable (MaxValue)); }}//^^ Maxtemperaturereducer

3) running on eclipse
A) Create the package ORG.HADOOP.NCDC and compile the above 3 Java classes.
b) Run As→java application→arguments
HDFS://MASTER:9000/USER/ROOT/INPUT/NCDC HDFS://MASTER:9000/USER/ROOT/OUTPUT/NCDC
c) # Hadoop Fs-ls OUTPUT/NCDC
d) # Hadoop FS Cat output/ncdc/part-r-00000
4) Javac mode execution
A) VI classpath.sh add

ExportHadoop_home=/usr/local/hadoop2.5Exportclasspath=.:/ Usr/local/jdk1.7/lib:/usr/local/jdk1.7/jre/lib forFinch $HADOOP _home/share/hadoop/common/hadoop-*.jar; Do     ExportClasspath=$CLASSPATH:$f Done forFinch $HADOOP _home/share/hadoop/hdfs/hadoop-*.jar; Do     ExportClasspath=$CLASSPATH:$f Done forFinch $HADOOP _home/share/hadoop/mapreduce/hadoop-*.jar; Do     ExportClasspath=$CLASSPATH:$f Done forFinch $HADOOP _home/share/hadoop/yarn/hadoop-*.jar; Do     ExportClasspath=$CLASSPATH:$f DoneExportClasspath=$CLASSPATH:$HADOOP _home/share/common/lib:$HADOOP _home/share/hdfs/lib:$HADOOP _home/share/mapreduce/lib:$HADOOP _home/share/tools/lib:$HADOOP _home/share/yarn/lib

b) cat/usr/local/hadoop2.5/classpath.sh>>/etc/profile
c) cat/usr/local/hadoop2.5/classpath.sh >>/usr/local/hadoop2.5/etc/hadoop/hadoop-env.sh
D) source/etc/profile
e) Restart the Hadoop service
f) # Javac maxtemperaturemapper.java-d.
Other classes, note that first compile the lowest class, compile the completed class file in the Java program's package path
g) # JAR-CVF Maxtemperature.jar org #打成jar包
h) # JAR-TVF Maxtemperature.jar #查看jar包目录结构
i) # Hadoop jar Maxtemperature.jar org/hadoop/ncdc/maxtemperature INPUT/NCDC OUTPUT/NCDC #运行jar包
Hadoop jar Package Name Program main class name Enter folder output folder
j) # Hadoop Fs-ls OUTPUT/NCDC
k) # Hadoop Fs-cat output/ncdc/part-r-00000
5) Eclipse output jar package runs
A) Right-click export→jar→ to generate the jar package
Due to the boast system, there is no need to add in the process of hitting the jar package. Classpath
b) The jar package can be configured for Classpath,linux.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Hadoop Learning Notes (vii)--HADOOP weather data Run in the authoritative guide