Mapreduce debugging query system. Out results

1. Preface

People who are new to mapreduce must have encountered such a problem:ProgramThe following command

System. Out. println (Year + "" + airtemperature); // invalid, no output on the console.However, the console does not output the corresponding results, which is a headache for many people who debug through system. Out. When I read chapter 5 of the hadoop authoritative guide version 2 yesterday, the book describes how to browse hadoop job information through the Web interface, you can see information about many jobs on the Web interface. The mapreduce job information is written in user logs and stored in the hadoop_home/logs/userlogs directory. For other log storage locations, see Table 5-2 of p152 in hadoop authoritative guide version 2. You can find these logs on the Web interface.

2. Provide the test program's Code

Newmaxtemperature. Java

Package hadoop. chapter2; </P> <p> // CC newmaxtemperature application to find the maximum temperature in the weather dataset using the new context objects mapreduce API <br/> Import Java. io. ioexception; </P> <p> Import Org. apache. hadoop. FS. path; <br/> Import Org. apache. hadoop. io. *; <br/> Import Org. apache. hadoop. mapreduce. *; <br/> Import Org. apache. hadoop. mapreduce. lib. input. fileinputformat; <br/> Import Org. Apache. hadoop. mapreduce. lib. output. fileoutputformat; </P> <p> // VV newmaxtemperature <br/> public class newmaxtemperature {</P> <p> static class newmaxtemperaturemapper <br/> extends mapper <longwritable, text, text, intwritable >{</P> <p> Private Static final int missing = 9999; </P> <p> Public void map (longwritable key, text value, context context) <br/> throws ioexception, interruptedexception {</P> <p> String line = value. tostring (); <br/> string year = line. substring (15, 19); <br/> int airtemperature; <br/> If (line. charat (87) = '+') {// parseint doesn' t like leading plus signs <br/> airtemperature = integer. parseint (line. substring (88, 92); <br/>} else {<br/> airtemperature = integer. parseint (line. substring (87, 92); <br/>}< br/> string Quality = line. substring (92, 93); <br/> If (airtemp Erature! = Missing & quality. matches ("[01459]") {<br/> context. write (new text (year), new intwritable (airtemperature); <br/> system. out. println (Year + "" + airtemperature); // invalid, no output in the console. <Br/>/* <br/> * stdout logs <br/> * 19500 <br/> * 195022 <br/> * 1950-11 <br/> * 1949111 <br/> * 194978 <br/> **/<br/>}</P> <p> static class newmaxtemperaturereducer <br/> extends CER <text, intwritable, text, intwritable >{</P> <p> Public void reduce (Text key, iterable <intwritable> values, <br/> context) <br/> throws ioexception, interruptedexception {</P> <p> int maxv Alue = integer. min_value; <br/> for (intwritable value: values) {<br/> maxvalue = math. max (maxvalue, value. get (); <br/> system. out. println (Key + "" + value. get (); // invalid, no output from the console. <Br/>/* <br/> * stdout logs <br/> * 1949111 <br/> * 194978 <br/> * 19500 <br/> * 195022 <br/> * 1950-11 <br/> **/<br/>}< br/> context. write (Key, new intwritable (maxvalue); <br/>}</P> <p> Public static void main (string [] ARGs) throws exception {<br/> If (ARGs. length! = 2) {<br/> system. err. println ("Usage: newmaxtemperature <input path> <output path>"); <br/> system. exit (-1 ); <br/>}< br/> // home/hadoop/input/sample.txt/home/hadoop/output/tmp1 <br/> job = new job (); <br/> job. setjarbyclass (newmaxtemperature. class); </P> <p> fileinputformat. addinputpath (job, new path (ARGs [0]); <br/> fileoutputformat. setoutputpath (job, new path (ARGs [1]); </P> <p> job. setmapp Erclass (newmaxtemperaturemapper. class); <br/> job. setreducerclass (newmaxtemperaturereducer. class); </P> <p> job. setoutputkeyclass (text. class); <br/> job. setoutputvalueclass (intwritable. class); </P> <p> system. exit (job. waitforcompletion (true )? 0: 1); <br/>}< br/> // ^ newmaxtemperature <br/>

3. then provide the data used to test the program.

Sample.txt

0067011990999991950051507004 + 68750 + 023550fm-12 + 00001 + 99999999999 <br/> 0043011990999991950051512004 + 68750 + 023550fm-12 + 00221 + 99999999999 + 0043011990999991950051518004 <br/> 68750 + 99999999999 + 023550fm-12 + <br/> 0043012650999991949032412004 + 62300 + 010750fm-12 + 048599999v0202701n00461220001cn0500001133 + 01111 + 99999999999 <br/> 0043012650999991949032418004 + 62300 + 010750fm-12 + 5E + 00781 + 99999999999

4. Finally, the input parameters of the program are given.

We run this program on Eclipse, not through the command line.

/Home/hadoop/input/sample.txt/home/hadoop/output/tmp1
5. Find the content of system. Out.

This isArticleIn the console, we do not find the output we want to print.

First, log on to the Web Console at http: // localhost: 50030 /. Here are some related web control interfaces:

Http: // localhost: 50030/-hadoop management interface <br/> http: // localhost: 50060/-hadoop task tracker status <br/> http: // localhost: 50070/-hadoop DFS status <br/>After logging on to the management interface, we can find the job we just run in completed jobs, as shown in:

We enter:Job_201110230923_0002:

Here we can see the details of the map task in reduce characters. First, click map To Go To The following interface:

Click task _ 201110230923_0002_m_000000 To Go To The following page:

Click All in the task logs column. The following page is displayed:

The above stdout logs is the content of system. Out. The reduce task can also obtain the content of system. Out in the same way. In fact, the content accessed on the Web management interface is written locally, which can be found in the local user log file. For example, the stdout logs in the preceding example can be found in the directory hadoop_home/logs/userlogs/attempt_201110230923_0002_m_000000_0. The directory contains the following files: log. Index stderr stdout syslog. The content in stderr isSystem. Err. printlnOutput.

6. Summary

Many related content such as job and task are mentioned in this Article. For details about these relations, refer to the article p147 of hadoop authoritative guide version 2.