Job conf XML file for MapReduce job on Yarn with job history Server's Web Console

Many times, Yarn users want to know the running parameters of a mapreduce job that they have run, and the Conf XML file contents of the job can be viewed from the Web console of the MapReduce history server. Of course, users can also log in to Yarn's Web console address, and then jump from above to the Job history Server Web console for review. This article will demonstrate this feature in a simple graphic example.

Steps:

1, before starting the Job history Server, in the Mapred-site.xml file, the relevant parameters are set, as follows:

<property>
<name>mapreduce.jobhistory.address</name>
<value>hostname:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hostname:19888</value>
</property>
2. Start the Job history Server by ordering "sbin/mr-jobhistory-daemon.sh start Historyserver"

3. Perform a simple MapReduce job, such as WordCount. When executed, the default setting is used without any parameter adjustments.

4, after the execution of direct login to the job History Server Web console, the URL is "Http://hostname:19888/jobhistory"

You can see the list of jobs that have been executed on the Web page.

5, click the job ID number (such as job_1417166623034_0343), enter the specific job description page

6. Click on the "Configuration" link to view the job Conf XML file contents of the MapReduce job.

7, filter to see the value of the parameter ' mapreduce.reduce.shuffle.input.buffer.percent '-The default value of "0.70":

8. Commit to execute a new MapReduce job and specify the value of the parameter ' mapreduce.reduce.shuffle.input.buffer.percent ' as "0.69": "Hadoop jar Hadoop-examples-2.2.0.jar wordcount-dmapreduce.reduce.shuffle.input.buffer.percent=0.69/tmp/wdinput/tmp/ Wdoutput ".

9. After executing the Mr Job, log in to the job history Server's Web console again, and then look at the configuration content of the job-at this point, its parameters ' The value of Mapreduce.reduce.shuffle.input.buffer.percent ' is "0.69". It is proved that the new parameter value of MR job is successfully passed through "-dmapreduce.reduce.shuffle.input.buffer.percent=0.69".

10, in addition, in addition to directly log into the job history server's Web console, users can also log in to Yarn's Web console address, and then jump from above to the Job History Server Web console for review:

Job conf XML file for MapReduce job on Yarn with job history Server's Web Console