Hadoop collects job execution status information

Hadoop collects job execution status information. A project needs to collect information about the execution status of hadoop jobs. I have provided the following solutions:
1. obtain the required information from jobtracker. jsp provided by hadoop. One problem encountered here is that the application scope is used.
Jobtracker tracker = (jobtracker) application. getattribute ("job. Tracker ");
The jetty server is embedded in hadoop,
Org. Apache. mapred. jobtracker. Java
 

    Inetsocketaddress infosocaddr = netutils. createsocketaddr (

              Conf. Get (jt_http_address, "0.0.0.0: 50030 "));

              Infoserver = new httpserver ("job", infobindaddress,
Tmpinfoport,

              Tmpinfoport = 0, conf );

    Infoserver. setattribute ("job. Tracker", this );

Therefore, if you want to obtain statistics through the JSP page, you must bypass the jetty server, or return a reference of infoserver in the jobtracker modification, which is implemented in the code, however, it is clear that the core code of hadoop needs to be modified, and the flexibility is not high.

2. parsing JSP scripts.

You can see through wget http: // localhost: 50030/jobtracker. jsp:


Bytes -----------------------------------------------------------------------------------------------

<B> state: </B> running <br>

<B> started:</B> Tue Dec 28 09:43:40 CST 2010 <br>

<B> Version: </B> 0.21.0,

                              985326 <br>

<B> compiled: </B> Tue Aug 17 01:02:28 EDT 2010

                                Tomwhite
From

                                Branches/branch-0.21 <br>

<B> identifier: </B> 201012280943 <br>                              

........................................ ........................................ ...............

All of this information can be parsed using Python beautiful Soup (http://www.crummy.com/software/BeautifulSoup.


3. Upgrade Your hadooop to Hadoop-0.21.0, cluster class

Provides a wide range of Apis
For example, to print the job information, you only need:
Configuration conf = new configuration ();
     Cluster
Cluster = new cluster (CONF );
     Job []
Job = cluster. getalljobs ();
     If (job
! = NULL ){
       For
(Job TMP: Job ){
         System. Out. println (TMP. getjobid ());
         System. Out. println (TMP. getjobname ());
         System. Out. println (TMP. getstarttime ());
         System. Out. println (TMP. getfinishtime ());
       }
     }