Hadoop custom Jobtracker and Namenode administration pages

To make it easy to customize the presentation in the management interface of Hadoop (Namenode and Jobtracker), the management interface of Hadoop is implemented using proxy servlet.

First of all

The constructors in Org.apache.hadoop.http.HttpServer public httpserver (string name, string bindaddress, int Port,boolean findport, Configuration conf, accesscontrollist adminsacl,connector Connector), add the following code to specify the resource bundle and URL prefix:

Servletholder Servlethoder = new Servletholder (); Servlethoder.setinitparameter (" Com.sun.jersey.config.property.packages "," com.test.agent "); Servlethoder.setservlet (new Com.sun.jersey.spi.container.servlet.ServletContainer ()); Webappcontext.addservlet (Servlethoder, "/agent/*");

After that, an abstract object is defined in the package Abstractresource

public class Abstractresource {protected Jsonmapper jsonmapper = Jsonmapper.nondefaultmapper ();p rotected @ Contextservletcontext context;public jsonmapper Getjsonmapper () {return jsonmapper;} public void Setjsonmapper (Jsonmapper jsonmapper) {this.jsonmapper = Jsonmapper;} Public ServletContext GetContext () {return context;} public void SetContext (ServletContext context) {This.context = context;}}

Then, the real resource code implements the Abstractresource object and defines the resource

@Path ("/job") public class Jobresource extends Abstractresource {private DecimalFormat Percentformat = new DecimalFormat ( "# #0.");p rivate simpledateformat dateformat = new SimpleDateFormat ("d-mmm-yyyy HH:mm:ss"); @Path ("/clustersummary") @GET @consumes (Mediatype.text_plain) @Produces (mediatype.text_plain) public String Getclustersummaryinfo () { Jobtracker JT = (jobtracker) context.getattribute ("Job.tracker"); Clustermetrics metrics = Jt.getclustermetrics (); String Taskspernode = metrics.gettasktrackercount () > 0? Percentformat.format (((Double) (metrics.getmapslotcapacity () + metrics.getreduceslotcapacity ()))/ Metrics.gettasktrackercount ()): "-"; map<string, string> summaryinfo = new linkedhashmap<string, string> (); Summaryinfo.put (" Usedheapmemorybytes ", Long.tostring (Runtime.getruntime () totalmemory ()); Summaryinfo.put (" TotalHeapMemoryBytes " , Long.tostring (Runtime.getruntime (). MaxMemory ())), Summaryinfo.put ("Runningmaptasks", integer.tostring ( Metrics.getrunningmaps ())); SUMMARYINFO.PUT ("Runningreducetasks", Integer.tostring (Metrics.getrunningreduces ())) Summaryinfo.put ("Totaljobsubmissions", Integer.tostring (Metrics.gettotaljobsubmissions ())); Summaryinfo.put ("Numtotaltasktrackers", Integer.toString ( Metrics.gettasktrackercount ())); Summaryinfo.put ("Occupiedmapslots", integer.tostring ( Metrics.getoccupiedmapslots ())); Summaryinfo.put ("Occupiedreduceslots", integer.tostring ( Metrics.getoccupiedreduceslots ())); Summaryinfo.put ("Reservedmapslots", integer.tostring ( Metrics.getreservedmapslots ())); Summaryinfo.put ("Reservedreduceslots", integer.tostring ( Metrics.getreservedreduceslots ())); Summaryinfo.put ("Maptaskcapacity", integer.tostring ( Metrics.getmapslotcapacity ())); Summaryinfo.put ("Reducetaskcapacity", integer.tostring ( Metrics.getreduceslotcapacity ())); Summaryinfo.put ("Avgtaskspertasktracker", Taskspernode); SummaryInfo.put (" Numblacklistedtasktrackers ", Integer.tostring (Metrics.getblacklistedtasktrackercount ())); SummaryInfo.put (" Numgraylistedtasktrackers ", Integer.tostring (Metrics.getGraylistedtasktrackercount ())); Summaryinfo.put ("Numdecommissionedtasktrackers", integer.tostring ( Metrics.getdecommissionedtasktrackercount ())); return Jsonmapper.tojson (SummaryInfo);}}

This can be accessed by accessing the URL, and the custom management interface accesses the same port as the original management page port. Jobtracker use 50030,namenode 50070, and so on. See the following unit test code:

public class Jobresourceinttest {HttpClient client; @Beforepublic void SetUp () { Client = new Defaulthttpclient ();} private string urlprefix = "http://ip:50030/agent/job/";p rivate string jobId = "job_201405121106_0001";p rivate string Tipid = "task_201405121106_0001_m_000000";p rivate String attemptid = "Attempt_201405121106_0001_m_000000_0";p rivate String regex = "\" total\ ": \" (. +?) \ ",";p rivate Pattern p = pattern.compile (regex); @Testpublic void Testclustersummary () {HttpGet clustersummary = new HTTPG ET (urlprefix + "clustersummary"); try {httpresponse response = Client.execute (clustersummary); String clustersummarystr = entityutils.tostring (Response.getentity ()); System.out.println (CLUSTERSUMMARYSTR); Assert.asserttrue (Clustersummarystr.contains ("usedheapmemorybytes")); Assert.asserttrue (Clustersummarystr.contains ("Runningmaptasks"));} catch (Exception e) {e.printstacktrace ();} Clustersummary.releaseconnection ();} @Afterpublic void TearDown () {}} 

The above is the backend implementation and unit testing, the front end can be based on the need for resource acquisition and interface design.

Hadoop custom Jobtracker and Namenode administration pages