Source-level analysis of MapReduce job initialization in Jobtracker

MapReduce Job submission process source level analysis (c) has already explained that the user finally calls the Jobtracker.submitjob method to submit the job to Jobtracker. The core submission method for this method is the Jobtracker.addjob (Jobid Jobid, Jobinprogress Job) method, which submits the job to the scheduler (AddJob by default) Listener Jobqueuejobinprogresslistener and Eagertaskinitializationlistener (this article only discusses the default scheduler), use the method jobadded (jobinprogress job), The Jobqueuejobinprogresslistener task is to monitor the changes in the lifecycle of each jobinprocess, and Eagertaskinitializationlistener to initialize the new job after it is discovered.

First, jobqueuejobinprogresslistener.jobadded (jobinprogress Job) method. For a code jobqueue.put (new Jobschedulinginfo (Job.getstatus ()), job, build a Jobschedulinginfo object first, Then it corresponds to the jobinprogress and puts it into the jobqueue. The Jobschedulinginfo class maintains some of the information that this job must have, such as priority (the default is normal), Jobid, and start-time starttime.

Second, eagertaskinitializationlistener.jobadded (jobinprogress Job) method.

/**
   * We Add the JIP to the Jobinitqueue, which are processed 
   * asynchronously to handle split-computation and Buil D up
   * the right tasktracker/block mapping.
   */
  @Override public
  void jobadded (jobinprogress job) {
    synchronized (jobinitqueue) {
      Jobinitqueue.add (Job); Add into list<jobinprogress> jobinitqueue
      resortinitqueue ();
      Jobinitqueue.notifyall (); Wake blocked Process
    }
    
  }

The Resortinitqueue () method in the above method is primarily to sort the jobinprocess in Jobinitqueue, first by priority, and then by the start time. Eagertaskinitializationlistener.start () is invoked when the scheduler Initializes Jobqueuetaskscheduler.start (), so it is called before the jobadded method. The Eagertaskinitializationlistener.start () code is as follows:

1 public void Start () throws IOException {
2 This.jobinitmanagerthread = new Thread (Jobinitmanager, "Jobinitmanager");
3 Jobinitmanagerthread.setdaemon (TRUE);
4 This.jobInitManagerThread.start ();
5}

The

Start () method starts a thread: Jobinitmanager.

Used to init new jobs that have just been created
       
    Class Jobinitmanager implements Runnable {
      public void Run () {jobinprogress job = null;
              while (true) {The try {synchronized (jobinitqueue) {while (Jobinitqueue.isempty ()) {
            Jobinitqueue.wait ();
          Job = jobinitqueue.remove (0);
        } threadpool.execute (New Initjob (Job));
          catch (Interruptedexception t) {log.info ("Jobinitmanagerthread interrupted.");
        Break
      } log.info ("Shutting down thread pool");
    Threadpool.shutdownnow ();
        
    } class Initjob implements Runnable {private jobinprogress job;
    Public initjob (jobinprogress job) {this.job = job; public void Run () {Ttm.iniTjob (Job)///corresponding Jobtracker corresponding method}} 

Back to the column page: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/