Source-level analysis of jobtracker start-up process

The Org.apache.hadoop.mapred.JobTracker class is a separate process with its own main function. Jobtracker is the core location for submitting and running Mr Tasks in a networked environment.

The main method has two major code lines:

Creating Jobtracker Objects

Jobtracker tracker = Starttracker (new jobconf ());

Start each service, including some important services or threads within JT

Tracker.offerservice ();

Starttracker (New jobconf ()) starts Jobtracker based on the configuration file, which invokes the Starttracker (Conf, Generatenewidentifier ()) method to initiate the operation. Generatenewidentifier () returns a string that is formatted as "YYYYMMDDHHMM" at the current time of the node.

The Starttracker function is a static function that calls the Jobtracker constructor to generate an instance of a Jobtracker class, whose main task is to initialize some important variables named result. Then, a series of initialization activities, including starting RPC server, starting the built-in jetty server, checking for the need to restart Jobtracker, and so on.
Important objects for initialization include:
1, Secretmanager:delegationtokensecretmanager case, Mr Safety management related classes;
2, aclsmanager:aclsmanager instance, job level and queue level management and access control;
3, the Taskscheduler:taskscheduler instance, the Scheduler object, the Hadoop default scheduler is the FIFO strategy Jobqueuetaskscheduler;
4. Example of Intertrackerserver:server, RPC Server;
5, infoserver:httpserver example, the job, Task, tasktracker related information to the Web front-end, the package is jetty;
6, recoverymanager:recoverymanager example, job recovery management, that is, Jobtracker startup, restore the last stop running the job, and restore the running status of individual tasks Recoverymanager.checkandaddjob (status) will check out those jobs that need to be restored and put into set<jobid> jobstorecover; Set of jobs to be recovered, for the back of the Recoverymanager.recover () to prepare;
7, jobhistoryserver:jobhistoryserver example, used to view the history of the operation of the server;
8, the dnstoswitchmapping:dnstoswitchmapping example, uses in constructs the cluster the network topology structure, it can map the node address (IP or host) to the network location.

Second, Tracker.offerservice ()

/** * Run Forever */public void Offerservice () throws Interruptedexception, IOException {//Prepare for Rec Overy.
    This are done irrespective the status of restart//flag.
        while (true) {try {recoverymanager.updaterestartcount ();
      Break
        catch (IOException IoE) {Log.warn ("Failed to initialize recovery manager.", IoE);
        Wait for some time thread.sleep (fs_access_retry_period);
      Log.warn ("retrying ...");
        
    } taskscheduler.start ();
    Start the recovery after starting the scheduler try {recoverymanager.recover (); The catch (Throwable t) {Log.warn ("Recovery manager crashed!
    Ignoring. ", t);
    }//Refresh the node list as the recovery manager might have added//disallowed trackers ();
                                          For discovering and clearing dead tasktracker this.expiretrackersthread = new Thread (This.expiretrackers, "ExpiretRackers ");
    This.expireTrackersThread.start ();
    Used to clean up long-running job information residing in memory This.retirejobsthread = new Thread (this.retirejobs, "retirejobs");
    This.retireJobsThread.start ();
    
    Used to discover tasks that have been assigned to a tasktracker but have not reported information Expirelaunchingtaskthread.start ();
                                            if (completedjobstatusstore.isactive ()) {completedjobsstorethread = new Thread (Completedjobstatusstore,
      "Completedjobsstore-housekeeper");
      Saves the completed job run information to HDFs and provides a set of APIs to access the information.
    Completedjobsstorethread.start ();
        
    }//Start the Inter-tracker server once the JT is ready This.interTrackerServer.start ();
    Synchronized (this) {state = state.running;
        
    } log.info ("Starting RUNNING");
    This.interTrackerServer.join ();
  Log.info ("Stopped intertrackerserver"); }

Back to the column page: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/