Analysis of initialization process of client submitting jobs and jobs in JT

Job submission and initialization is mainly for the follow-up of the Mr Program to do the preparation work.

It is divided into four steps namely 1 configuration Mr Job environment, 2 upload job information, 3 submit job, 4 job initialization.

Here are four steps 1.1 to set up the job environment

This step is the configuration that we often use in Mr. Configure the parameters that guarantee the basic operation of Mr.

For example, WordCount is configured as follows:

 Configuration conf = new Configuration (); Job Job = new Job (conf); When our application is packaged into jars, it must have the line code Job.setjarbyclass (wordcountapp.class) when executed in Hadoop; Tell the job to execute our custom mapper class Job.setmapperclass (Wordcountmapper.class); Job.setcombinerclass (Intsumreducer.class); Job.setpartitionerclass (Hashpartitioner.class); Tell the job to execute our custom reducer class Job.setreducerclass (Wordcountreducer.class); Tell job,k3 what type of Job.setoutputkeyclass (Text.class); Tell job,v3 what type of Job.setoutputvalueclass (Intwritable.class); Tell the job the number of reduce that will be executed. Job.setnumreducetasks (4); Tell the job where the input source is; input can be multiple file Fileinputformat.addinputpath (Job, New Path (Baseconf.uri_hdfs_prefix +/test/wd) /input ")); Tell the job where the output path is; Fileoutputformat.setoutputpath (Job, New Path (Baseconf.uri_hdfs_prefix +/TEST/WD/OUTPU) T ")); Job.waitforcompletion (True); 

Of course, for a much-needed configuration of MR, the default configuration, which focuses on the set interface provided in Jobconf.java

Here focuses on the implementation process of the entire architecture to understand how it works.

The settings above are done through the job's member jobconf.

1.1.2 Preparation before submission

 public void Submit () throws IOException, Interruptedexception, CLASSNOTFO
    undexception {ensurestate (jobstate.define);//Confirm that the job is not running.  Setusenewapi (); Based on the configuration "Mapred.mapper.new-api" and "Mapred.reducer.new-api" settings, whether a new Mr API//Connect to the Jobtracker and submit the
    Job Connect ();
    info = jobclient.submitjobinternal (conf);
    Super.setjobid (Info.getid ());
   state = jobstate.running; 

1.1.3 connects to JT

     uses Java secure Access controller to check security policy to allow connection to JT.

     This will be done by creating a jobclient instance and connecting to JT via RPC.

jobsubmissionprotocolrpcjobsubmitclient = (jobsubmissionprotocol) rpc.getproxy (Jobsubmissionprotocol.class,... c1>1.2 upload job information

1> from JT gets the directory where the job is saved on HDFs

Path Jobstagingarea =jobsubmissionfiles.getstagingdir (Jobclient.this, jobcopy);

Default is ${"Mapreduce.jobtracker.staging.root.dir"}/tmp/hadoop/mapred/staging/${user}

2> assign Jobid.

Jobidjobid = Jobsubmitclient.getnewjobid ();

3> Specify Submit Directory

Pathsubmitjobdir = new Path (Jobstagingarea, jobid.tostring ()); Permission defaults to 700

4> Save job information to Submitjobdir

Copyandconfigurefiles (Jobcopy,submitjobdir);

In which the specified file is created and copied to the specified directory in Submitjobdir.

Submitjobdir/archives/the corresponding startup parameter-archives the specified file

Submitjobdir/files/the corresponding startup parameter-files the specified file.

submitjobdir/libjars/to save dependent third-party jar files.

Submitjobdir/job.jar corresponds to the MR Program jar file that will be run.

5> compute inputsplit Information and upload

Intmaps = writesplits (context, submitjobdir);

->inputformat<?,? > Input = reflectionutils.newinstance (Job.getinputformatclass (), Conf);

->list<inputsplit>splits = input.getsplits (Job);

-> jobsplitwriter.createsplitfiles (jobsubmitdir,conf, Jobsubmitdir.getfilesystem (conf), array);

Computes the Inputsplit information for the input file through the getsplits of the InputFormat class set in the configuration. and upload it to the following two files:

Submitjobdir/job.split

Submitjobdir/job.splitmetainfo

The number of maps is then returned.

The Splitmetainfo information contains the

1. Location of job.split files

2. Position of Inputsplit in Job.split file

3. Length of Inputsplit data

4. Inputsplit is in the host list.

6> exports the configuration information in jobconf to Submitjobdir/job.xml on the HDFs.

Jobcopy.writexml (out); 1.3 Submit Job 1.3.1 Submit Job

Commit the job to JT via JT on the client's proxy object.

Status= jobsubmitclient.submitjob (Jobid, submitjobdir.tostring (), jobcopy.getcredentials ());

When JT receives the client's submission job Submitjob executes the following figure:

1.3.1.1 Create jobinprogress monitoring Job

Jobinfo = new Jobinfo (Jobid, NewText (Ugi.getshortusername ()), NewPath (Jobsubmitdir));

Job = new Jobinprogress (this, this.conf,jobinfo, 0, TS);

At the beginning of the jobinprogress, the corresponding Conf object is created according to HDFs, and jobinprogress related information is set according to the Job.xml.

Several important collections are also created to save some job information when the job runs.

 //No need to consider the maptask of data locality,//If Maptask inputsplit.location[] is empty, data locality is not considered in this collection during task scheduling this.
     Nonlocalmaps = new linkedlist<taskinprogress> ();
     This is a sort of tip collection based on the number of failures, and a collation failcomparator is specified.
     This.failedmaps = new treeset<taskinprogress> (failcomparator);
     A task collection that is not yet running.
     This.nonlocalrunningmaps = new linkedhashset<taskinprogress> ();
     The Maptask set in operation, This.runningmapcache = new identityhashmap<node,set<taskinprogress>> ();
     The not-running reduce collection, sorted by Failcomparator (does this mean that failed reduce will rejoin the running reduce collection?)
     This.nonrunningreduces = newtreeset<taskinprogress> (failcomparator); The running Reducetask collection this.runningreduces = new Linkedhashset<taskinprogress> ();  

1.3.1.2 Check Memory requirements

Checkmemoryrequirements (Job);

Determine if the memory of the Configuration "Mapred.task.maxvmem" (default -1l) for Mr Programs exceeds the configured "Mapred.task.limit.maxvmem" that the cluster allows. 1.3.1.3 join the job to the queue.

       Status = AddJob (Jobid, job);
       Jobs.put (Job.getprofile (). Getjobid (), job);
       for (Jobinprogresslistener listener:jobinprogresslisteners) {
         listener.jobadded (Job);
       }

Add jobinprogress to JT's jobs map. Then notify the Task Scheduler

When the scheduler starts, it adds its own listeners to the listener queue of JT. When a job joins, all listeners in the queue are notified to join a job job. This tells the scheduler that a job is coming. 1.4 Job Initialization 1.4.1 Notification Scheduler

Here we take the Capacitytaskscheduler altimeter as an example, when the Capacitytaskscheduler Jobqueuesmanager listener receives the jobadded event,

   Add job to waiting queue. It'll end up in the right place and
   //based on priority.
   Queue.addwaitingjob (Job);
   Let scheduler know.
   scheduler.jobadded (Job);

1. The queue is determined by the configuration "Mapred.job.queue.name" of the Mr Program by adding the job to the waitingjobs mapping of the scheduler's specified queue capacityschedulerqueue. defaults to default.

2. Tell dispatcher to dispatch a job in the queue

   Inform the queue
     queue.jobadded (job);  The number of execution jobs submitted to the user in the queue plus 1.
     Setup Scheduler specific job information
     preinitializejob (Job);

Prepare for job initialization. This is mainly based on the configuration of the cluster "MAPRED.CLUSTER.MAP.MEMORY.MB" (the amount of memory per map slot Slotsizepermap) and the configuration of Mr Program "MAPRED.TASK.MAXVMEM" (The maximum amount of memory per task Getmemoryformaptask ()) determines how many slots each maptask will occupy the cluster.

the formula is :(int) (Math.ceil (float) getmemoryformaptask ()/(float) slotsizepermap)

Reducetask Ibid. use cluster's "MAPRED.CLUSTER.REDUCE.MEMORY.MB" Configuration 1.4.2 Scheduler Dispatch job initialization

Several background service threads are also started when the scheduler is started. One of the threads is a jobinitializationpoller thread, and the following is the execution of its Run method:

Jobinitializationpoller.run () Code:

while (running) {
       cleanupinitializedjobslist ()//clearing out those jobs that are in the running state from the initialization collection, or the job
       that has been completed Selectjobstoinitialize ();
       if (!this.isinterrupted ()) {
         thread.sleep (sleepinterval);//configure "Mapred.capacity-scheduler.init-poll-interval" Default is 3000.
       }
    }

1.4.2.1 Select the job to initialize.

Selectjobstoinitialize () Code:

For (String queue:jobQueueManager.getAllQueues ()) {
     arraylist<jobinprogress> jobstoinitialize = Getjobstoinitialize (queue);
     Jobinitializationthread t = threadstoqueuemap.get (queue);
     for (jobinprogress job:jobstoinitialize) {
       t.addjobstoqueue (queue, job);
     }
}

First, iterate through all the Jobinprogress instances in the queue queues in all clusters and find all the jobstatus.prep sets.

The thread jobinitializationthread that is used to initialize the job is then found (this thread is the boot thread corresponding to the queue specified in the Threadstoqueuemap map.) This is based on the Capacity-scheduler.xml configuration "Mapred.capacity-scheduler.init-worker-threads" at startup, and the default will be to open 5 threads to check the job to initialize for each queue. and initialize it.)

Note: When there are too many jobs in the cluster environment, you can increase this configuration with the right amount of thread to perform job initialization.

Finally, the uninitialized jobs corresponding to queue queues are added to the jobinitializationthread thread for one by one initialization.

Jobinitializationthread.run () Code:

public void Run () {while
     (startiniting) {
       initializejobs (); 
           Thread.Sleep (Sleepinterval); Configure "Mapred.capacity-scheduler.init-poll-interval" defaults to 3000.
     }
    }

The Initializejobs () iterates through all uninitialized job calls to JT's initjob to initialize the job.

        Setinitializingjob (Job);
           Ttm.initjob (Job); Tasktrackermanager is the Jobtracker.

1.4.2.2 Initialization Job

taskinprogress

The Taskinprogress class maintains all the information during a task run. In Hadoop, because a task can be inferred or rerun. So there will be multiple taskattempt, and at the same time there may be multiple tasks with the same task trying to execute simultaneously. These tasks are managed and tracked by the same Taskinprogress object.

Primary properties when initializing it:

     Privatefinal Tasksplitmetainfo Splitinfo; Maptask the split information to be processed privatejobinprogress job;
     The jobinprogress of taskinprogress.
     The mapping relationship between the running TaskID and the Tasktrackerid.
     Privatetreemap<taskattemptid, string> activetasks = Newtreemap<taskattemptid, String> ();
     All Taskattemptid that have been run, including the completed and running.
     privatetreeset<taskattemptid> tasks = new treeset<taskattemptid> ();
     The mapping relationship between TaskID and TaskStatus.
     privatetreemap<taskattemptid,taskstatus> taskstatuses = newtreemap<taskattemptid,taskstatus> (); The corresponding relationship between Cleanuptaskid and Tasktracker privatetreemap<taskattemptid, string> cleanuptasks = newTreeMap<
     Taskattemptid, string> ();
     The list of task nodes that have failed to run privatetreeset<string> machineswherefailed = new treeset<string> ();
     List of tasks to be killed.
     Privatetreemap<taskattemptid, boolean> Taskstokill = Newtreemap<taskattemptid, Boolean> (); Waiting to be submitted to the taskattemp, Privatetaskattemptid tasktocommit;

Call JT's Initjob (Job) in the thread Jobinitializationthread run environment. There are two main things to do:

1 Initializing job-related tasks

Tasksplitmetainfo[] splits = Createsplits (Jobid);
Nummaptasks = splits.length;
Maps = new Taskinprogress[nummaptasks];
"Mapreduce.job.locality.wait.factor"
localitywaitfactor = Conf.getfloat (Locality_wait_factor, Default_ Locality_wait_factor);
This.reduces = new Taskinprogress[numreducetasks];
Start scheduling Reducetask Startup According to the configuration calculation when the Maptask completes a number of times. Default is 5%.
Completedmapsforreduceslowstart = Conf.getfloat ("Mapred.reduce.slowstart.completed.maps") * nummaptasks)
Cleanup = new Taskinprogress[2];
Setup = new Taskinprogress[2];
 

1.1MapTaskInprogress

Read the Splitmetainfo information from the job submission directory of the job and parse it into the associated tasksplitmetainfo[] array, and create a corresponding number of maptaskinprogress instances to distribute the split to prepare for execution. That is, specify the source data that it will process for each maptaskinprogress instance.

1.2 reducetaskinprogress

The reducetaskinprogress instance is then created according to Mr's configuration "Mapred.reduce.tasks".

1.3cleanupTaskInprogress and Setuptaskinprogress.

Cleanup and setup-type tasks are created for map and reduce, respectively.

2 notifies the scheduler to update the job.

Notifies the scheduler to update the job if the status is inconsistent before and after the initialization job-related task is judged.

Jobstatuschangeevent event = newjobstatuschangeevent (Job, eventtype.run_state_changed, Prevstatus,
              newStatus);
       Synchronized (jobtracker.this) {for
          (Jobinprogresslistener listener:jobinprogresslisteners) {
               Listener.jobupdated (event);
          }
       

The job that has not been scheduled by the scheduler is still prep state for the first initialization.