Jobsubmitter of MapReduce Source Analysis (I.)

Jobsubmitter, as the name implies, is the job submitter in MapReduce, and in fact Jobsubmitter except the constructor method, the only non-private member variable or method provided externally is the submitjobinternal () method, It is the internal method of submitting the job, which implements all the business logic for submitting the job. In this article, we will delve deeper into the component jobsubmitter for submitting jobs in MapReduce.

First, let's look at the Jobsubmitter class member variable, as follows:

  File system FileSystem Instance  private FileSystem jtfs;  Client communication Protocol ClientProtocol instance  private ClientProtocol submitclient;  The host name of the submission job is  private String submithostname;  The host address of the submission job is  private String submithostaddress;

It has a total of four class member variables, respectively:

1, File system filesystem instance Jtfs: For the operation of operations required to run a variety of files, etc.;

2, Client communication protocol ClientProtocol instance submitclient: Used to interact with the cluster, complete job submission, job status query, etc.

3, the host name of the submitted job submithostname;

4, the host address of the job submission submithostaddress.

wherein, the client communication protocol ClientProtocol instance submitclient is assigned by cluster Client communication protocol ClientProtocol instance clients, we MapReduce Source code Analysis of the new API job submission (ii): Connected to the cluster "in the article mentioned, It is based on the configuration of the parameter Mapreduce.framework.name in MapReduce as yarn or local, there are two cases of yarn mode Yarnrunner and local mode Localjobrunner.
Next, let's look at the Jobsubmitter constructor, as follows:

  Jobsubmitter (FileSystem submitfs, ClientProtocol submitclient)   throws IOException {  // According to the parameter assignment member variable submitclient, jtfs    this.submitclient = submitclient;    This.jtfs = Submitfs;  }

Very simple, according to the entry parameter assignment member variable submitclient, JTFS only.

The key comes, we look at Jobsubmitter the only external core function method Submitjobinternal (), which is used to submit jobs to the cluster, the code is as follows:

  /** * Internal method for submitting jobs to the system. * * <p>the Job submission process involves: * <ol> * <li> * Checking the input and output s   Pecifications of the job.   * </li> * <li> * Computing The {@link inputsplit}s for the job. * </li> * <li> * Setup the requisite accounting information for the * {@link Distributedcache}   Of the job, if necessary.  * </li> * <li> * Copying The job ' s jar and configuration to the Map-reduce system * directory on    The distributed File-system.   * </li> * <li> * Submitting the job to the <code>JobTracker</code> and optionally *   Monitoring it ' s status. * </li> * </ol></p> * @param job The configuration to submit * @param cluster the handle to the Cluster * @throws classnotfoundexception * @throws interruptedexception * @throws IOException */Jobstatus Submit JoBinternal (Job Job, Cluster Cluster) throws ClassNotFoundException, Interruptedexception, IOException {//validate the Jobs output Specs//Call the Checkspecs () method to verify that the job export path is configured and exists,//the correct situation should be configured and not present.    The output path of the WordCount job is configured to Hdfs://nameservice1/output/output before the output path configuration parameter is mapreduce.output.fileoutputformat.outputdir,//    Checkspecs (Job);        Obtaining configuration information from job Jobs conf Config conf = job.getconfiguration ();    Call the Addmrframeworktodistributedcache () method to add the application framework path to the distributed cache in Addmrframeworktodistributedcache (conf); Get the job execution Stage zone path Jobstagingarea//by Jobsubmissionfiles's Getstagingdir () static method Take the parameter yarn.app.mapreduce.am.staging-dir, the parameter is not configured by default to/tmp/hadoop-yarn/staging//and then the job user name is followed by/submitted/.staging// Through the execution of the previous WordCount task, we looked at the history, learned that the parameter Yarn.app.mapreduce.am.staging-dir configuration for/user,//and submitted the job user name is HDFs, so the complete path should be/user/        hdfs/.staging Path Jobstagingarea = Jobsubmissionfiles.getstagingdir (cluster, conf); Configure the command line options correctly on the submitting DFS//Get current native address inetaddress IP = InetaDdress.getlocalhost ();  Determine the host address, host name of the submission job, and set the in configuration information conf, corresponding parameters are//mapreduce.job.submithostname//mapreduce.job.submithostaddress if (IP      ! = null) {submithostaddress = Ip.gethostaddress ();      Submithostname = Ip.gethostname ();      Conf.set (Mrjobconfig.job_submithost,submithostname);    Conf.set (mrjobconfig.job_submithostaddr,submithostaddress);        }//Generate job ID, i.e. JobID instance JobID JobID JobID = Submitclient.getnewjobid ();        Set the jobId into job job.setjobid (JOBID); Constructs the submit job path instance Submitjobdir,jobstagingarea followed by/jobid, such as/job_1459913635503_0005//before the full path of the WordCount job is/user/hdfs/.    staging/job_1459913635503_0005 path Submitjobdir = new Path (Jobstagingarea, jobid.tostring ());        Jobstatus status = NULL; Set job Parameters: try {//Set Mapreduce.job.user.name as current user, previous WordCount example configured for HDFs user Conf.set (mrjobconfig.user_n            AME, Usergroupinformation.getcurrentuser (). Getshortusername ()); Set Hadoop.http.filter.initializers toAmfilterinitializer conf.set ("Hadoop.http.filter.initializers", "ORG.APACHE.HADOOP.YARN.SERVER.WEBPROXY.AMF Ilter.            Amfilterinitializer "); Set Mapreduce.job.dir to Submitjobdir, such as/user/hdfs/.staging/job_1459913635503_0005 Conf.set (mrjobconfig.mapreduce_      Job_dir, submitjobdir.tostring ());            Log.debug ("Configuring Job" + JobId + "with" + Submitjobdir + "as the Submit Dir"); Get delegation token for the DIR//Get authorization token for path: Call Tokencache's Obtaintokensfornamenodes () static method Tokencache.obtainto            Kensfornamenodes (Job.getcredentials (), new path[] {submitjobdir}, conf);      Get the keys and tokens and store them in token cache Tokencache populatetokencache (conf, job.getcredentials ()); Generate a secret to authenticate shuffle transfers if (Tokencache.getshufflesecretkey (job.getcredentials ()) = = Nu        ll) {Keygenerator keyGen; try {int keylen = cryptoutils.isshuffleencrypted (conf)? conf.getinT (Mrjobconfig.mr_encrypted_intermediate_data_key_size_bits, Mrjobconfig.default_mr_encrypted_intermediat          e_data_key_size_bits): shuffle_key_length;          KeyGen = Keygenerator.getinstance (shuffle_keygen_algorithm);        Keygen.init (Keylen);        } catch (NoSuchAlgorithmException e) {throw new IOException ("Error Generating Shuffle secret key", E);        } Secretkey Shufflekey = Keygen.generatekey ();      Tokencache.setshufflesecretkey (shufflekey.getencoded (), job.getcredentials ());      }//Copy and configure related files copyandconfigurefiles (job, Submitjobdir);            Get configuration file path: Job.xml path submitjobfile = Jobsubmissionfiles.getjobconfpath (Submitjobdir);            Create the splits for the job log.debug ("Creating splits at" + jtfs.makequalified (submitjobdir)); Call the Writesplits () method, write the Shard data file Job.split and the Shard metadata file Job.splitmetainfo,//And get the calculated number of map tasks maps int maps = Writesplits (Jo  b, Submitjobdir);          Set the number of map tasks in the configuration information mapreduce.job.maps for the above-obtained maps conf.setint (mrjobconfig.num_maps, maps);      Log.info ("Number of splits:" + maps);            The write "queue Admins of the queue to which job was being submitted"//to Job file. Get Job Queue name queue, take parameter mapreduce.job.queuename, parameter not configured defaults to default,//prior to WordCount task example, job queue name queue is default String Queu            E = Conf.get (Mrjobconfig.queue_name, jobconf.default_queue_name); Gets the access control list for the queue Accesscontrollist instance ACL, passed the Getqueueadmins () method of the client communication protocol ClientProtocol instance submitclient, the queue name passed in,//            In fact, in the previous example of the WordCount task, we get the * accesscontrollist ACL = submitclient.getqueueadmins (queue); Configuration information in the WordCount task example that sets the queue parameter Mapred.queue.default.acl-administer-jobs//before the parameter is set to * Conf.set (Tofullpropertynam      E (queue, QueueACL.ADMINISTER_JOBS.getAclName ()), acl.getaclstring ()); Removing Jobtoken referrals before copying the jobconf to HDFS//As the Tasks don ' t need this settiNg, actually they may break//Because of it if present as the referral would point to a//different job.      Empties the cached token tokencache.cleanuptokenreferral (conf); Depending on the parameters, determine if you need to trace the token ID//Take parameter mapreduce.job.token.tracking.ids.enabled, the parameter is not configured by default to False if (Conf.getboolean (M rjobconfig.job_token_tracking_ids_enabled, mrjobconfig.default_job_token_tracking_ids_enabled)) {//Pass The job gets the token ID and is stored in the Trackingids list//ADD HDFS tracking ids arraylist<string> trackingids = new Arraylist<st        Ring> (); for (token< extends tokenidentifier> t:job.getcredentials (). Getalltokens ()) {Trackingids.add        (T.decodeidentifier (). Gettrackingid ()); }//Set the contents of the Trackingids list to the parameter Mapreduce.job.token.tracking.ids conf.setstrings (mrjobconfig.job_token_      Tracking_ids, Trackingids.toarray (New String[trackingids.size ())); }//Set reservation info if it exists//if necessary, setThe reservation information exists//parameter is mapreduce.job.reservation.id reservationid Reservationid = Job.getreservationid ();      if (Reservationid! = null) {Conf.set (mrjobconfig.reservation_id, reservationid.tostring ());            }//write job file to submit dir//Call Writeconf () method, write job configuration information to file Job.xml writeconf (conf, submitjobfile); Now, actually submit the job (using the Submit name)//Call Printtokens () method to print the token information to the log file Printt            Okens (JobId, Job.getcredentials ()); Submit the job through the Submitjob () method of the client communication protocol ClientProtocol instance submitclient//And get the job status Jobstatus instance status//analysis by Cluster Connection article we can know that this Su          Bmitclient is actually a yarnrunner or Localjobrunner object,//The SubmitJob () method of the two is finally called, and we leave it to analyze status = Submitclient.submitjob (            JobId, Submitjobdir.tostring (), job.getcredentials ());      If the job status Jobstatus instance status is not NULL, return directly, otherwise throw the IO exception that cannot load the job if (status! = NULL) {return status;     } else {throw new IOException ("Could not launch job"); }} finally {//Finally, before throwing an IO exception that fails to load the job, call the Delete () method of the file system filesystem instance Jtfs,//delete the related directory or file submitted by the job Submitjobdir        if (status = = null) {Log.info ("cleaning up the Staging area" + Submitjobdir);      if (Jtfs = null && Submitjobdir! = null) Jtfs.delete (Submitjobdir, true); }    }  }

Submitjobinternal () method length is relatively long, logic is also very complex, this article first introduced its general logic, the subsequent penny will introduce the details of each link, and the following is related to the previous WordCount job examples in the Hadoop2.6.0 version Mapreudce example WordCount (a) and its sister articles, please note! The Submitjobinternal () method is broadly logical as follows:

1. Call the Checkspecs () method to verify that the job output path is configured and that it already exists:

The correct situation should be configured and not present, the output path configuration parameter is Mapreduce.output.fileoutputformat.outputdir, before the output path of the WordCount job is configured to hdfs://nameservice1/ Output/output;

2, obtain the configuration information from the job conf;

3. Call the Addmrframeworktodistributedcache () method to add the application framework path to the distributed cache;

4. Get job execution Phase zone path Jobstagingarea by Jobsubmissionfiles Getstagingdir () static method:

Take the parameter yarn.app.mapreduce.am.staging-dir, the parameter is not configured by default to/tmp/hadoop-yarn/staging, and then the job user name/.staging is followed/submitted. Through the execution of the previous WordCount task, we looked at the history, learned that the parameter Yarn.app.mapreduce.am.staging-dir was configured as/USER, and the submitting job user name is HDFs, so the complete path should be/user/hdfs /.staging;

5. Obtain the current native address IP;

6, determine the host address of the job submission, hostname, and set the configuration information conf, the corresponding parameters are Mapreduce.job.submithostname, mapreduce.job.submithostaddress;

7. Generate job ID, that is, jobid instance Jobid:

The job ID is generated by the Getnewjobid () method of the client communication protocol ClientProtocol instance submitclient, which is the Jobid instance Jobid;

8, the Jobid set into the job;

9. Construct submit Job Path instance Submitjobdir:

Jobstagingarea followed by/jobid, such as/job_1459913635503_0005, before wordcount the full path of the job is/user/hdfs/.staging/job_1459913635503_ 0005;

10, set some parameters of the job:

10.1, set Mapreduce.job.user.name as the current user, the previous WordCount example is configured for HDFS users;

10.2, set hadoop.http.filter.initializers for Amfilterinitializer;

10.3, set Mapreduce.job.dir for Submitjobdir, such as/user/hdfs/.staging/job_1459913635503_0005;

11. Get authorization Token for path: Call Tokencache's Obtaintokensfornamenodes () static method;

12. Obtain the keys and tokens through the Populatetokencache () method and store them in the token cache tokencache;

14, copy and configure the relevant files: through the Copyandconfigurefiles () method to achieve;

15, get the configuration file path: job.xml;

16, call the Writesplits () method, write the Shard data file job.split and shard metadata file Job.splitmetainfo, and get the number of map tasks calculated maps;

17. Set map task number in configuration information mapreduce.job.maps for the above maps;

18, Get Job queue name queue, take parameter mapreduce.job.queuename, parameter is not configured defaults to default, the previous WordCount task example, the job Queue name queue is default;

19. Get the Access control list for the queue Accesscontrollist instance ACL:

Through the client communication protocol ClientProtocol instance Submitclient's Getqueueadmins () method, the queue name is passed in, and in fact the previous WordCount task example, obtained here is *;

20, the configuration information set the queue parameter Mapred.queue.default.acl-administer-jobs, in the previous WordCount task example, the parameter is set to be *;

21. Empty the cached token: through the Tokencache cleanuptokenreferral () method;

22, according to the parameters to determine whether you need to track the token ID, if necessary:

Take the parameter mapreduce.job.token.tracking.ids.enabled, the parameter is not configured by default to False, get the token ID through the job, and store it in the Trackingids list, set the contents of the Trackingids list to the parameter MapReduce . in Job.token.tracking.ids;

23, if necessary, set up the existing booking information: the parameter is mapreduce.job.reservation.id;

24, call Writeconf () method, write job configuration information to file Job.xml;

25. Call the Printtokens () method to print the token information to the log file;

26. Submit the job through the Submitjob () method of the client communication protocol ClientProtocol instance submitclient and get the job status Jobstatus instance status:

From the analysis of the cluster connection article we can know that this submitclient is actually a yarnrunner or Localjobrunner object, and the final call is the Submitjob () method of both, which we leave to analyze later;

27. If the job status Jobstatus instance status is not NULL, return directly, or throw an IO exception that cannot load the job:

Finally, before throwing an IO exception that fails to load the job, call the file system filesystem instance Jtfs Delete () method to delete the related directory or file submitjobdir that the job submitted.

The overall process, for the key steps of the main details, confined to space, please pay attention to "MapReduce source analysis Jobsubmitter (ii)"!

Jobsubmitter of MapReduce Source Analysis (I.)