hadoop2.x Yarn Job Submission (service side)

When RM receives a client job submission request, it responds via RPC server, in which case the client submits the job through the Applicationclientprotocol RPC client, and the client's submission process is shown in the previous article, in the submission phase of the Code, Getnewapplication is invoked first to obtain a getnewapplicationresponse that contains the ApplicationID of the app and the Scheduler resource information. Note that there are multiple rpcserver on the service side of the RM, server for the job submission is Clientrmservice, and the default listener is 18032 ports. Can be configured through the yarn.resourcemanager.address, the following is the getnewapplication of the server, included in the Clientrmservice.java

@Override public
getnewapplicationresponse getnewapplication (
    getnewapplicationrequest request) throws yarnexception {
  //Build Getnewapplicationresponse object
  getnewapplicationresponse response = Recordfactory
      . Newrecordinstance (getnewapplicationresponse.class);
  Set Job ID
  Response.setapplicationid (getnewapplicationid ());
  Set the scheduler resource information, the job ID is set up, then set the scheduler resources, currently includes two parts of CPU memory information, related functions are: YARN.SCHEDULER.MINIMUM-ALLOCATION-MB Yarn.scheduler.minimum-allocation-vcores YARN.SCHEDULER.MAXIMUM-ALLOCATION-MB Yarn.scheduler.maximum-allocation-vcores, this information is specified when the scheduler is started.


  response.setmaximumresourcecapability (Scheduler
      . getmaximumresourcecapability ());       
  
  return response;
}

The job ID is obtained by Getnewapplicationid, which is computed by the cluster startup timestamp and counter

ApplicationID Getnewapplicationid () {
  ApplicationID ApplicationID = Org.apache.hadoop.yarn.server.utils.BuilderUtils
      . Newapplicationid (Recordfactory, Resourcemanager.getclustertimestamp (),
          applicationcounter.incrementandget ());
  Log.info ("Allocated new ApplicationID:" + Applicationid.getid ());
  return applicationid;
}

Build function for Job ID

public static ApplicationID newinstance (long clustertimestamp, int id) {
  applicationid appId = Records.newrecord ( Applicationid.class);
  Appid.setclustertimestamp (clustertimestamp);
  Appid.setid (ID);
  Appid.build ();
  return appId;
}

After the client receives the return information, it knows its job ID, the maximum resource allocation, the following into the submission phase, still in the Clientrmservice

@Override public submitapplicationresponse submitapplication (Submitapplicationrequest request) throws Yarnexception {
  Get Submit context Applicationsubmissioncontext Submissioncontext = Request. Getapplicationsubmissioncontext ();


  Get Job id ApplicationID ApplicationID = Submissioncontext.getapplicationid ();  Applicationsubmissioncontext needs to is validated for safety-only//those fields, that are independent of the RM ' s 
	Configuration would be//checked here, those this are dependent on RM configuration are//in validated.
  Enter a series of security checksum String user = null;
  try {//Submit account is safe user = Usergroupinformation.getcurrentuser (). Getshortusername ();
    catch (IOException IE) {Log.warn ("Unable to get", IE); Rmauditlogger.logfailure (user, Auditconstants.submit_app_request, ie.getmessage (), "Clientrmservice", "Exc
    Eption in submitting Application ", ApplicationID);
  Throw rpcutil.getremoteexception (IE);
}

  Though duplication'll checked again when app was put into rmcontext,//But it's good to fail the invalid SUBMI
  Ssion as early as possible. If the job ID already exists if (Rmcontext.getrmapps (). Get (ApplicationID)!= null) {String message = ' Application with ID ' + appli Cationid + "is already present!
    Cannot add a duplicate! ";
    Log.warn (message); Rmauditlogger.logfailure (user, auditconstants.submit_app_request, message, "Clientrmservice", "Exception in SUBMIT"
    Ting application ", ApplicationID);
  Throw rpcutil.getremoteexception (message); //Set Job queue if (submissioncontext.getqueue () = = null) {Submissioncontext.setqueue (yarnconfiguration.default_queue_n
  AME); //Set Job default name if (submissioncontext.getapplicationname () = = null) {Submissioncontext.setapplicationname (Y
  Arnconfiguration.default_application_name); //Set Job type if (submissioncontext.getapplicationtype () = = null) {Submissioncontext. Setapplicationtype (YarNconfiguration.default_application_type); else {//job type length ...
    What's the use. if (Submissioncontext.getapplicationtype (). Length () > Yarnconfiguration.application_type_length) {submissionCon Text.setapplicationtype (Submissioncontext. Getapplicationtype (). substring (0, Yarnconfiguration.applicati
    On_type_length)); } try {//Call Rmappmanager to submit application directly//start submitting Job Rmappmanager.submitapplication (SU


    Bmissioncontext, System.currenttimemillis (), user, false, NULL);
    Log.info ("Application with ID" + applicationid.getid () + "submitted by user" + user);
  Rmauditlogger.logsuccess (User, auditconstants.submit_app_request, "Clientrmservice", ApplicationID); catch (Yarnexception e) {log.info ("Exception in submitting application with ID" + Applicationid.getid (), E
    );
      Rmauditlogger.logfailure (user, Auditconstants.submit_app_request, e.getmessage (), "Clientrmservice",  "Exception in submitting Application", ApplicationID);
  Throw e;
  } submitapplicationresponse response = Recordfactory. Newrecordinstance (Submitapplicationresponse.class);
return response; }

Job Submission Phase

@SuppressWarnings ("unchecked") protected void Submitapplication (Applicationsubmissioncontext submissioncontext, Long submittime, String user, Boolean isrecovered, Rmstate state) throws Yarnexception {//Get job ID ApplicationID AP
  Plicationid = Submissioncontext.getapplicationid (); Build an app and put it into applicationacls rmappimpl application = Createandpopulatenewrmapp (Submissioncontext, submitTime, user)
  ;
    Determine if the need to recover if (isrecovered) {recoverapplication (State, application);
    Rmappstate rmappstate = State.getapplicationstate (). Get (ApplicationID). GetState (); 
      if (Isapplicationinfinalstate (rmappstate)) {//We are synchronously moving the application into final Momentarily client would not be this application in the NEW state.
      Also//For finished applications we'll avoid renewing tokens.
      Application. Handle (new Rmappevent (ApplicationID, rmappeventtype.recover));
    Return } if (UsergroupinformAtion.issecurityenabled ()) {Credentials Credentials = null;
    try {credentials = Parsecredentials (Submissioncontext);
      catch (Exception e) {Log.warn ("Unable to parse credentials.", e); Sending app_rejected are fine, since we assume that the//Rmapp are in NEW state and thus we haven ' t yet informed
      The//scheduler about the existence of the application assert application.getstate () = = Rmappstate.new; This.rmContext.getDispatcher (). Geteventhandler (). Handle (new Rmapprejectedevent (ApplicationID, E.getmessage ())
      );
    Throw Rpcutil.getremoteexception (e); } this.rmContext.getDelegationTokenRenewer (). Addapplication (ApplicationID, credentials, Submissioncon
  Text.getcanceltokenswhencomplete (), isrecovered); 
            else {//Trigger app Startup Event This.rmContext.getDispatcher (). Geteventhandler (). Handle (new Rmappevent (ApplicationID, Isrecovered? RMAppEventType.RECOVER:RMAppEvenTtype.start)); }
}

Application creates and joins the appropriate collection in the following function, which represents a successful job submission if the collection succeeds

Private Rmappimpl Createandpopulatenewrmapp (Applicationsubmissioncontext submissioncontext, long submitTime, Stri
  ng user) throws Yarnexception {ApplicationID ApplicationID = Submissioncontext.getapplicationid ();
  Validateresourcerequest (Submissioncontext); The build App,submissioncontext contains most of the information about an app Rmappimpl application = new Rmappimpl (ApplicationID, Rmcontext, This.con F, Submissioncontext.getapplicationname (), User, Submissioncontext.getqueue (), Submissioncon


  Text, This.scheduler, This.masterservice, Submittime, Submissioncontext.getapplicationtype ()); Concurrent app submissions with same ApplicationID'll fail here//Concurrent app. submissions with different applic Ationids will isn't//influence each others//again to determine if the job exists, if it does not exist, put it in HashMap, and if it succeeds, it indicates that the job was submitted successfully if (Rmcontext.getrmapps). Put
        Ifabsent (ApplicationID, application)!= null) {String message = ' Application with ID ' + ApplicationID + "IS already present!
    Cannot add a duplicate! ";
    Log.warn (message);
  Throw rpcutil.getremoteexception (message); }//Inform the ACLs Manager this.applicationACLsManager.addApplication (ApplicationID, Submissioncontext.getamco
  Ntainerspec (). GETAPPLICATIONACLS ());
return application; }

One app contains more information, as follows (Rmappimpl.java)

private static Final log = Logfactory.getlog (Rmappimpl.class);


Private static final String unavailable = "n/a";
Immutable fields private final ApplicationID ApplicationID;
Private final Rmcontext Rmcontext;
Private final Configuration conf;
Private final String user;
Private final String name;
Private final Applicationsubmissioncontext Submissioncontext;
Private final Dispatcher Dispatcher;
Private final Yarnscheduler Scheduler;
Private final Applicationmasterservice Masterservice;
Private final StringBuilder diagnostics = new StringBuilder ();
private final int maxappattempts;
Private final Readlock Readlock;
Private final Writelock Writelock; Private final Map<applicationattemptid, rmappattempt> attempts = new Linkedhashmap<applicationattemptid, RMAp
Pattempt> ();
Private final long submittime;
Private final set<rmnode> updatednodes = new hashset<rmnode> ();


Private final String Applicationtype;
mutable fields Private long starttime; Private long Finishtime = 0;
Private long storedfinishtime = 0;
Private Rmappattempt currentattempt;
Private String queue;
@SuppressWarnings ("Rawtypes") private EventHandler handler;
private static final Finaltransition final_transition = new Finaltransition ();


private static final Appfinishedtransition finished_transition = new Appfinishedtransition ();
These states stored are only valid when the app is at killing or final_saving.
Private Rmappstate statebeforekilling;
Private Rmappstate statebeforefinalsaving;
Private Rmappevent eventcausingfinalsaving;
Private Rmappstate targetedfinalstate;


Private Rmappstate recoveredfinalstate; Object Transitiontodo;