Client MapReduce commit to yarn process (bottom)

Client via RPC protocol Clientrmprotocol commits application, is actually submitted a submitapplicationrequest, in the Hadoop 1.0 era, is used writable as a serialization and deserialization framework , and in 2.0, Hadoop has been discarded and replaced by Protocolbuffer, which has the greatest benefit in addition to supporting multiple languages-backwards compatibility, so that different versions of AM,CLIENT,RM and NM can communicate with each other.

For example, for Submitapplicationrequest, the source code only PB implementation Submitapplicationrequestpbimpl, of course, users can also implement other serialized deserialization framework implementation, including writable implementation

The submit context information Applicationsubmissioncontext is encapsulated in Submitapplicationrequest. Clientrmprotocol the implementation class in RM is Clientrmservice, RM initializes the service when it is started, which is responsible for the port listening for RPC requests ( Set by Yarn.resourcemanager.address)

The service side constructs the Rmappmanagersubmitevent object according to the submission context information, which is rmappmanager to handle, rmappmanager the role in RM is to manage all application, such as submitting/ Complete application, restore a group of applications and so on.

The Submitapplication method of Clientrmservice:

Public Submitapplicationresponse submitapplication (Submitapplicationrequest request) throws Yarnremoteexception  
         {Applicationsubmissioncontext Submissioncontext = Request. Getapplicationsubmissioncontext ();  
         ApplicationID ApplicationID = Submissioncontext.getapplicationid ();  
         String user = Submissioncontext.getuser ();  
         user = Usergroupinformation.getcurrentuser (). Getshortusername (); if (Rmcontext.getrmapps (). Get (ApplicationID)!= null) {Throw the new IOException ("Application with ID" + applicat Ionid + "is already present!  
         Cannot add a duplicate! ");}  
         Submissioncontext.setuser (user); Submit application information to Rmappmanager to start Applicationmaster rmappmanager.handle (new Rmappmanagersubmitevent (submission  
               
         Context, System. Currenttimemillis ()); If recovery is enabled then store the application information in A//Blocking call so make sure this RM has stored the information needed//To restart the AM after RM restart W  
         Ithout further client communication Rmstatestore Statestore = Rmcontext.getstatestore ();  
         Log.info ("Storing Application with ID" + ApplicationID);  
         try {statestore.storeapplication (Rmcontext.getrmapps (). Get (ApplicationID));  
         catch (Exception e) {log.error ("Failed to store Application:" + ApplicationID, E);  
         Exitutil.terminate (1, e);  
         } log.info ("Application with ID" + applicationid.getid () + "submitted by user" + user);  
      
      
         Rmauditlogger.logsuccess (User, auditconstants.submit_app_request, "Clientrmservice", ApplicationID); Submitapplicationresponse response = recordfactory. Newrecordinstance (Submitapplicationresp  
         Onse.class);  
return response; }

Yarn uses an event-driven programming model, where a change in state can trigger one or more events while triggering changes in other states. For Rmappmanagersubmitevent He has two states of App_submit and app_completed, in Rmappmanager handle this event, if it is app_ Completed executes the finishapplication ApplicationID into the Completedapps queue and executes the App_submit method to generate a submitapplication if the status is Rmapp. Put the Rmcontext Map<applicationid, rmapp> applications Total, and then generate a state for the start of the rmappevent referred to the Asyncdispatcher Central scheduler to deal with.

More Wonderful content: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/