YARN Distributedshell Analysis

The source code for Hadoop 2.0 implements two yarn application, one is MapReduce, and the other is a sample program for how to write application----Distributedshell, It can be considered to be the Yarn Workcount sample program.

Distributedshell function, like its name, distributed shell execution, a string of shell commands submitted by the user or a shell script, controlled by Applicationmaster, assigned to a different container execution.

Distributedshell's source code in "hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\ Hadoop-yarn-applications-distributedshell "

Contains three requirements to achieve a application:

Client and RM (Client.java)

Client Submit Application

AM and RM (Applicationmaster.java)

Register am, Request assignment container

AM and NM (Applicationmaster.java)

Start container

To execute a command:

Hadoop jar Hadoop-yarn-applications-distributedshell-2.0.5-alpha.jar Org.apache.hadoop.yarn.applications.distributedshell.client-jar Hadoop-yarn-applications-distributedshell-2.0.5-alpha.jar-shell_command '/bin/date '-num_containers 10

Start 10 container, each executing ' date ' command

To execute the code flow:
1. The client submits application to RM via org.apache.hadoop.yarn.applications.distributedshell.Client, providing Applicationsubmissioncontext
2. Org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster submits the containers request and executes the user-submitted command Containerlaunchcontext.comm ANDs

Client (Client.java):
1. Yarnclient.getnewapplication
2. Fill Applicationsubmissioncontext,containerlaunchcontext (Start am container)
3. Yarnclient.submitapplication
4. Call Yarnclient.getapplicationreport to get application Status every once in a while

Create the context information for AM containerlaunchcontext Amcontainer = Records.newrecord (Containerlaunchcontext.class);  
Set up local resources, Appmaster.jar package, log4j.properties amcontainer.setlocalresources (localresources);  
environment variable, shell script at HDFs address, CLASSPATH amcontainer.setenvironment (env);  
Set the command and parameters for start am vector<charsequence> vargs = new vector<charsequence> (30);  
Vargs.add ("${java_home}" + "/bin/java");  
Vargs.add ("-xmx" + ammemory + "M");  
Am Main class Vargs.add ("Org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster?");  
Vargs.add ("--container_memory" + string.valueof (containermemory));  
Vargs.add ("--num_containers" + string.valueof (numcontainers));  
Vargs.add ("--priority" + string.valueof (shellcmdpriority));  
if (!shellcommand.isempty ()) {Vargs.add ("--shell_command" + Shellcommand + "");  
} if (!shellargs.isempty ()) {Vargs.add ("--shell_args" + Shellargs + ""); For (map.entry<string, string> entry:shellEnv.entrySet ()) {Vargs.add ("--shell_eNV "+ entry.getkey () +" = "+ Entry.getvalue ());  
} vargs.add ("1>" + Applicationconstants.log_dir_expansion_var + "/appmaster.stdout");  
      
Vargs.add ("2>" + Applicationconstants.log_dir_expansion_var + "/appmaster.stderr");  
Amcontainer.setcommands (commands);  
Set up resource requirements, currently only set memory Capability.setmemory (ammemory);  
Amcontainer.setresource (capability);  
Appcontext.setamcontainerspec (Amcontainer); Submit Application to RM super.submitapplication (APPCONTEXT);

Applicationmaster (Applicationmaster.java)
1. Amrmclient.registerapplicationmaster
2. Provide containerrequest to Amrmclient.addcontainerrequest
3. Get container
4. Container into the newly created launchcontainerrunnable thread by Amrmclient.allocate
5. Create Containerlaunchcontext , set Localresource,shellcommand, Shellargs, etc. container startup information
6. Containermanager.startcontainer (Startreq)
7. Response information obtained after the next RPC call Amresponse.getcompletedcontainersstatuses
8. Amrmclient.unregisterapplicationmaster

The new Amrmclient,2.1beta version implements the asynchronous Amrmclient, where it is synchronized ResourceManager = new Amrmclientimpl (Appattemptid);  
Resourcemanager.init (conf);  
Resourcemanager.start (); Register yourself with RM registerapplicationmasterresponse response = ResourceManager. Registerapplicationmaster (appmasterhostname  
, Appmasterrpcport, Appmastertrackingurl); while (Numcompletedcontainers.get () < numtotalcontainers &&!appdone) {//Encapsulate container request, set resource requirements,  
This side only set the memory containerrequest Containerask = SETUPCONTAINERASKFORRM (Askcount);  
      
Resourcemanager.addcontainerrequest (Containerask);  
Send the request to RM log.info ("Asking RM for containers" + ", askcount=" + askcount);  
      
Amresponse Amresp = Sendcontainerasktorm (); Retrieve List of allocated containers from the response list<container> allocatedcontainers = Amresp.getallocat  
Edcontainers (); for (Container allocatedcontainer:allocatedcontainers) {//Create a new thread to submit the Container launch request, so the main thread will not be blocked. LAunchcontainerrunnable Runnablelaunchcontainer = new launchcontainerrunnable (Allocatedcontainer);  
    Thread launchthread = new Thread (Runnablelaunchcontainer);  
    Launchthreads.add (Launchthread);  
Launchthread.start ();  
} list<containerstatus> completedcontainers = Amresp.getcompletedcontainersstatuses (); }//unregister itself Resourcemanager.unregisterapplicationmaster (appstatus, appmessage, null) to RM;

Attach log information for AM

More Wonderful content: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/