In traditional MapReduce, Jobtracker is also responsible for Job Scheduling (scheduling tasks to corresponding tasktracker) and task Progress Management (monitoring tasks, failed restart or slow tasks ). in YARN, Jobtracker is divided into two independent daemprocesses: Resource Manager (resourcemanager) is responsible for managing all resources of the cluster,
In traditional MapReduce, Jobtracker is also responsible for Job Scheduling (scheduling tasks to corresponding tasktracker) and task Progress Management (monitoring tasks, failed restart or slow tasks ). in YARN, Jobtracker is divided into two independent Daemon Processes: resource manager (resource manager) is responsible for managing all the resources of the cluster,
In traditional MapReduce, Jobtracker is also responsible for Job Scheduling (scheduling tasks to corresponding tasktracker) and task Progress Management (monitoring tasks, failed restart or slow tasks ). in YARN, Jobtracker is divided into two independent Daemon Processes: resource manager (resource manager) is responsible for managing all resources of the cluster, and application manager (application master) manages the lifecycle of tasks on the cluster. the specific method is that the Application Manager puts forward resource requirements to the resource manager, in the unit of iner, and then runs the application-related processes in these container. the container is monitored by the Node Manager running on the cluster node to ensure that the application does not use excessive resources. each application instance, that is, a MapReduce job, has its own application manager.
In summary, YARN includes the following roles:
- The client submits MapReduce jobs to the entire cluster.
- YARN Resource Manager is responsible for scheduling computing resources of the entire cluster.
- YARN Node Manager, which starts and monitors container on the machine in the Cluster
- MapReduce Application Manager schedules all tasks of a job. The application manager and tasks run in the container. The container is scheduled by the resource manager and managed by the Node Manager.
- Distributed File System, usually HDFS.
The process for running a job in YARN is shown in:
1. Submit a job
The API for Job submission in YARN is similar to the classic MapReduce (step 1 ). the job submission process is similar to the classic MapReduce. The new job ID (Application ID) is allocated by the Resource Manager (step 1 ). the client of the job verifies the output of the job, computes the split of input, and copies the job resources (including the Jar package, configuration file, and split information) to HDFS (step 1 ). finally, submit the job by calling Resource Manager submitApplication () (step 1 ).
2. Job Initialization
When the resource manager receives a submitApplciation () request, it sends the request to the scheduler (schediner), The scheduler allocates the iner, and then the resource manager starts the Application Manager Process in the container, monitored by the Node Manager (Step 5a and step 5b ).
The Application Manager of MapReduce jobs is a Java application whose main class is MRAppMaster. it creates some bookkeeping objects to monitor the job progress and obtain the task progress and completion report (step 1 ). then, the distributed file system is used to obtain the input split calculated by the client (step 1 ). create a map task for each input split according to mapreduce. job. reduces creates a reduce task object.
The application manager then decides how to run the task that constitutes the entire job. if the job is small, the application manager will choose to run the job in its own JVM. This job is called unerized or runs in uber task Mode. before running a job, the job setup method is called to create an output path. unlike a task called by tasktracker in MapRuduce 1, this method is called by the Application Manager in YARN.
3. Task Allocation
If it is not a small job, the application manager Requests the container from the resource manager to run all map and reduce tasks (step 1 ). these requests are transmitted through heartbeat, including the data location of each map task, such as the host name and rack (rack) of the input split ). the scheduler uses this information to schedule a task. It tries its best to allocate the task to the node that stores data, or to the node in the same rack as the node that stores the split.
The request also includes the memory requirements of the task. By default, the memory requirements of map and reduce tasks are both 1024 MB. you can use mapreduce. map. memory. mb and mapreduce. reduce. memory. mb to configure.
The memory allocation method is different from that in MapReduce 1. In MapReduce 1, each tasktracker has a fixed number of slots. Slots are configured in the cluster and each task runs in one slot, each slot has a maximum memory limit, which is also fixed throughout the cluster. this method is not flexible.
In YARN, the granularity of Resource Division is finer. The memory requirement of an application can be between the minimum memory and the maximum memory, and must be a multiple of the Minimum Memory.
4. task running
After a task is assigned to a container by the scheduler of the resource manager, the application manager starts the container (Step 9a and step 9b) through the exercise Node Manager ). the task has a Java application whose main class is YarnChild. before running a task, you must first localize the resources required by the task, such as job configuration, JAR files, and all distributed cache files (step 1 ). finally, run the map or reduce task (step 1 ).
YarnChild runs in a dedicated JVM, but YARN does not support JVM reuse.
5. Update progress and status
Tasks in YARN return their progress and status (including counter) to the application manager. The latter shows the view of the entire job through the umbilical interface every 3 seconds ). this is not the same as MapRduce 1. The latter's progress flow is from tasktracker to jobtracker. for the progress update stream in MapReduce 2:
The client requests progress updates to the application manager every second (through mapreduce. client. progressmonitor. pollinterval) and displays the updates to the user.
In MapReduce 1, The jobtracker UI has a list of running tasks and their corresponding progress. In YARN, the Resource Manager UI displays the UIS of all applications and their respective application managers.
6. job completed
In addition to requesting the job progress from the application manager, the client will call waitForCompletion () every five minutes to check whether the job is completed. The interval can be set through mapreduce. client. completion. pollinterval.
After the job is completed, the application manager and the container will clean up the job status, and the OutputCommiter job cleanup method will also be called. The job information will be stored by the job history server for user verification.
References:
[1]. Hadoop: The Definitive Guide. 3rd Edition. Chapter 6, YARN.
485 total views, no views today
Original article address: YARN job running mechanism. Thank you for sharing it.
