Go Hadoop Yarn Task Submission Process

Yarn is a new framework created with the development of Hadoop, the full name of yet another Resource negotiator, which can be translated as "another explorer". Yarn replaces the previous role of Jobtracker (abbreviated JT) in Hadoop, because previously the task of JT was too heavy, responsible for task scheduling, tracking, failure restart, and can only run MapReduce jobs, not support other programming modes, which also limits the use of JT, Yarn came into being to solve these two problems.

In order to express clearly, we can first look at the Hadoop version of this article, I would like to say here is hadoop2.0, that is, the addition of yarn after the version.

1. Yarn (or MRV2)

Yarn separates Jobtracker's mission into:

• ResourceManager (RM) is responsible for managing the allocation of global resources
• Applicationmaster (AM), am corresponds to each specific task and is responsible for managing all matters throughout the life cycle of the task

In addition to the above two, Tasktracker is replaced by NodeManager (abbreviated NM), and RM and NM form the computing platform for the cluster. This design allows a number of ancillary services to be run on NM over a long period of time, and these ancillary services are generally application-specific and are loaded at NM startup by configuration item designation. For example, when running a mapreduce program on yarn, the shuffle is a secondary service that is loaded by NM. It is important to note that, prior to Hadoop 0.23, shuffle was part of the Tasktracker.

The AM, which is related to each application, is a framework class library that communicates with RM to negotiate how to allocate resources, collaborate with NM, and monitor application execution. In yarn design, MapReduce is just a programming pattern, and yarn allows application architectures like MPI (Message Passing interface) and spark to be deployed on yarn.

2. Yarn Design

is a typical yarn cluster. You can see that RM has two main services:

• Pluggable Scheduler, only responsible for scheduling the user to submit tasks
• Applicationsmaster (abbreviated ASM) is responsible for managing applicationmaster of each task in the cluster, responsible for task monitoring, failure restart, etc.

In the hadoop1.0, the Resource allocation unit is the slot, and then specifically divided into the map slot and reduce slot, and the number of these slots are defined before the task runs, the task can not be changed during the operation, it is obvious that this will cause the allocation of resources is not the problem. In haodop2.0, yarn uses the concept of container to allocate resources. Each container is composed of some properties that can be dynamically changed, so far, only two kinds of memory and CPU are supported. But this kind of resource management method of yarn is universal, the community will add more attributes, such as network bandwidth, local hard disk size and so on.

3. Yarn Main Components

In this section, we mainly introduce the various components of yarn and how they communicate with each other.

3.1 Client<->rm

The diagram above is the process at which the client submits the task to RM.
(1) client informs the ASM build in RM via new application Request
(2) ASM typically returns a newly generated global ID, in addition to the information that is passed to the cluster, so that the client can request resources to run the first container of the task when needed.
(3) After that, the client can construct and send ASC. ASC includes scheduling queue, priority, user authentication information, in addition to these basic information, but also includes the CLC information used to start AM, a CLC includes jar package, dependent files, security token, as well as other files needed during the running of the task.

After these three steps, a client completes a task submission. After that, the client can query the status of the task directly through RM and, if necessary, require RM to kill the application. Such as:

3.2 rm<->am

After the RM receives the ASC sent by the client, it queries the container to meet its resource requirements to run AM, and when found, RM communicates with the NM on the container machine to start AM. The following figure depicts the details of this.

(1) am Register with RM, this process includes the handshaking process, and transmits some information, including the RPC port of AM listener, the URL to monitor the running status of the task, etc.
(2) The Scheduler component in RM responds. This process passes the information needed for am, such as the maximum and minimum resource usage for the cluster. AM uses this information to calculate and request the resources required for the task.
(3) This process is am to request resources from RM. The information passed mainly contains the list of request container, and may contain a list of the container that AM has released.
(4) After AM passes (3) Request resources, at a later time, the heartbeat packet and task progress information will be sent to RM
(5) Scheduler after receiving the resource request from AM, the container is allocated according to the scheduling policy to satisfy the AM request.
(6) After the task is completed, am sends an end message to RM and exits.

Between (5) and (6) above, am receives the container list returned by RM, communicates with the NM of the machine where each container is located, to start the container, and to say the process.

3.2 am<->nm

(1) AM sends the CLC to the container machine's nm to start the container
(2) (3) During the container operation, am can query its running state

4. API

From the above description, developers in the development of yarn on the application of the main need to focus on the following interfaces:

• Applicationclientprotocol

  The client uses this protocol to communicate with RM to start a new application, check the running status of a task, or kill a task

• Applicationmasterprotocol

  AM uses this protocol to register/revoke to RM and request resources to run the task.

• Containermanagementprotocol

  AM uses this protocol to communicate with NM to start/stop container and query the status of container.

5. Summary

The MapReduce written by the user using the Hadoop1.0 API can run directly on yarn without modification, but with yarn's development, backward compatibility is not yet known. Anyway, the new yarn platform is definitely worth our use.

Original

Go Hadoop Yarn Task Submission Process