Distributed computing MapReduce and yarn working mechanism

First generation of Hadoop composition and structure

The first generation of Hadoop consists of the distributed storage System HDFS and the distributed computing framework MapReduce, in which HDFs consists of a namenode and multiple datanode, MapReduce consists of a jobtracker and multiple tasktracker, which corresponds to Hadoop 1.x and 0.21.x,0.22.x.

1. MapReduce Role Assignment

Client: Job submission initiator.

Jobtracker: Initializes the job, allocates the job, communicates with Tasktracker, and coordinates the entire job.

Tasktracker: Maintains Jobtracker communication and performs a mapreduce task on the allocated data fragment.

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M00/89/DB/wKioL1ggCdOAjDcXAABfYoWNtd8580.png "title=" 123. PNG "alt=" Wkiol1ggcdoajdcxaabfyowntd8580.png "/>

2. MapReduce Execution Process

650) this.width=650; "src=" http://s4.51cto.com/wyfs02/M01/89/DB/wKioL1ggCe-TLNr4AACcJpwl7sY912.jpg "title=" 456. JPG "alt=" wkiol1ggce-tlnr4aaccjpwl7sy912.jpg "/>

(1) Submit a job

The job needs to be configured before the job is submitted

program code, mainly written by their own MapReduce program.

Input/output path

Other configurations, such as output compression.

After the configuration is complete, submit via Jobclinet

(2) Initialization of the job

After the client commits, Jobtracker will queue the job and dispatch it, and the default scheduling method is the FIFO debug mode.

(3) Assignment of tasks

The communication and assignment between Tasktracker and Jobtracker is done through the heartbeat mechanism.

Tasktracker will take the initiative to jobtracker ask whether there is homework to do, if you can do, then will apply to the job task, this task can make map may also be reduce task.

(4) Execution of tasks

After applying to the task, Tasktracker will do the following things:

Copy code to Local

Copy the information of the task to the local

Start JVM Run Task

(5) Update of status and tasks

Task in the process of operation, the first will be their status to Tasktracker, and then by Tasktracker summary of the Jobtracker.

The progress of the task is achieved by counter.

(6) Completion of the operation

Jobtracker the task is marked as successful after it has been accepted until the last task is completed.

This will do the removal of intermediate results and other aftercare work.

Ii. composition and structure of the second generation of Hadoop

The second generation of Hadoop was proposed to overcome various problems with HDFs and MapReduce in Hadoop 1.0. In view of the scalability problem of single namenode restricting HDFs in Hadoop 1.0, the HDFs Federation is proposed, which allows multiple namenode to be assigned different directories in order to achieve access isolation and scale out; for Hadoop The yet of MapReduce in 1.0 is a new resource management framework yarn (another Resource negotiator), which separates the resource management and job control functions in Jobtracker, with the lack of extensibility and multi-frame support. Implemented separately by components ResourceManager and Applicationmaster, where ResourceManager is responsible for resource allocation for all applications, while Applicationmaster is responsible for managing only one application. The Hadoop version corresponds to Hadoop 0.23.x and 2.x.

1.yarn Running architecture

YARN is the next generation of Hadoop computing platforms, as shown below:

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/89/DE/wKiom1ggCgrBLQrPAACl4zW6UwU879.png "title=" 789. PNG "alt=" Wkiom1ggcgrblqrpaacl4zw6uwu879.png "/>

In the YARN architecture, a global ResourceManager runs in the form of a primary background process that typically runs on a separate machine to arbitrate the available cluster resources between competing applications.

ResourceManager tracks how many active nodes and resources are available in the cluster, and coordinates which applications submitted by the user should obtain those resources. ResourceManager is the only process that has this information, so it can make allocation (or scheduling) decisions in a shared, secure, multitenant way (for example, by application priority, queue capacity, ACLs, data location, and so on).

When a user submits an application, a lightweight process instance called Applicationmaster is started to coordinate the execution of all tasks within the application. This includes monitoring tasks, restarting failed tasks, presumably running slow tasks, and calculating the sum of application counter values. These responsibilities were previously assigned to a single jobtracker to complete. Applicationmaster and the tasks belonging to its application run in a resource container controlled by NodeManager.

NodeManager is a more general and efficient version of the Tasktracker. There are not a fixed number of map and reduce slots,nodemanager that have many dynamically created resource containers. The size of the container depends on the amount of resources it contains, such as memory, CPU, disk, and network IO. Currently, only memory and CPU (YARN-3) are supported. The number of containers on a node is determined by the total amount of node resources (such as total CPU count and total memory).

It is necessary to note that Applicationmaster can run any type of task within the container. For example, MapReduce applicationmaster requests a container to start a map or reduce task, and giraph Applicationmaster requests a container to run the Giraph task.

We can also implement a custom applicationmaster to run specific tasks, thus inventing a new Distributed application framework that transforms the big data landscape.

In yarn, MapReduce is downgraded to a role in a distributed application (but still a very popular and useful role), now known as MRV2. MRv2 is a reproduction of the classic MapReduce engine (known as MRV1) that runs on yarn.

2. Yarn can run any distributed application

ResourceManager, NodeManager, and containers do not care about the type of application or task. All code that is specific to the application framework is transferred to Applicationmaster so that any distributed framework can be supported by YARN.

Thanks to this general approach, Hadoop yarn clusters can run many different distributed computing models, such as MapReduce, Giraph, Storm, Spark, Tez/impala, MPI, and so on.

3. Submit application in Yarn

The following discusses how ResourceManager, Applicationmaster, nodemanagers, and containers interact with each other when an application submits to a yarn cluster. Shows an example.

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/89/DB/wKioL1ggCjfA2tMDAABq4OUH9Vw685.png "title=" 111. PNG "alt=" Wkiol1ggcjfa2tmdaabq4ouh9vw685.png "/>

Suppose the user types the Hadoop jar command in the same way as in MRV1 and submits the application to ResourceManager. ResourceManager maintains a list of applications running on the cluster, as well as a list of available resources on each active NodeManager.

ResourceManager need to determine which application should then get a subset of the cluster resources. This decision is subject to many limitations, such as queue capacity, ACLs, and fairness. The ResourceManager uses a pluggable Scheduler. Scheduler only executes the schedule, it manages who gets the cluster resource (in the form of a container), but does not perform any monitoring of the tasks within the application, so it does not attempt to restart the failed task.

When ResourceManager accepts a new application submission, the first decision Scheduler made is to select the container that will be used to run Applicationmaster. After Applicationmaster is started, it will be responsible for the entire life cycle of this application. First and foremost, it sends a resource request to ResourceManager, requesting a container to run the application's tasks.

A resource request is a request to some container to meet some resource requirements, such as:

• A quantitative resource that currently uses MB of memory and CPU share to represent

• A preferred location, specified by host name, rack name

• A priority in this application, not across multiple applications

If possible, ResourceManager assigns a container that satisfies the Applicationmaster request in the resource request (expressed as the container ID and host name). The container allows the application to use a given amount of resources on a particular host. Once a container is assigned, Applicationmaster will require NodeManager (the host that manages the allocation container) to use these resources to start an application-specific task. This task can be any process written in any framework (such as a MapReduce task or a giraph task).

NodeManager does not monitor the task; it only monitors resource usage in the container, for example, if a container consumes more memory than originally allocated, it ends the container.

Applicationmaster will do its best to coordinate the container and start all the required tasks to complete its application. It also monitors the progress of the application and its tasks, restarts the failed tasks in the new request's container, and reports progress to the client submitting the application.

Once the application is complete, Applicationmaster shuts itself down and releases its own container.

Although ResourceManager does not perform any monitoring of the tasks within the application, it checks the health of the Applicationmaster. If the applicationmaster fails, ResourceManager can restart it in a new container. We can assume that ResourceManager is responsible for managing Applicationmaster, and Applicationmasters is responsible for managing tasks.

This article is from the "Love Linux" blog, make sure to keep this source http://ixdba.blog.51cto.com/2895551/1870189

Distributed computing MapReduce and yarn working mechanism