Hadoop Tutorial (vi) 2.x MapReduce process diagram

Looking at the trends in the industry's use of distributed systems and the long-term development of the Hadoop framework, MapReduce's jobtracker/tasktracker mechanism requires massive tweaks to fix its flaws in scalability, memory consumption, threading model, reliability, and performance. The Hadoop development team has done some bug fixes over the past few years, but the cost of these fixes has increased recently, suggesting that it is becoming more difficult to make changes to the original framework.

To fundamentally address the performance bottlenecks of the old MapReduce framework and promote the longer-term development of the Hadoop framework, the MapReduce framework of Hadoop has been completely reconstructed from the 0.23.0 version. The new Hadoop MapReduce framework is named MapReduceV2 or Yarn, and its architecture is shown in the following illustration:

The fundamental idea of refactoring is to separate the two main functions of jobtracker into separate components, which are resource management and task scheduling/monitoring. The new resource manager globally manages the allocation of all application computing resources, and each application's applicationmaster is responsible for the corresponding scheduling and coordination. An application is nothing more than a single traditional MapReduce task or a DAG (with a direction-free graph) task. ResourceManager and each machine's node Management Server can manage the user's processes on that machine and can organize the calculations.

In fact, the applicationmaster of each application is a detailed framework library that combines the resources obtained from ResourceManager and NodeManager to work together to run and monitor tasks.

The ResourceManager supports hierarchical application queues, which enjoy a certain proportion of the resources in the cluster. In a sense it is a pure scheduler, which does not monitor and status track the application during execution. Similarly, it cannot reboot a task that failed because of an application failure or a hardware error.

ResourceManager is based on the requirements of the application of the resource scheduling; Each application requires a different type of resource and therefore requires a different container. Resources include: Memory, CPU, disk, network, and so on. As can be seen, this is significantly different from the current Mapreduce fixed type Resource usage model, which has a negative impact on the use of the cluster. The Resource Manager provides a plug-in for a scheduling policy that assigns cluster resources to multiple queues and applications. Scheduling plug-ins can be based on existing capacity scheduling and fair scheduling models.

In the figure above, NodeManager is a proxy for each machine framework, a container for executing applications, monitoring the application's resource usage (CPU, memory, hard disk, network) and reporting to the scheduler.

Applicationmaster's responsibilities for each application are to ask the scheduler for the appropriate resource containers, run tasks, track the status of applications, and monitor their processes to handle the failure reasons for the task.

See more highlights of this column: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/