A brief analysis of Hadoop yarn

The first time you touch Hadoop, the nodes that start Hadoop appear are:

NameNode

Secondarynamenode

Jobtracker

Tasktracker

DataNode

NameNode

The nodes that are now starting Hadoop appear:

Secondarynamenode

NodeManager

ResourceManager

NameNode

DataNode

Found in Hadoop now, Jobtracker and Tasktracker disappeared, more NodeManager and ResourceManager

Later, I found that the original Hadoop framework has changed.

Below is an introduction to the new Hadoop framework, Yarn.

Hadoop is an open source Distributed file storage and processing framework,

First of all, the process of the original mapreduce and design ideas:

1. First the user program (Jobclient) submits a job,job message sent to Jobtracker, Jobtracker is the center of the MapReduce framework,

You need to have timed communication with machines in the cluster (heartbeat), need to manage which programs should run on which machines, and manage all job failures, reboots, and so on.

2.TaskTracker is a part of every machine in the MapReduce cluster, and the main thing he does is to monitor the resources of his own machine.

3.TaskTracker monitors the tasks health of the current machine at the same time. Tasktracker needs to send this information through heartbeat to Jobtracker,

Jobtracker collects this information to run the newly submitted job assignment on which machines.

The following pictures are easy to understand:

MapReduce Run:

The first is the client to write a mapreduce program, the configuration of the MapReduce job is job,

Next is to submit the job, submit job is submitted to Jobtracker, this time Jobtracker will build this job, specifically, assign a new job task ID value,

Next it will do a check operation, this check is to determine whether the output directory exists, if there is a job can not run normally, Jobtracker will throw an error to the client,

The next step is to check if the input directory exists, and if there is no same throw error, if there is a jobtracker, the input shard will be computed based on the input, and if the Shard is not computed it will throw an error. These are all done jobtracker will configure the resources required for the job.

Once the resource is allocated, Jobtracker initializes the job, and the initialization is primarily done by putting the job into an internal queue so that the configured job scheduler can dispatch to the job, and the job scheduler initializes the job. Initialization is the creation of a running Job object (encapsulating tasks and logging information) so that Jobtracker tracks the status and process of the job.

When the initialization is complete, the job scheduler obtains the input shard information (input split), creating a map task for each shard. Next is the task assignment, this time Tasktracker will run a simple loop mechanism to send the heartbeat to Jobtracker regularly, the heartbeat interval is 5 seconds, the programmer can configure this time, the heartbeat is Jobtracker and Tasktracker Communication Bridge,

Through the heartbeat, Jobtracker can monitor whether the tasktracker is alive or not, and can get the status and problems of tasktracker processing, and Tasktracker can also get the operation instructions Jobtracker to it through the return value in the heartbeat. The task is performed after the assignment is done. At the time of task execution, Jobtracker can monitor the state and progress of tasktracker through the heartbeat mechanism, and also can calculate the status and progress of the whole job, and Tasktracker can monitor its status and progress locally.

When Jobtracker obtains the last notification of the successful Tasktracker operation of the specified task, Jobtracker will set the entire job status to success and then when the client queries the job runtime (note: This is an asynchronous operation), The client will check for notification of job completion. If the job fails in the middle, MapReduce will also have a mechanism to deal with, generally, if not the programmer program itself has bug,mapreduce error handling mechanism can ensure that the submitted job can be completed properly.

But with the scale of distributed system cluster and the increase of workload, there are many problems in the original framework, the main manifestations are as follows:

1.JobTracker is a centralized processing point for mapreduce with a single point of failure.

2.JobTracker has done too many tasks, resulting in excessive resource consumption, which can cause a lot of memory overhead when the Map-reduce job is very large.

Potentially, it also increases the risk of jobtracker fail, which is the industry's general conclusion that the map-reduce of old Hadoop can only support the upper limit of 4000-node hosts.

3. On the tasktracker side, the representation of the Map/reduce task as a resource is too simple to take into account the cpu/memory usage,

If two of the large memory-consuming tasks are dispatched to a piece, OOM is easy to appear.

4. On the Tasktracker side, the resource is coerced into the map task slot and the reduce task slot, which can be wasteful if only the map task or reduce task is in the system, which is the cluster resource that was mentioned earlier Use of the problem.

5. Source code level analysis, you will find the code is very difficult to read, often because a class did too many things, the code volume of more than 3,000 lines, resulting in a class task is not clear, increase the difficulty of bug repair and version maintenance.

6. From an operational point of view, the current Hadoop MapReduce framework enforces system-level upgrade updates when there are any important or unimportant changes, such as bug fixes, performance improvements, and characterization. What's worse, it does not matter what the user likes, forcing each client side of the distributed cluster system to update at the same time. These updates will allow users to waste a lot of time trying to verify that their previous application is applying a new version of Hadoop.

The new frame----YARN

The fundamental idea of refactoring is to separate the Jobtracker two main functions into separate components,

These two functions are resource management and task scheduling/monitoring .

The new resource manager globally manages the allocation of all application compute resources, and the applicationmaster of each application is responsible for scheduling and coordinating accordingly.

An application is nothing more than a single traditional MapReduce task or a DAG (directed acyclic graph) task.

The node Management Server for ResourceManager and each machine can manage the user's processes on that machine and can organize the calculations.

In fact, each application's Applicationmaster is a detailed framework library that combines the resources obtained from ResourceManager and NodeManager to work together to run and monitor tasks.

The ResourceManager supports hierarchical application queues that enjoy a certain percentage of the cluster's resources. In a sense it is a purely scheduler that does not monitor and state the application during execution. Similarly, it cannot restart tasks that failed because of an application failure or a hardware error.

ResourceManager is based on the application of resource demand scheduling; Each application requires a different type of resource and therefore requires a different container. Resources include: Memory, CPU, disk, network and so on. As you can see, there is a significant difference between this and the existing Mapreduce fixed type Resource usage model, which has a negative impact on the use of the cluster. The resource Manager provides a plug-in for scheduling policies that is responsible for assigning cluster resources to multiple queues and applications. Scheduling plug-ins can be based on existing capacity scheduling and fair scheduling models.

NodeManager is the agent for each machine framework, a container for executing applications, monitoring the application's resource usage (CPU, memory, hard disk, network) and reporting to the scheduler.

The applicationmaster responsibilities of each application are to ask the scheduler for the appropriate resource containers, to run tasks, to track the status of the application and to monitor their processes, and to handle the reasons for the failure of the task.

New and old Hadoop mapreduce framework Comparison

Let's make a detailed analysis and comparison of the old and new MapReduce framework, and we can see the following notable changes:

First the client is unchanged, its calling API and interface are mostly compatible, this is also to the development of the user transparency, so that it does not have to make large changes to the original code (see 2.3 Demo Code Development and detailed), but the original framework of the core Jobtracker and Tasktracker disappeared, take and It is ResourceManager, Applicationmaster and NodeManager three parts.

Let's explain these three parts in detail,

First ResourceManager is a center of service, it does the thing is to dispatch, start each Job belongs to the Applicationmaster, another monitoring applicationmaster the existence of the situation. Careful readers will find that the tasks inside the Job are monitored, restarted, and so on. This is the reason why Appmst exists.

ResourceManager is responsible for the scheduling of jobs and resources. Receive Jobsubmitter submitted jobs, follow the context information of the job, and the status information collected from NodeManager, start the scheduling process, assign a Container as the App MSTR

NodeManager function is more exclusive, is responsible for the maintenance of Container state, and to RM to maintain the heartbeat.

Applicationmaster is responsible for all work within a job life cycle, similar to the old framework of Jobtracker. But note that each Job (not each) has a applicationmaster that can run on a machine other than ResourceManager.

What are the advantages of Yarn frames relative to the old MapReduce framework? We can see:

1. This design greatly reduces the resource consumption of Jobtracker (now ResourceManager) and makes it more secure and graceful to distribute the programs that monitor the status of each job subtask.

2. In the new Yarn, Applicationmaster is a changeable part that allows users to write their own appmst on different programming models, allowing more types of programming models to run in the Hadoop cluster, as referenced in the Hadoop Yarn official configuration template Mapred-site.xml configuration.

3. The representation of the resource in memory (in the current version of Yarn does not take into account the CPU footprint) is more reasonable than the number of slots remaining.

4. In the old framework, jobtracker a big burden is to monitor the operation of tasks under the job, and now this part is thrown to Applicationmaster, and ResourceManager has a module called Applicationsmasters (note not applicationmaster), which monitors the health of Applicationmaster and restarts it on other machines if a problem occurs.

5.Container is a framework that Yarn proposes for future resource isolation. This should draw on the work of Mesos, is currently a framework, only to provide the isolation of Java Virtual machine memory, Hadoop team design ideas should be able to support more resource scheduling and control, since the resources expressed as the amount of memory, then there is no previous map slot/reduce slot The embarrassing situation that separates the cluster resources from idle.

After a simple analysis, we have a new understanding of the next generation of Hadoop.

A brief analysis of Hadoop yarn