Faster and stronger--parsing Hadoop's next generation MapReduce framework yarn (CSDN)

absrtact: This paper introduces the new MapReduce Framework (yarn) principle, advantages, operation mechanism and configuration method of Hadoop since the 0.23.0 version, and emphatically introduces the difference and improvement of the new Yarn frame relative to the original frame.

Editor's note: For the industry's big data storage and distributed processing systems, Hadoop is a familiar open source Distributed file storage and processing framework, the introduction of the Hadoop framework is no longer described here, with the development of demand, Yarn framework surfaced, @ Still glorious Renaissance The blog gives us a very detailed introduction, the reader through the new and old Hadoop MapReduce framework comparison, more able to understand the new yarn framework of the technical principles and design ideas.

background

Yarn is a distributed resource management system that improves resource utilization in distributed cluster environments, including memory, IO, network, disk , and so on. The reason for this is to solve the shortcomings of the original MapReduce framework. The original MapReduce Committer can also be periodically modified on the existing code, but as the code increases and the original MapReduce framework is not designed, it becomes increasingly difficult to modify the original MapReduce framework, So MapReduce's committer decided to redesign MapReduce from the architecture, enabling the next generation of MapReduce (Mrv2/yarn) frameworks to have better extensibility, usability, reliability, Backward compatibility and higher resource utilization, as well as more computational frameworks that support beyond the MapReduce computing framework.

the inadequacy of the original MapReduce framework (--Although Hadoop is powerful, it's not everything.--)

• Jobtracker is a centralized processing point for cluster transactions and there is a single point of failure
• Jobtracker needs to do too much to maintain the status of the job and to maintain the status of the job's task, resulting in excessive resource consumption
• On the Tasktracker side, using the Map/reduce task as a resource is too simple to take into account the CPU, memory and other resources, when the two need to consume large memory of task scheduling together, it is easy to appear OOM
• The resource is forced into the map/reduce slot, and the reduce slot is not available when there is only a map task, and when only the reduce task is available, the map slot is not available and is prone to insufficient resource utilization .

Yarn Architecture

Yarn/mrv2 's most basic idea is to separate the primary Jobtracker resource management and job scheduling/monitoring functions as two separate daemons . There is a global ResourceManager (RM) and each application has a applicationmaster (AM), application equivalent to Map-reduce Job or dag jobs. ResourceManager and NodeManager (NM) Form the basic Data computing framework. ResourceManager coordinate the resource utilization of the cluster, any client or running Applicatitonmaster who wants to run the job or task will have to request a certain amount of resources from RM. Applicatonmaster is a framework-specific library, for the MapReduce framework has its own AM implementation, users can also implement their own am, when running, am will work with NM to start and monitor tasks.

ResourceManager

ResourceManager as a resource coordinator has two main components:Scheduler and Applicationsmanager (AsM).

Scheduler is responsible for allocating the minimum amount of resources required to meet the application run to application. Scheduler is only scheduled based on resource usage, and is not responsible for monitoring/tracking the status of application and, of course, does not handle failed tasks. RM uses the resource container concept to manage the resources of the cluster, resource container is the abstraction of resources, each container includes a certain amount of memory, IO, network resources, but the current implementation includes only one resource of memory.

Applicationsmanager is responsible for handling client-submitted jobs and negotiating the first container for Applicationmaster to run. And the applicationmaster will be restarted when the applicationmaster fails. The following describes some of the functions that RM specifically accomplishes.

2. Resource Scheduling: Scheduler builds a global resource allocation plan from all running application, and then allocates resources based on application special restrictions and some constraints on the global environment.
4. Resource monitoring: Scheduler periodically receives monitoring information for resource usage from NM, and applicationmaster can get status information from scheduler for completed container that belong to it.
6. Application submission:
• Client to ASM to obtain a applicationidclient will application definition and the required jar package
• The client uploads the application definition and the required jar package file to the specified directory in HDFs, specified by the Yarn-site.xml yarn.app.mapreduce.am.staging-dir
• The client constructs a resource request for the object and the submission context of the application is sent to ASM
• ASM receives application's submission context
• ASM negotiates a container for applicationmaster based on application information, and then launches Applicationmaster Scheduler
• Sending Launchcontainer information to the NM to which the container belongs initiates the container, which is also the start Applicationmaster, ASM provides the client with the status information of the running AM.
8. life cycle of am: ASM is responsible for the life cycle management of all am in the system. ASM is responsible for the start of AM, when AM is started, am will periodically send heartbeat to ASM, the default is 1s,asm to understand the survival of AM, and in AM failure is responsible for restarting AM, if after a certain time (default 10 minutes) did not receive AM's heartbeat, ASM thought the AM was a failure.

The availability of ResourceManager is not yet well implemented, but the Cloudera Company's CDH4.4 later version achieves a simple high availability, using the code of the HA section of the Hadoop-common project, using a similar the design of the HDFs Namenode high availability introduces the active and standby states to RM , but there is no role associated with Journalnode, but zookeeper is responsible for maintaining the RM state. This design is just one of the simplest solutions, avoiding the manual restart of RM and a distance from real production.

NodeManager

NM is primarily responsible for starting RM container for AM and container representing AM, and monitoring the operation of container. When starting container, NM will set up some necessary environment variables and download the jar packages, files, etc. required for container to run from HDFs to local, so-called resource localization; when all the preparation is done, The script that represents the container will start up the program. When started, NM periodically monitors the resource usage of the container run, and if it exceeds the amount of resources declared by the container, it will kill the process represented by the container.

In addition, NM provides a simple service to manage the local directory of the machine on which it resides. Applications can continue to access the local directory even though that machine has no container on it running. For example, the Map-reduce application uses this service to store Map output and shuffle them to the corresponding reduce task.

You can also extend your services on NM, yarn provides a yarn.nodemanager.aux-services configuration that allows users to customize some of the services, such as Map-reduce's shuffle functionality, which is implemented in this way.

NM generates the following directory structure locally for each running application:

the directory structure under the container directory is as follows:

At the start of a container, NM executes the container default_container_executor.sh, which executes launch_container.sh inside the script. LAUNCH_CONTAINER.SH will set some environment variables and finally start the command to execute the program. For MapReduce, start am to execute org.apache.hadoop.mapreduce.v2.app.MRAppMaster; start map/reduce The task executes the org.apache.hadoop.mapred.YarnChild.

Applicationmaster

Applicationmaster is a framework-specific library that has its own applicationmaster implementation for the Map-reduce computational model, and for other computational models that want to run on yarn, It is necessary to implement the Applicationmaster for this computational model to run the task with the RM application resource, such as the spark frame running on yarn, and the corresponding Applicationmaster implementation, in the final analysis, yarn is a resource management framework, is not a computational framework, but the implementation of a specific computing framework is needed to run the application on yarn. Since yarn is present along with MRV2, the following is a brief overview of MRV2 's running process on yarn.

MRV2 Running Process:

2. MR Jobclient submits a job to ResourceManager (AsM)
4. ASM asks scheduler for a container to run for Mr Am, and then launches it
6. MR am starts up and registers with ASM
8. Mr Jobclient obtains information about Mr AM from ASM and communicates directly with Mr AM
10. MR am calculates the splits and constructs a resource request for all maps
12. Mr AM is doing the necessary preparation for Mr Outputcommitter
14. MR am initiates a resource request to RM (Scheduler), obtains a set of container for the Map/reduce task to run, and, together with NM, performs some necessary tasks for each container, including resource localization, etc.
16. The MR AM monitors the running task until it finishes, and when the task fails, it requests a new container to run the failed task
18. When each map/reduce task is completed, Mr am runs the cleanup code of Mr Outputcommitter, which is to do some finishing work
20. When all map/reduce are complete, MR am runs the necessary job commit or abort APIs for Outputcommitter
22. MR am exits.

write the application on yarn

Writing applications on yarn and unlike the MapReduce applications we know, it is important to keep in mind that yarn is a resource management framework and not a computational framework that can run on yarn . All we can do is apply container to RM and start container with NM. Just like MRv2, jobclient requests the container for the Mr am run, sets the environment variables and the Start command, and then goes to NM to start Mr am, and then the Map/reduce task is fully accountable to Mr Am, and of course the task is initiated by Mr AM applies container to RM and starts with NM. So to run a non-specific computing framework program on yarn, we have to implement our own client and Applicationmaster. In addition, our custom am needs to be placed under the classpath of each nm, as am may run on the same machine as any nm.

Original link: Yarn detailed (Zebian: Arron)

Faster and stronger--parsing Hadoop's next generation MapReduce framework yarn (CSDN)