Apache hadoop next-generation mapreduce (yarn)

Original article link

Mapreduce has gone through a thorough overhaul in the hadoop-0.23, and now we have a new framework called mapreduce2.0 (mrv2) or yarn.

The basic concept of mrv2 is to split two main functions (resource management and Job Scheduling/monitoring) in jobtracker into separate daemon processes. The idea is to have a global resourcemaager (RM) and the applicationmaster (AM) corresponding to each application ). An application is a map-reduce or Dag job in the traditional sense.

ResourceManager and nodemanager (Nm) on each node constitute the data computing framework. Rm is the final authority of arbitration system resources in all applications ).

In fact, the applicationmaster of each application is a specific framework library used to negotiate resources with RM, work with nodemanager (s), execute and monitor tasks.

ResourceManager has two important components: schedager and applicationsmanager.

Scheduler allocates restricted resources for various running applications, such as similar constraints, capacity, and queues. Scheduler is just a scheduler and does not monitor or track the application status. Also, he does not guarantee to restart tasks that fail due to application failure or hardware failure. Schediner implements its scheduling function based on application resource requirements. It is based on the abstract concept of combining memory, CPU, network, disk, and other elements-resource container iner. In the first version, only memory is supported.

Scheduler has an insertion policy plug-in that partitions cluster resources in different queues and applications. Currently, map-Reduce schedulers, such as capacityscheduler, and fairscheduler are examples of this plug-in.

Capacityscheduler makes it easier to predict shared cluster resources and supports hierarchical queues.

Applicationsmanager is responsible for receiving the submission of tasks and negotiating the first container used to execute the specific applicationmaster ?) Restart the service of the applicationmaster container if the container fails.

Nodemanager is the proxy framework for each machine. It monitors the resource usage (CPU, memory, disk, and network) of each machine and reports it to ResourceManager/schedager.

The applicationmaster of each application is responsible for negotiating with scheduler appropriate resource containers, tracking their status, and monitoring progress.

Mrv2 is compatible with previous stable versions (hadoop-1.x), which means that the desired map-reduce jobs can run on mrv2.

& #160;

& #160;

Understanding: the yarn framework is built on the previous map-Reduce framework. It splits the two main functions of jobtracker, namely, the home, the resource (RM boss), and the monitoring (nm, applicationmaster), with a clear division of labor.

Rm again assigned his jobs to two small leaders (sched smanager). The job is sent to applicationsmanager, the job is scheduled to schedmanager, and applicationsmanager is responsible for restarting applicationmaster when the job fails, I have multiple roles.

The monitoring activity is also subdivided. The node status (memory, CPU, hard disk, network, etc.) is monitored by nm and reported to the lead (RM, it should be reported to the scheduler leader, so that he will allocate tasks according to the status of your node during scheduling. The status and progress of each application are monitored by the respective applicationmaster. If the applicationmaster fails (task ?), Applicationsmanager will help you restart.

Apache hadoop next-generation mapreduce (yarn)