Mapreduce on Yarn

Mapreduce computing framework

The computing process is divided into two stages: map and reduce.

The map stage processes input data in parallel;

The reduce stage summarizes the map results.

Shuffle connects two phases: map and reduce.

Map task writes data to a local disk;

Reduce tasks read one copy of data from each map task;

Suitable for offline batch processing only

Good fault tolerance and scalability;

Suitable for batch processing tasks;

Disadvantages:

Large overhead for starting MAP/reduce tasks, frequent Io due to excessive disk usage, etc;

 

Mapreduce on yarn (mrv2)

Yarn is responsible for resource management and scheduling; applicationmaster is responsible for task management;

Mrappmaster task scheduling:

1) Yarn allocates resources to mrappmaster;

2) mrappmaster further allocates resources to internal tasks;

 

Mrappmaster Fault Tolerance:

1) After mrappmaster fails, yarn restarts;

2) After the task fails, mrappmaster applies for resources again;

 

How mapreduce on Yarn works

1) The user submits applications/jobs to yarn, including applicaitonmaster, commands for starting applicationmaster, and user programs;
2) ResourceManager allocates the first container for the job and communicates with the corresponding nodemanager. It is required to start the applicationmaster of the job in the containter;

3) applicationmaster first registers with ResourceManager, so that you can directly query the running status of the job through ResourceManager. Then, it applies for resources for each task and monitors the running status of the task until the running ends. Repeat steps 4-7;

4) applicationmaster requests and receives resources from ResourceManager through RPC requests through polling;

5) once the applicationmaster applies for resources, it will communicate with the corresponding nodemanager and require it to start the task;

6) nodemanager starts the task;

7) each task reports its status and progress to the applicationmaster through the RPC protocol, so that the applicaitonmaster can keep abreast of the running status of each task and restart the task when the task fails;
During job running, you can query the current job running status from applicationmaster through rpc at any time;

8) after the job is completed, applicationmaster logs out of ResourceManager and closes himself;