Yarn Architecture Basic Overview (i)

1) Introduction

For MRV1, there are obvious shortcomings in the support of expansibility, reliability, resource utilization and multi-framework, and then the next generation of MapReduce's computational framework MapReduce Version2 is born. There is a big problem in MRV1 is that the resource management and job scheduling are thrown to the jobtracker, resulting in a serious single point bottleneck problem, all MRV2 mainly at this point of improvement, he has the resource management module built into a separate universal system called yarn, Suddenly the core of MRV2 from MapReduce to yarn. There is a variety of development and evolution about yarn in the back.

yarn design concept and basic architecture

Yarn is specially designed to solve the shortcomings of MRV1.

1.yarn Resource utilization is high because he can support different computing frameworks running on it2. Low cost of operation, the previous cluster a framework of the model needs a lot of administrators to maintain the system, now can run multiple frames on yarn, you can choose the real-time calculation framework storm, Memory iterative computing framework Spark, The traditional MPI method can also be used to solve the data mining algorithms which require high data calculation. In a nutshell, yarn is a lightweight elastic computing platform.

Yarn's basic composition is still used in the overall structure of master/slave, but in the resource management this piece, or did a little change, a ResourceManager is a master, under each sub-node has a nodemanager, The RM assigns a resource to NM, and in each node there will be Applicationmaster (hereafter referred to as AM), which will be responsible for communicating with RM to get the resources, as well as the NM communication to start or stop the task. You can understand the role of Tasktracker in MRV1. NM is the resource and Task Manager on each node, and there is a lot of collaboration between the NM,AM,RM and the entire application from start to finish. Here is a simulation diagram:

Yarn Architecture Basic Overview (i)