Hadoop on Yarn each component detailed principle

The ResourceManager and NodeManager , which run on separate nodes, form the core of yarn and build the entire platform. Applicationmaster and the corresponding container together make up a yarn application system.

ResourceManager provides scheduling of applications, each of which is managed by a applicationmaster that requests compute resources for each task in the form of Container . The container is dispatched by the Resourcemangaer and is managed locally by the NodeManager on each node.

ResourceManager components

　Got a big ResourceManager, you know, haha--that's all. ResourceManager is master, the quorum cluster is all available resources to help manage distributed applications running on the yarn platform. Collaboration with others includes receiving Applicationmaster resource requests, nodemanager each node from ResourceManager, and managing available resources on a single node. (Applicationmaster's job is to apply resources to ResourceManager and work with NodeManager, start, monitor, and stop container.)

One, client and ResourceManager interaction

Here the client side, ResourceManager mainly through the client service,admininistration service, application ACL Manager ResourceManager Web application and Web service components interact with the client.

1. Client Service

This service implements the basic client -to- ResourceManager interface applicationclientprotocol. This component handles all Remote Procedure Call (RPC) traffic from the client to ResourceManager. Includes: Application submission, application termination, get applications, queues, cluster statistics, user ACLs, and more. In safe mode, the client service ensures that requests from the user are authenticated (for example, through Kerberos), and that Clientrmservice also provides APIs for clients that are not directly authenticated by Kerberos. Including the ResourceManager proxy token, the proxy token is a special object of the Kerberos authentication client that is able to communicate with the ResourceManager for the first time and pass it to its own amount of non-certified processes.

2. Administration Service

Since the client service is responsible for handling the general user application submission or termination, in order to ensure that the Administrator's request will not be starved to the general user's request , provide a priority operation command, there is more separation of an interface for the administrator to use. A communication protocol that is implemented through Resourcemanageradministrationprotocol . The main actions included are refreshing the queue , reassigning the queue, or changing the properties of the queue, refreshing the list of nodes processed by the ResourceManager , such as adding a newly installed node or retiring an existing node .

3. Application ACL Manager

Requests such as clients and administrators such as the user-facing Api,resourcemanager can launch an ACL for the application by setting the configuration property yarn.acl.enable to True. View the ACL accessed to determine who can view some of the goods through the RPC interface details of all applications, Web UI services and Web services, modified ACLs determine which users can "modify" the Application (killing programs, etc.)

　　when the same ACL information is passed to Applicationmaster, so that applicationmaster can use this information to allow the user to access some services that are running inside the Applicationmaster, As part of the Containerlaunchcontext , when a container is pulled up, NodeManager also receives the same ACL information and uses them to control requests for application/container, mainly including their state, Application logs, and so on.

4. ResourceManager Web application and Web Service

This is ResourceManager. A Web application to output the status information of the cluster, indicators, node active list, and so on, to the Applicationmaster Web interface hyperlinks and a scheduled private interface. (The web.address that we added in the cluster configuration)

II. communication between applications and ResourceManager

Once the application passes through the client-facing service in ResourceManager, it passes through the interior of the ResourceManager and is responsible for pulling up the Applicationmaster state machine . Mainly includes the following communication aspects:

1. Applicationmaster Service

This component corresponds to all requests from Applicationmaster, implements the applicationmasterprotocol protocol, This is the only protocol that Applicationmaster communicates with ResourceManager. This includes the following tasks: registering a new applicationmaster, terminating/canceling a registration request from any of the closed Applicationmaster , Certify all requests from different applicaitonmaster to ensure that legitimate Applicationmaster sent requests are passed to the Application object in ResourceManager, Gets the allocation and release requests from all container running applicationmaster, asynchronously forwards to the yarn scheduler.

Applicaitonmaster service ensures that only one thread can send a request to ResourceManagerat any point in time, any applicaitionmaster. Because all RPC requests from Applicaitonmaster on ResourceManager are serialized.

　2, Applicationmaster Survival monitoring

This monitor tracks each applicationmaster and its last heartbeat time. The time can be configured in the configuration file, Applicationmaster will be marked as dead in the configured time, and ResourceManager will reschedule the application. Run a new Applicaitonmaster instance on a new container, which, by default, allows up to two attempts.

Third, node and ResourceManager communication

The following are the ResourceManager components and the NodeManager that run on the cluster nodes to communicate.

1. ResourceManager Tracker Service

NodeManager sends the heartbeat to Resourcemanager,resourcemanager that the component is responsible for the corresponding RPC from all nodes. The Resourcetracker interface is implemented to communicate with all NodeManager. Main responsibility: Register new node , receive the heartbeat of the front registered node , ensure the legitimate node can communicate with ResourceManager . Reource Tracker Service forwards a legitimate heartbeat to the yarn scheduler , and the yarn scheduler then makes scheduling decisions based on the node's idle resources and resource requests from different applications.

2, nodemanagers Survival monitoring

To track active nodes and determine which nodes are dead, the component tracks each node's identifier (ID) and its last heartbeat time.

3. Nodes-list Manager

is a collection in ResourceManager memory that includes the active node and the excluded node. It also tracks nodes that are explicitly decommissioned by the administrator.

Applicationmaster components

Applicationmaster is responsible for managing the collection of submitted applications. After the application commits, check its legitimacy first, and then determine that no other committed application has used the same ID. The component is also responsible for documenting and managing closed applications that are removed from ResourceManager's memory for a period of time.

First, Applicationmaster Launcher

In yarn, the pull up of each other type of container is initiated by Applicationmaster, and the container of Applicationmaster itself is applied by ResourceManager, and prepared and pulled up on the NodeManager. The component maintains a line pool settings environment, and communicates with NodeManager to pull up the applicationmaster of a newly submitted application, or a applicationmaster that failed for some reason to fail the previous application instance. It is also responsible for telling NodeManager to clean up the applicaitonmaster when the application ends normally or when it is forced to terminate. (Kill the corresponding process)

Second, Yarnscheduler

The Yarn Scheduler is responsible for allocating resources to running applications that are limited by capacity, queues, and so on.

Third, Containerallocationexpirer

This component is responsible for ensuring that all assigned container are eventually used by Applicationmaster and pulled up on the corresponding NodeManager.

NodeManager components

NodeManager accepts requests from applicationmaster to start or stop container, management container is the core function of NodeManager . Before actually pulling up a container, NodeManager will download all the required library files locally, including data files, executables, tarball, jar files, and shell script waits. This allows the downloaded library files to be shared by multiple container of the same application through the Local app level cache .

NodeManager interior can also be divided into a number of nested components, lying trough a lot of ah ...

First, Nodestatusupdater

At the start of NodeManager, the Nodestatusupdater component registers with ResourceManager, sends the available resources for this node, and nodemanager the listening ports for Web server and RPC server. ResourceManager during the registration process, security-related Key,nodemanager to NodeManager will use this key as the container request for Applicationmaster authentication.

Second , Containermanager

Is the core management component of NodeManager. It also contains many subcomponents:RPC Server, resource Localization service ,localization of public resources , and so on.

The specific components are not listed .... Too much.......

Overall operation of the whole operation process:

1, the application submitted to ResourceManager.

2, Applicationmaster start, and register to ResourceManager.

3, Applicationmaster to ResourceManager request container carry out the actual work.

4, Applicationmaster will be assigned from the ResourceManager container information (including a variety of resource configuration Ah, etc.) to NodeManager launch.

5, with this, the calculation process in the container, these container will keep communication with Applicationmaster, send heartbeat and so on to inform the task process.

6. When the application is complete, container is stopped and Applicationmaster is logged off from the ResourceManager.

　　

Hadoop on Yarn each component detailed principle