Hadoop service library and event library and its workflow

Hadoop service library:

& http://www.aliyun.com/zixun/aggregation/37954.html "> nbsp; YARN uses a service-based object management model, the main features are:

Specific classes, please see org.apache.hadoop.service package. The core interface is Service abstract implementation AbstractService

In YARN, ResourceManager and NodeManager belong to a combined service, which contains multiple single and combined services, so as to realize the unified management of multiple internal services.

Hadoop Incident Library:

YARN using event-driven concurrency model, the various logic abstracted into events, into the event queue, and then by the central asynchronous scheduler is responsible for passing to the appropriate event scheduler to deal with, or scheduler re-passed until the completion of the task.

See, for example, org.apache.hadoop.yarn.event. The main classes and interfaces are: Event, AsyncDispatcher, EventHandler

By convention, first give a Demo, and then follow the Demo research code.

Example I copied directly <hadoop technology insider>:

Examples involve the following modules:

package com.demo1;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.service.CompositeService;

import org.apache.hadoop.service.Service;

import org.apache.hadoop.yarn.event.AsyncDispatcher;

import org.apache.hadoop.yarn.event.Dispatcher;

import org.apache.hadoop.yarn.event.EventHandler;

/ **

* Created by yang on 2014/8/25.

* /

public class SimpleService extends CompositeService {

private Dispatcher dispatcher;

private String jobID;

private int taskNum;

private String [] taskIDs;

public SimpleService (String name, String jobID, int taskNum) {

super (name);

this.jobID = jobID;

this.taskNum = taskNum;

this.taskIDs = new String [taskNum];

for (int i = 0; i <taskNum; i ++) {

taskIDs [i] = new String (jobID + "_task_" + i);

}

}

public Dispatcher getDispatcher () {

return dispatcher;

}

public void serviceInit (Configuration conf) throws Exception {

dispatcher = new AsyncDispatcher ();

dispatcher.register (JobEventType.class, new JobEventDIspatcher ());

dispatcher.register (TaskEventType.class, new TaskEventDIspatcher ());

addService ((Service) dispatcher);

super.serviceInit (conf);

}

private class JobEventDIspatcher implements EventHandler <JobEvent> {

@Override

public void handle (JobEvent jobEvent) {

if (jobEvent.getType () == JobEventType.JOB_KILL) {

System.out.println ("JOB KILL EVENT");

for (int i = 0; i <taskNum; i ++) {

dispatcher.getEventHandler (). handle (new TaskEvent (taskIDs [i], TaskEventType.T_KILL));

}

} else if (jobEvent.getType () == JobEventType.JOB_INIT) {

System.out.println ("JOB INIT EVENT");

for (int i = 0; i <taskNum; i ++) {

dispatcher.getEventHandler (). handle (new TaskEvent (taskIDs [i], TaskEventType.T_SCHEDULE));

}

}

}

}

private class TaskEventDIspatcher implements EventHandler <TaskEvent> {

@Override

public void handle (TaskEvent taskEvent) {

if (taskEvent.getType () == TaskEventType.T_KILL) {

System.out.println ("TASK KILL EVENT" + taskEvent.getTaskID ());

} else if (taskEvent.getType () == TaskEventType.T_SCHEDULE) {

System.out.println ("TASK INIT EVENT" + taskEvent.getTaskID ());

}

}

}

}

package com.demo1;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.yarn.conf.YarnConfiguration;

/ **

* Created by yang on 2014/8/25.

* /

public class Test {

public static void main (String [] args) throws Exception {

String jobID = "job_1";

SimpleService ss = new SimpleService ("test", jobID, 5);

YarnConfiguration config = new YarnConfiguration (new Configuration ());

ss.serviceInit (config);

ss.init (config);

ss.start ();

ss.getDispatcher (). getEventHandler (). handle (new JobEvent (jobID, JobEventType.JOB_KILL));

ss.getDispatcher (). getEventHandler (). handle (new JobEvent (jobID, JobEventType.JOB_KILL));

}

}

Not surprisingly, the results should be similar:

14/08/25 16:02:20 INFO event.AsyncDispatcher: Registering class com.yws.demo1.JobEventType for class com.yws.demo1.SimpleService $ JobEventDIspatcher

14/08/25 16:02:42 INFO event.AsyncDispatcher: Registering class com.yws.demo1.TaskEventType for class com.yws.demo1.SimpleService $ TaskEventDIspatcher

14/08/25 16:02:54 INFO event.AsyncDispatcher: Registering class com.yws.demo1.JobEventType for class com.yws.demo1.SimpleService $ JobEventDIspatcher

14/08/25 16:03:03 INFO event.AsyncDispatcher: Registering class com.yws.demo1.TaskEventType for class com.yws.demo1.SimpleService $ TaskEventDIspatcher

JOB KILL EVENT

JOB KILL EVENT

TASK KILL EVENTjob_1_task_0

TASK KILL EVENTjob_1_task_1

TASK KILL EVENTjob_1_task_2

TASK KILL EVENTjob_1_task_3

TASK KILL EVENTjob_1_task_4

TASK KILL EVENTjob_1_task_0

TASK KILL EVENTjob_1_task_1

TASK KILL EVENTjob_1_task_2

TASK KILL EVENTjob_1_task_3

TASK KILL EVENTjob_1_task_4

We begin to analyze:

The so-called Task, Job, is actually divided by the business logic, they inherit AbstractEvent class.

SimpleService is a composite service, which put EventHandler and Dispatcher

Start with Test and see how Service is created

Constructor is relatively simple, that is, a job is split into taskNum a Task

ss.serviceInit (config); What did it do:

Create a central event scheduler: AsyncDispatcher (specific implementation we later analysis)

And register Event and Task Event and eventHandler corresponding to 2 to the scheduler.

Here is the initialization and start the service. The last two lines simulate 2 events JOB_KILL event.

We went to ss.getDispatcher (). GetEventHandler () and found that he actually created a GenericEventHandler

What does this handler do?

Just put it

Plug in to BlockingQueue <Event> eventQueue; in.

I do not know if you found that this method is only a team operation ah. That specific call JobEventDIspatcher.handler is where?

Then think of before there is not a central scheduler Well, AsyncDispatcher, Line 80 line, he created a thread, and constantly from the EventQueue said before constantly take Event, and then execute, where the implementation is called specific The handler

So an event-driven program so done.

According to Hadoop early version, the business logic is achieved through function calls, that is, serial. Now based on event-driven, greatly increased concurrency. It is worth learning.

To Zhang family portrait:

HandlerThread is the hidden thread mentioned earlier. EventHandler will generate some new Event, and then re-enter the queue.