Topology dispatch of Jstorm

? Topology in the server submission process, a series of validation and initialization: TP structure Check, create a local folder and copy the serialized file Jar package, generate Znode used to hold TP and task information, the last step to task assignment. For example, with:

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvbglobtbfmq==/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/ Dissolve/70/gravity/center "/>
The commit main function is in Servicehandler.java

private void Makeassignment (String topologyname, String Topologyid, Topologyinitialstatus status) throws failedassigntopologyexception {//1, Create topology allocation event Topologyassignevent assignevent = new Topologyassignevent (); Assignevent.settopologyid (Topologyid); Assignevent.setscratch (false); Assignevent.settopologyname (TopologyName); Assignevent.setoldstatus (Thrift.topologyinitialstatustostormstatus (status));  2, throw into the event processing queue Topologyassign.push (assignevent);  3. Wait Time returns Boolean issuccess = Assignevent.waitfinish (), if (issuccess = = True) {Log.info ("Finish submit for" + Topologynam e);} else {throw new Failedassigntopologyexception (Assignevent.geterrormsg ());}}

The most basic of these is the process that might be handled after the event is dropped into the queue. The event assignment is handled by the topologyassign thread, and the process of the thread is very clear, listening to the event queue. Once an event has entered, take it out immediately and proceed with the dotopologyassignment, for example:

public void Run () {log.info ("Topologyassign thread has been started"); Runflag = True;while (runflag) {topologyassignevent Event;try {event = Queue.take ();} catch (Interruptedexception E1) {continue;} if (event = = null) {continue;} Boolean issuccess = Dotopologyassignment (event);.....}

The core code of the task assignment is in Topologyassign.java

Public assignment Mkassignment (Topologyassignevent event) throws Exception {String Topologyid = Event.gettopologyid (); Log.info ("Determining Assignment for" + Topologyid); Topologyassigncontext context = Preparetopologyassign (event); set<resourceworkerslot> assignments = null;if (! Stormconfig.local_mode (nimbusdata.getconf ())) {Itoplogyscheduler Scheduler = Schedulers.get (default_scheduler_name );//start the job scheduling assignments = scheduler.assigntasks (context);} else {assignments = mklocalassignment (context);} ............}

The call stack is as follows:

The principle of allocation is to obtain all available supervisor first, infer that the supervisor available standard is whether there is a spare slot, that is, whether all supervisor.slots.ports specified port is occupied, and then calculate the need to allocate a few woker. Since a woker corresponds to a port, and of course the collection of this information is from zookeeper, we now analyze the core code of the allocation:
Workermaker.java
Note that result is the slot required for this job. Prior to incoming only know the number of slots required, detailed allocation to which supervisor has not been specified
Supervisors refers to all available supervisor in the current cluster. That has spare port.

private void Putworkertosupervisor (List<resourceworkerslot> result,list<supervisorinfo> supervisors) {  int key = 0;//is traversed by the desired slot, each time assigning a for (Resourceworkerslot Worker:result) {//First make the necessary inference and set the position if (supervisors.size () = = 0) return;if (Worker.getnodeid () = null) continue;if (key >= supervisors.size ()) key = 0;//1, remove first Supervisorsupervisorinfo Supervisor = Supervisors.get (key); Worker.sethostname (Supervisor.gethostname ()); Worker.setnodeid ( Supervisor.getsupervisorid ()); Worker.setport (Supervisor.getworkerports (). iterator (). Next ());//The slot is removed from the collection when it is exhausted, No longer participates in assigning supervisor.getworkerports (). Remove (Worker.getport ()); if (Supervisor.getworkerports (). Size () = = 0) Supervisors.remove (supervisor);//When a supervisor is allocated, it is no longer used. Unless supervisor is not enough for key++;}}

From the above code, we can see that slot allocation does not consider the machine load, slot allocation is not necessarily average, for example, the first supervisor has 10 slots, the remaining supervisor only two, then each supervisor assigned a woker.

Note A problem in the above code that supervisors this collection is sorted, such as the following:

private void Putallworkertosupervisor (List<resourceworkerslot> result,list<supervisorinfo> Supervisors ) {.... Supervisors = This.getcanusesupervisors (supervisors);; Collections.sort (Supervisors, new comparator<supervisorinfo> () {@Overridepublic int compare (Supervisorinfo O1, Supervisorinfo O2) {//TODO auto-generated method Stubreturn-numberutils.compare (O1.getworkerports (). Size (), O2.getworkerports (). Size ());}); This.putworkertosupervisor (result, supervisors), ......}

Can see. The current collation is the number of slots per slot, and we may consider some of the factors in the version number that might account for the machine load.

Topology dispatch of Jstorm