The start-up process of Storm's topology (II.)

In one of the topology presented to Nimbus

Nimbus

Nimbus can be said to be the most central part of Storm, its main functions are two:

• Assigning resources to topology tasks
• Receive the user's commands and do the corresponding processing, such as topology, Kill, activate and so on

Nimbus itself is based on the thrift framework, using the Thrift Thshaserver Service, the semi-synchronous semi-asynchronous service mode, using a separate thread to process the network IO, using a standalone thread pool to process the message, greatly improving the concurrency of the message processing ability.

The definition of the service interface is defined in the Storm.thrift file, with some code attached:

Service Nimbus {void submittopology (1:string name, 2:string uploadedjarlocation, 3:string jsonconf, 4:stormtopology  Topology) throws (1:alreadyaliveexception E, 2:invalidtopologyexception ite); void Submittopologywithopts (1:string name, 2:string uploadedjarlocation, 3:string jsonconf, 4:stormtopology topology,  5:submitoptions options) throws (1:alreadyaliveexception E, 2:invalidtopologyexception ite);  void Killtopology (1:string name) throws (1:notaliveexception e);  void Killtopologywithopts (1:string name, 2:killoptions options) throws (1:notaliveexception e);  void Activate (1:string name) throws (1:notaliveexception e);  void Deactivate (1:string name) throws (1:notaliveexception e); void rebalance (1:string name, 2:rebalanceoptions options) throws (1:notaliveexception E, 2:invalidtopologyexception it  e); Need to add functions for asking about status of storms, what nodes they ' re running on, looking at task logs string is  Ginfileupload (); void UPLOADCHUNK (1:string location, 2:binary chunk);    void Finishfileupload (1:string location);  String beginfiledownload (1:string file);  Can stop downloading chunks when receive 0-length byte array back binary downloadchunk (1:string ID);  Returns JSON string getnimbusconf ();  Stats functions Clustersummary getclusterinfo ();  Topologyinfo Gettopologyinfo (1:string id) throws (1:notaliveexception e);  Returns JSON string gettopologyconf (1:string id) throws (1:notaliveexception e);  Stormtopology gettopology (1:string id) throws (1:notaliveexception e); Stormtopology getusertopology (1:string id) throws (1:notaliveexception e);}

When the command nohup ${storm_home}/bin/storm Nimbus & is executed, the Nimbus service is started and the specific code executes:

Storm python script code, which starts the Backtype.storm.daemon.nimbus program by default:

def nimbus (klass= "Backtype.storm.daemon.nimbus"): "" "    Syntax: [Storm Nimbus]    launches the Nimbus daemon. This command should is run under     supervision with a tool like Daemontools or Monit.     See Setting-a Storm cluster for more information.    (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)    "" " Cppaths = [Cluster_conf_dir]    jvmopts = Parse_args (Confvalue ("nimbus.childopts", cppaths) + [        "-dlogfile.name =nimbus.log ",        "-dlogback.configurationfile= "+ Storm_dir +"/logback/cluster.xml ",    ]    exec_storm_ Class (        Klass,         jvmtype= "-server",         extrajars=cppaths,         jvmopts=jvmopts)

Then execute the NIMBUS.CLJ script, which mainly involves two methods--launch-server! (Nimbus's start-up portal) and Service-handler (where the processing logic is really defined).

After Nimbus launch, we provide some services, topology submission, UI information, topology kill,rebalance and so on. In article one, the submission of topology to Nimbus, the processing logic of these services is all in the Service-handler method. The following intercept Service-handler inside processing the logic of submitting topology

(Reify nimbus$iface (^void submittopologywithopts [This ^string storm-name ^string uploadedjarlocation ^String serializedconf ^stormtopology topology ^submitoptions Submitoptions] (Try (Assert (Not-nil? submi toptions)) (validate-topology-name! storm-name) (check-storm-active! Nimbus storm-name false) (              Let [topo-conf (From-json serializedconf)] (try (validate-configs-with-schemas topo-conf) (Catch IllegalArgumentException ex (throw (invalidtopologyexception).            (. GetMessage ex)))))                       (. Validate ^backtype.storm.nimbus.itopologyvalidator (: Validator Nimbus) Storm-name topo-conf topology)) (swap! (: Submitted-count Nimbus) Inc) (Let [Storm-id (str storm-name "-" @ (: Submitted-count nimbus) "-" (Current-time-se    CS)) storm-conf (normalize-conf conf                        (Serializedconf From-json ()                OC Storm-id Storm-id) (Assoc topology-name storm-name)) topology) total-storm-conf (Merge conf storm-conf) topology (Normalize-topology total-storm-conf Topol ogy) storm-cluster-state (: Storm-cluster-state Nimbus)] (system-topology! total-storm-conf Topol ogy);;  This validates the structure of the topology (Log-message "Received topology submission for" Storm-name "with conf "storm-conf);; Lock protects against multiple topologies being submitted at once and;;              Cleanup thread killing topology in b/w assignment and starting the Topology (Locking (: Submit-lock Nimbus)              (Setup-storm-code conf storm-id uploadedjarlocation storm-conf topology) (. setup-heartbeats! storm-cluster-state Storm-id) (let [thrift-status->kw-status {topologyinitialstatus/inactive:inactive Topologyinitialstatus/active:active}] (Start-storm Nimbus storm-name Storm-id (Thrift-statu            S->kw-status (. Get_initial_status submitoptions)))) (Mk-assignments Nimbus))) (Catch Throwable E (Log-warn-error E "topology submission exception. (Topology name= ' "Storm-name" ') ") (throw e))) (^void submittopology [This ^string Storm-nam E ^string uploadedjarlocation ^string serializedconf ^stormtopology topology] (. submittopologywithopts this storm-n Ame uploadedjarlocation serializedconf topology (submitoptions. topologyinitialstatus/active)))

Check to see if the Dag diagram for topology is a valid connection graph, and if the topology name already exists, then assign the resource and task scheduling (Mk-assignments) method, and then write the data to zookeeper after allocating the resources. Watcher found data, notify supervisor read data to start a new worker, a worker is a JVM process, the worker starts the task according to the user's predetermined task number, a task is a thread

In Executor.clj mk-threads:spout, theMk-threads:bolt method is to start the task, and the task is the corresponding spout or bolt component, and then the spout Open, The Nexttuple method, as well as the Preapre,execute method of the bolt, are called here, combined with article I mentioned, for

Spout Method Invocation Order:

Declareoutputfields-> Open, Nexttuple, fail/ack or other

Bolt Method Invocation Order:

Declareoutputfields-> Prepare, execute

It is important to note that in spout, the fail, Ack method, and Nexttuple are called sequentially in the same thread, so there is no significant delay in nexttuple.

At this point, a topology can officially start work.

The start-up process of Storm's topology (II.)