Jstorm topology submitting the client

one topology contains one or more spout Bolt,spout is responsible for obtaining data from the data source and sending it to the bolt, each of which is responsible for processing and sending it to the next bolt. Typically, the creation of topology is created by Topologybuilder, which records which spout bolts are included and verifies that each component has an ID conflict, and the checksum method is as follows:

private void Validateunusedid (String id) {if (_bolts.containskey (ID)) {throw new IllegalArgumentException ("Bolt has Already been declared for ID "+ ID);} if (_spouts.containskey (ID)) {throw new IllegalArgumentException ("Spout have already been declared for ID" + ID);} if (_statespouts.containskey (ID)) {throw new IllegalArgumentException ("state spout have already been declared for ID" + ID );}} <span style= "font-family: ' Courier New '; Background-color:rgb (255, 255, 255); " > </span>

Topologybuilder will save each component to the corresponding data structure, with the following data structure:

public class Topologybuilder {//Store all boltprivate map<string, irichbolt> _bolts = new hashmap<string, IRichBolt& gt; ();//Store all spoutprivate map<string, irichspout> _spouts = new hashmap<string, irichspout> ();// Store configuration information for each component private map<string, componentcommon> _commons = new hashmap<string, componentcommon> (); ..........}

Component configuration information is stored in the following ways

private void Initcommon (String ID, IComponent component, number parallelism) {Componentcommon common = new Componentcommon (); Common.set_inputs (New Hashmap<globalstreamid, grouping> ()); if (parallelism! = null) common.set_parallelism_ Hint (Parallelism.intvalue ()); else {    common.set_parallelism_hint (integer.valueof (1));} Map conf = component.getcomponentconfiguration (); if (conf! = null) common.set_json_conf (Utils.to_json (conf)); _ Commons.put (ID, common);}

Once the information is saved, builder will create a Stormtopology instance based on this information, which is then submitted by Stormsubmitter.submittopology, in two steps: 1, uploading the jar file 2, Submit Job

public static void Submittopology (String name, Map stormconf,stormtopology topology, Submitoptions opts) throws Alreadyaliveexception, Invalidtopologyexception {//Read configuration information and perform relevant check if (! Utils.isvalidconf (stormconf)) {throw new IllegalArgumentException ("Storm conf is not valid. Must be Json-serializable ");} stormconf = new HashMap (stormconf); Stormconf.putall (Utils.readcommandlineopts ()); Map conf = Utils.readstormconfig (); Conf.putall (stormconf);p utuserinfo (conf, stormconf); try {String serconf = utils.to_ JSON (stormconf), if (localnimbus! = null) {Log.info ("submitting Topology" + name + "in local mode"); localnimbus.submittop ology (name, NULL, serconf, topology);} else {nimbusclient client = nimbusclient.getconfiguredclient (conf);//Verify that there is no same name topology in the cluster running if (topologynameexists ( conf, name) {throw new RuntimeException ("Topology with Name '" + name+ "' already exists on Cluster");} Upload the jar file, which will explain this method in detail Submitjar (conf); try {Log.info ("Submitting Topology" + name+ "in distributed mode with conf" + SerC ONF); Commit topology, will invoke the server-side Servicehandler Submittopology method, start the topology, that belongs to the service side of the thing if (opts! = null) {client.getclient () . submittopologywithopts (Name, path,serconf, topology, opts);} else {//This was for backwards compatibilityclient.getclient (). Submittopology (name, path, serconf,topology);}} finally {client.close ();}} Log.info ("Finished Submitting topology:" + name);} catch (Invalidtopologyexception e) {...}}

The jar file upload contains two parts, the jar file itself and its dependent library file will be uploaded to the server, the default upload buf size is 512K, can be adjusted by nimbus.thrift.max_buffer_size to BUF size, the server saved directory structure as follows:

[[Email protected] ~] $tree/home/hongmin.lhm/jstorm_data/nimbus/inbox//home/hongmin.lhm/jstorm_data/nimbus/inbox/ '--7c1b7d1e-9134-4ed8-b664-836271b49bd3    '--Stormjar-7c1b7d1e-9134-4ed8-b664-836271b49bd3.jar

private static void Submitjar (Map conf) {if (Submittedjar = = null) {nimbusclient client = Nimbusclient.getconfiguredclient (conf); try {log.info ("Jar not uploaded to master yet. Submitting jar ... "); String Localjar = System.getproperty ("Storm.jar");p ath = client.getclient (). Beginfileupload (); string[] Pathcache = Path.split ("/"); String uploadlocation = path + "/stormjar-" + pathcache[pathcache.length-1] + ". Jar"; List<string> lib = (list<string>) conf.get (genericoptionsparser.topology_lib_name); map<string, string> LibPath = (map<string, string>) conf.get (Genericoptionsparser.topology_lib_path); if ( Lib! = null && lib.size ()! = 0) {for (string libname:lib) {String jarpath = path + "/" + Libname;client.getclien T (). Beginlibupload (Jarpath); Submitjar (conf, Libpath.get (libname), jarpath, client);} if (Localjar! = null) Submittedjar = Submitjar (conf, localjar,uploadlocation, client);} else {Submittedjar = Submitjar (conf, Localjar, uploadlocation,client);}} catch (ExCeption e) {throw new RuntimeException (e);} finally {client.close ();}} else {log.info ("Jar already uploaded to master. Not submitting jar. ");}}

Jstorm topology submitting the client