(ii) Kafka-jstorm cluster real-time log analysis---------Jstorm integration Spring

In order to introduce Dubbo RPC framework (with spring configuration), first introduce spring into the jstorm, please first understand the Jsorm multi-threaded document: http://storm.apache.org/documentation/ Understanding-the-parallelism-of-a-storm-topology.html.

　　

A worker process executes a subset of a topology. A worker process belongs to a specific topology and could run one or more executors for one or more components (spouts or Bo LTS) of this topology. A running topology consists of many such processes running on many machines within a Storm cluster.

An executor is a thread, which is spawned by a worker process. It may run one or more tasks for the same component (spout or bolt).

A Task performs the actual data Processing-each spout or bolt that is implement in your code executes as many Tasks across the cluster. The number of the tasks for a component are always the same throughout the lifetime of a topology, but the number of executors (threads) for a component can change over time. This means, the following condition holds true: #threads ≤ #tasks . By default, the number of the tasks are set to being the same as the number of executors, i.e. Storm would run one task per thread.

both spout and bolts are initialized by each thread (you can print the log, or observe the breakpoint). The prepare method of the bolt, or the open method of the spout method, is invoked with the instantiation, which you can think of as a special constructor.

Every instance of each bolt in a multithreaded environment can be executed by different machines. The service required for each bolt may be different, which requires the loading of spring in each bolt class, that is, initialization. Like what

<span style= "FONT-SIZE:18PX;" >public Static class WordCounterBoltCh03 extends Baserichbolt {private static final Logger log = Loggerfactory.getlogge R (wordcounterboltch03.class);p rivate static final long Serialversionuid = 1l;private map<string, integer> Countmap;private outputcollector collector;private String name;private int id;private static final ApplicationContext Applicationcontext;private static final Logmanager Logmanager;static{applicationcontext = Springutil.of (" Learningjstormconfig/spring-kafkabolt-context.xml ");//spring initialization Logmanager = Applicationcontext.getbean ( Logmanager.class) log.info ("--------------applicationcontext initialized from learningjstormconfig/ Spring-kafkabolt-context.xml ");} @Overridepublic void Prepare (Map stormconf, Topologycontext context, Outputcollector collector) {this.countmap = new Hash Map<> (); this.collector = Collector;this.name = Context.getthiscomponentid (); this.id = Context.getThisTaskId (); Log.info ("-----------------WordcounterboltCh03 prepare ");} @Overridepublic void Execute (Tuple input) {String Word = null;try {word = Input.getstringbyfield ("word");} catch (Throwabl E e) {}if (null! = Word) {if (!countmap.containskey (word)) {countmap.put (Word, 1);} else {Integer count = Countmap.get (wor d); Count++;countmap.put (Word, count); Logmanager.write (Word + ":" + countmap.get (Word));}} else {if ("signals."). Equals (Input.getsourcestreamid ()) && "RefreshCache". Equals (Input.getstringbyfield ("action")) {cleanup (); Countmap.clear ();}} This.collector.ack (input);} @Overridepublic void Declareoutputfields (Outputfieldsdeclarer declarer) {} @Overridepublic void Cleanup () {Log.info ("{ Cleanup, ...} ";"); "); Countmap.foreach (k, v), {log.info ("{Clean up ...}");;;} "; Log.info ("K: {}, V: {}", K, V);});}} </span>

The spring configuration file is as follows:

<span style= "FONT-SIZE:18PX;" ><?xml version= "1.0" encoding= "UTF-8"? ><beans xmlns= "Http://www.springframework.org/schema/beans" Xmlns:xsi= "Http://www.w3.org/2001/XMLSchema-instance" xmlns:context= "http://www.springframework.org/schema/ Context "xmlns:mvc=" Http://www.springframework.org/schema/mvc "xmlns:dubbo=" http://code.alibabatech.com/schema/ Dubbo "xmlns:task=" Http://www.springframework.org/schema/task "xsi:schemalocation="/HTTP/ www.springframework.org/schema/beanshttp://www.springframework.org/schema/beans/spring-beans.xsdhttp:// Www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http ://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd Http://code.a Libabatech.com/schema/dubbohttp://code.alibabatech.com/schema/dubbo/dubbo.xsd http://www.springframework.org/ Schema/task http://www.springframework.org/schema/task/spring-task.xsd "> &LT;import resource= "Classpath:/learningjstormconfig/dubbo-provider.xml"/> <context:component-scan Base-package= "Com.doctor.kafkajstrom.log.manager.imp"/> <context:component-scan base-package= " Com.doctor.kafkajstrom.log.service.imp "/> <context:component-scan base-package=" Com.doctor.kafkajstrom.component "/></beans></span>

It is best to customize the spring configuration file for each spout, each bolt. That is, each spout, each bolt loads and initializes the spring environment.

　

(ii) Kafka-jstorm cluster real-time log analysis---------Jstorm integration Spring