Twitter storm: DRPC Learning

Study path Author: xumingming | can be reproduced, but the original source and author information and copyright statement must be indicated in hyperlink form
Web: http://xumingming.sinaapp.com/756/twitter-storm-drpc/

 

Translated from: https://github.com/nathanmarz/storm/wiki/distributed-rpc.

The introduction of DRPC in storm mainly utilizes storm's real-time computing capability to parallel CPU intensive computing. Storm topology of DRPC uses the parameter stream of the function as the input, and the return value of these function calls as the output stream of the topology.

DRPC is not actually a feature of storm itself. It is a pattern formed by combining storm primitive spout, Bolt, and topology ). We should have packaged the DRPC into a single package, but the DRPC is so useful that we bundled it with storm.

Create a Bolat for data processing in the DRPC example.

/*** Exclaimbolt. java * copyright (c) 2013 * created: cuiran 17:26:42 */package COM. stormdemo. DRPC; import Java. util. map; import Org. apache. commons. logging. log; import Org. apache. commons. logging. logfactory; import COM. stormdemo. demo. demotopology; import JUnit. framework. testcase; import backtype. storm. DRPC. lineardrpctopologybuilder; import backtype. storm. task. topologycontext; import backtype. storm. topology. Basicoutputcollector; import backtype. storm. topology. ibasicbolt; import backtype. storm. topology. outputfieldsdeclarer; import backtype. storm. tuple. fields; import backtype. storm. tuple. tuple; import backtype. storm. tuple. values;/**** todo * @ author cuiran * @ version todo */public class exclaimbolt implements ibasicbolt {Private Static log = logfactory. getlog (exclaimbolt. class. getname ();/* (non-javadoc) * @ See backtype. storm. topology. ibasicbolt # cleanup () */@ overridepublic void cleanup () {// todo auto-generated method stub}/* (non-javadoc) * @ see backtype. storm. topology. ibasicbolt # execute (backtype. storm. tuple. tuple, backtype. storm. topology. basicoutputcollector) */@ overridepublic void execute (tuple, basicoutputcollector collector) {// todo auto-generated method stublog. debug ("processing data"); string Input = tuple. getstring (1); log. debug ("received data:" + input); collector. emit (new values (tuple. getvalue (0), input + "! ");}/* (Non-javadoc) * @ see backtype. storm. topology. ibasicbolt # Prepare (Java. util. map, backtype. storm. task. topologycontext) */@ overridepublic void prepare (MAP stormconf, topologycontext context) {// todo auto-generated method stub}/* (non-javadoc) * @ see backtype. storm. topology. icomponent # declareoutputfields (backtype. storm. topology. outputfieldsdeclarer) */@ overridepublic void declareoutputfields (outputfieldsdeclarer declarer) {// todo auto-generated method stub declarer. declare (new fields ("ID", "result");}/* (non-javadoc) * @ see backtype. storm. topology. icomponent # getcomponentconfiguration () */@ overridepublic Map <string, Object> getcomponentconfiguration () {// todo auto-generated method stubreturn NULL ;}}

Then write a test class

/*** Drpctest. java * copyright (c) 2013 * created: cuiran 17:25:37 */package COM. stormdemo. DRPC; import Org. apache. commons. logging. log; import Org. apache. commons. logging. logfactory; import backtype. storm. config; import backtype. storm. localcluster; import backtype. storm. localdrpc; import backtype. storm. DRPC. lineardrpctopologybuilder; import backtype. storm. topology. topologybuilder; import JUnit. framework. testcase;/**** todo * @ author cuiran * @ version todo */public class drpctest extends testcase {Private Static log = logfactory. getlog (drpctest. class. getname (); Public void testdrpc () {log. debug ("testdrpc start"); config conf = new config (); Conf. setdebug (true); Conf. setnumworkers (2); Conf. setmaxspoutpending (1); localdrpc DRPC = new localdrpc (); localcluster cluster = new localcluster (); lineardrpctopologybuilder builder = new lineardrpctopologybuilder ("Exclamation"); builder. addbolt (New exclaimbolt (), 3); cluster. submittopology ("drpctest", Conf, builder. createlocaltopology (DRPC); log. debug ("result returned by the input parameter:" Export drpc.exe cute ("Exclamation", "hello"); cluster. shutdown (); DRPC. shutdown ();}}

Running result