Custom Sink Interceptor in Flume

Sinkprocessor:============================failover:load balancing://Load Balancer Processor//Round_robin Polling 1-2-3-1-2-3-...//random randomly 1-3-2-3-1-...1, Round_robin polling 1-2-3-1-2-3-...                2, random randomly: custom sink&&Interceptor============================================= 1, Pom<dependency> <groupId>org.apache.flume</groupId> &LT;ARTIFACTID&GT;FLUME-NG-CORE&L T;/artifactid> <version>1.8.0</version> </dependency> 2, writing sink: Public classMysinkextendsAbstractsink { PublicStatus process ()throwseventdeliveryexception {//Initialize StatusStatus result =Status.ready; //get the Channel objectChannel Channel =Getchannel (); Transaction Transaction=channel.gettransaction (); Event Event=NULL; Try {                //Open TransactionTransaction.begin (); //getting events from the channelevent =Channel.take (); if(Event! =NULL) {                //adding headers manually in an eventmap<string,string> map =NewHashmap<string, string>(); Map.put ("TimeStamp", System.currenttimemillis () + "");                Event.setheaders (map); //Get Body                byte[] BODY =Event.getbody (); //valueString headval = Event.getheaders (). Get ("TimeStamp"); System.out.println ("Head:" + Headval + "\tbody:" +NewString (body)); } Else {                //no event, that is Backoffresult =Status.backoff; }                //Commit a transactionTransaction.commit (); } Catch(Exception ex) {//rolling back a transactionTransaction.rollback (); Throw NewEventdeliveryexception ("Failed to log event:" +event, ex); } finally {                //Close TransactionTransaction.close (); }            returnresult; }        }            3. Pack and place in/soft/flume/Lib under4, using the custom sink a1.sources=R1 a1.sinks=K1 a1.channels=C1 # Configuring the source A1.sources.r1.type=netcat A1.sources.r1.bind=localhost a1.sources.r1.port= 8888# Configure sink A1.sinks.k1.type=Com.oldboy.flume.MySink # Configuring channel A1.channels.c1.type=Memory A1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100# Binding Channel-source, channel-Sink A1.sources.r1.channels=C1 A1.sinks.k1.channel=C1 Custom Interceptor: Interceptor Event Size Interceptor================================Custom Speed limit interceptors:1, set parameters2, setting the parameter structure3, creating the builder inner class, constructing the object through builder1) to get the value or default value of the parameter by configure2build the Interceptor object using the Build method4, set constant values in the constants inner class Public classSpeedinterceptorImplementsInterceptor {Private intSpeed ;  PublicSpeedinterceptor (intSpeed ) {             This. Speed =Speed ; }             Public voidInitialize () {}/*** Modify Event * Speed limit interceptor, speed limit range, single event * Time range to be aware of the first time * Speed = Bodysize/time * * Calculates the previous event, if it is too fast, sleep * lasttime * LASTBO Dysize*/            Private LongLasttime =-1 ; Private LongLastbodysize = 0;  PublicEvent Intercept (event event) {Map<string, string> headers =event.getheaders (); //gets the length of the body            LongBodysize =event.getbody (). length; //Get current Time            LongCurrent =System.currenttimemillis (); //First Event            if(Lasttime = =-1) {Lasttime=Current ; Lastbodysize=bodysize; }            //not the first event            Else {                LongDuration = Current-Lasttime; intCurrspeed = (int) ((Double) lastbodysize/duration * 1000); //The speed is not super                if(Speed >=currspeed) {                returnevent; }                //It's super fast.                Else {                Try{thread.sleep (lastbodysize/speed * 1000-duration); } Catch(Exception e) {e.printstacktrace (); }} lastbodysize=bodysize; Lasttime=System.currenttimemillis (); }            returnevent; }             PublicList<event> Intercept (list<event>events) {             for(event event:events) {intercept (event); }            returnevents; }             Public voidclose () {} Public Static classBuilderImplementsInterceptor.builder {Private intSpeed ;  Public voidConfigure (Context context) {//equivalent to Context.getinteger ("speed", 1024x768);Speed =Context.getinteger (Constants.speed, Constants.speed_default); }             PublicInterceptor Build () {return NewSpeedinterceptor ( This. Speed); }            }             Public Static classConstants { Public Static FinalString speed = "Speed";  Public Static Final intSpeed_default = 1024; }} Custom Interceptor Usage:============================== 1, write code, package and put in/soft/flume/Lib under2, writing the configuration file i_speed.conf # Name the agent component A1.sources=R1 a1.sinks=K1 a1.channels=C1 # Configuring the source A1.sources.r1.type=seq # named Interceptor A1.sources.r1.interceptors=I1 # Specifies the Interceptor type A1.sources.r1.interceptors.i1.type=Com.oldboy.flume.speedinterceptor$builder a1.sources.r1.interceptors.i1.speed= 1A1.sources.r1.interceptors.i1.speed2= 10# Configure sink A1.sinks.k1.type=Logger # Configuring channel A1.channels.c1.type=Memory A1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100# Binding Channel-source, channel-Sink A1.sources.r1.channels=C1 A1.sinks.k1.channel=C13, flume-ng agent-n A1-F i_speed.conf Note: Sinkprocessor and channelselector channelselector: Select Channel---->sink sinkprocessor: Pick sink When configuring Sinkprocessor, be aware that Channelselector is set to default (not configured) using ZK for flume configuration management:============================================ 1, creating nodes in ZK Client (/FLUME/A1)//Note: Node A1 is the agent nameZkcli.sh-server s102:2181 2, add data to the/FLUME/A1 node, use Zooinspector to//Chinese characters will appear garbledYou can use the idea plugin (zookeeper) a1.sources=R1 a1.sinks=K1 a1.channels=C1 A1.sources.r1.type=netcat A1.sources.r1.bind=localhost a1.sources.r1.port= 8888A1.sinks.k1.type=Logger A1.channels.c1.type=Memory A1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100A1.sources.r1.channels=C1 A1.sinks.k1.channel=C13, try to start flume flume-ng agent-n a1-z s102:2181-p/flume

Custom Sink Interceptor in Flume