Custom Sink Interceptor in Flume

Source: Internet
Author: User

Sinkprocessor:============================failover:load balancing://Load Balancer Processor//Round_robin Polling 1-2-3-1-2-3-...//random randomly 1-3-2-3-1-...1, Round_robin polling 1-2-3-1-2-3-...                2, random randomly: custom sink&&Interceptor============================================= 1, Pom<dependency> <groupId>org.apache.flume</groupId> &LT;ARTIFACTID&GT;FLUME-NG-CORE&L T;/artifactid> <version>1.8.0</version> </dependency> 2, writing sink: Public classMysinkextendsAbstractsink { PublicStatus process ()throwseventdeliveryexception {//Initialize StatusStatus result =Status.ready; //get the Channel objectChannel Channel =Getchannel (); Transaction Transaction=channel.gettransaction (); Event Event=NULL; Try {                //Open TransactionTransaction.begin (); //getting events from the channelevent =Channel.take (); if(Event! =NULL) {                //adding headers manually in an eventmap<string,string> map =NewHashmap<string, string>(); Map.put ("TimeStamp", System.currenttimemillis () + "");                Event.setheaders (map); //Get Body                byte[] BODY =Event.getbody (); //valueString headval = Event.getheaders (). Get ("TimeStamp"); System.out.println ("Head:" + Headval + "\tbody:" +NewString (body)); } Else {                //no event, that is Backoffresult =Status.backoff; }                //Commit a transactionTransaction.commit (); } Catch(Exception ex) {//rolling back a transactionTransaction.rollback (); Throw NewEventdeliveryexception ("Failed to log event:" +event, ex); } finally {                //Close TransactionTransaction.close (); }            returnresult; }        }            3. Pack and place in/soft/flume/Lib under4, using the custom sink a1.sources=R1 a1.sinks=K1 a1.channels=C1 # Configuring the source A1.sources.r1.type=netcat A1.sources.r1.bind=localhost a1.sources.r1.port= 8888# Configure sink A1.sinks.k1.type=Com.oldboy.flume.MySink # Configuring channel A1.channels.c1.type=Memory A1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100# Binding Channel-source, channel-Sink A1.sources.r1.channels=C1 A1.sinks.k1.channel=C1 Custom Interceptor: Interceptor Event Size Interceptor================================Custom Speed limit interceptors:1, set parameters2, setting the parameter structure3, creating the builder inner class, constructing the object through builder1) to get the value or default value of the parameter by configure2build the Interceptor object using the Build method4, set constant values in the constants inner class Public classSpeedinterceptorImplementsInterceptor {Private intSpeed ;  PublicSpeedinterceptor (intSpeed ) {             This. Speed =Speed ; }             Public voidInitialize () {}/*** Modify Event * Speed limit interceptor, speed limit range, single event * Time range to be aware of the first time * Speed = Bodysize/time * * Calculates the previous event, if it is too fast, sleep * lasttime * LASTBO Dysize*/            Private LongLasttime =-1 ; Private LongLastbodysize = 0;  PublicEvent Intercept (event event) {Map<string, string> headers =event.getheaders (); //gets the length of the body            LongBodysize =event.getbody (). length; //Get current Time            LongCurrent =System.currenttimemillis (); //First Event            if(Lasttime = =-1) {Lasttime=Current ; Lastbodysize=bodysize; }            //not the first event            Else {                LongDuration = Current-Lasttime; intCurrspeed = (int) ((Double) lastbodysize/duration * 1000); //The speed is not super                if(Speed >=currspeed) {                returnevent; }                //It's super fast.                Else {                Try{thread.sleep (lastbodysize/speed * 1000-duration); } Catch(Exception e) {e.printstacktrace (); }} lastbodysize=bodysize; Lasttime=System.currenttimemillis (); }            returnevent; }             PublicList<event> Intercept (list<event>events) {             for(event event:events) {intercept (event); }            returnevents; }             Public voidclose () {} Public Static classBuilderImplementsInterceptor.builder {Private intSpeed ;  Public voidConfigure (Context context) {//equivalent to Context.getinteger ("speed", 1024x768);Speed =Context.getinteger (Constants.speed, Constants.speed_default); }             PublicInterceptor Build () {return NewSpeedinterceptor ( This. Speed); }            }             Public Static classConstants { Public Static FinalString speed = "Speed";  Public Static Final intSpeed_default = 1024; }} Custom Interceptor Usage:============================== 1, write code, package and put in/soft/flume/Lib under2, writing the configuration file i_speed.conf # Name the agent component A1.sources=R1 a1.sinks=K1 a1.channels=C1 # Configuring the source A1.sources.r1.type=seq # named Interceptor A1.sources.r1.interceptors=I1 # Specifies the Interceptor type A1.sources.r1.interceptors.i1.type=Com.oldboy.flume.speedinterceptor$builder a1.sources.r1.interceptors.i1.speed= 1A1.sources.r1.interceptors.i1.speed2= 10# Configure sink A1.sinks.k1.type=Logger # Configuring channel A1.channels.c1.type=Memory A1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100# Binding Channel-source, channel-Sink A1.sources.r1.channels=C1 A1.sinks.k1.channel=C13, flume-ng agent-n A1-F i_speed.conf Note: Sinkprocessor and channelselector channelselector: Select Channel---->sink sinkprocessor: Pick sink When configuring Sinkprocessor, be aware that Channelselector is set to default (not configured) using ZK for flume configuration management:============================================ 1, creating nodes in ZK Client (/FLUME/A1)//Note: Node A1 is the agent nameZkcli.sh-server s102:2181 2, add data to the/FLUME/A1 node, use Zooinspector to//Chinese characters will appear garbledYou can use the idea plugin (zookeeper) a1.sources=R1 a1.sinks=K1 a1.channels=C1 A1.sources.r1.type=netcat A1.sources.r1.bind=localhost a1.sources.r1.port= 8888A1.sinks.k1.type=Logger A1.channels.c1.type=Memory A1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100A1.sources.r1.channels=C1 A1.sinks.k1.channel=C13, try to start flume flume-ng agent-n a1-z s102:2181-p/flume

Custom Sink Interceptor in Flume

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.