Flume Custom Source

Hello everyone.

The company has a need. Requires Flumne to store the message from MQ to DFS, and writes the flume custom source. , as I was just touching flume. So please forgive me if there is anything wrong with you.

See the source code for Flume-ng. are generally based on different scenes extends Abstractsource implements Eventdrivensource, configurable

The Mqsource code is as follows:

1  Public classMqsourceextendsAbstractsourceImplementsEventdrivensource, configurable {2 3     PrivateLogger Logger = Org.slf4j.LoggerFactory.getLogger (mqsource.class);4 5     Private LongHeartbeat;6 7     Privatemqreceiver receiver;8 9     Privatehandlelinecallback handle;Ten  One     PrivateThread t; A  - @Override -      Public voidConfigure (Context context) { the  -String Mq_url =context.getstring (Mqcontext.mq_broker_uri, activemqconnection.default_broker_url); -String Mq_username =context.getstring (Mqcontext.mq_username, activemqconnection.default_user); -String Mq_password =context.getstring (Mqcontext.mq_password, Activemqconnection.default_password); +String Mq_queuekey = context.getstring (mqcontext.m Q_Q ueuekey, "NULL"); -String Handleclass = context.getstring (Mqcontext.handleclass, "NULL"); +  A         LongMq_recivetimeout = Context.getlong (Mqcontext.mq_recivetimeout, 3000L); at         LongHeartbeat = Context.getlong (Mqcontext.heartbeat, 3000L); -          This. Heartbeat =Heartbeat; -         if("NULL". Equals (Mq_queuekey)) { -Logger.error ("{}: Unable to load Mq_queuekey", GetName ()); -             return; -         } in         if("NULL". Equals (Handleclass)) { -Logger.warn ("{}: Unable to Handleclass using Defaulthandle", GetName ()); toHandleclass = "Com.bidlink.handle.DefaultHandle"; +         } -  theMqconfig Mqconfig =Newmqconfig (Mq_url, Mq_username, Mq_password, Mq_queuekey, mq_recivetimeout); *Logger.info ("{} MQ Configuration: {}", GetName (), mqconfig.tostring ()); $Receiver =MQFactory.MQ.getReceiver (mqconfig);Panax NotoginsengLogger.info ("{}. Get Recerver key is {}. Obj is: {} ", GetName (), Mqconfig.getqueuekey (), receiver); -  the         Try { +@SuppressWarnings ("Unchecked") AClass<?> Handleclazz = (class<?extendsHandlelinecallback>) Class.forName (handleclass); theHandle =(Handlelinecallback) handleclazz.newinstance (); +}Catch(ClassNotFoundException e) { -Logger.error ("{} Unable to load class {}. {} ", GetName (), Handleclass, e); $}Catch(instantiationexception E1) { $Logger.error ("{} instance class error {}. {} ", GetName (), Handleclass, E1); -}Catch(illegalaccessexception E2) { -Logger.error ("{} occur exception {}. {} ", GetName (), Handleclass, E2); the         } -     }Wuyi  the @Override -      Public synchronized voidstart () { WuLogger.info ("Mqsource start ..."); -         //TODO auto-generated Method Stub About         Try { $t =NewThread () { -                  Public voidrun () { -                      while(true) { -                         Try { AList<string> lines =Receiver.gettext (); +                              for(String line:lines) { the                                 //logger.info ("Message line: {}", line); -Event e =Newsimpleevent (); $String Refstr =Handle.refactor (line); theE.setbody (Refstr.getbytes ("GBK")); the getchannelprocessor (). processevent (e); the                             } the                             Super. Start (); - Thread.Sleep (heartbeat); in}Catch(Exception E1) { the e1.printstacktrace (); the                         } About                          the                     } the                 }; the             }; + T.start (); -}Catch(Exception E1) { theLogger.error ("error starting Mqresource {}", E1.getmessage ());Bayi e1.printstacktrace (); the         } the     } -  - @Override the      Public synchronized voidStop () { theLogger.info ("Mqsource stoping ..."); the         if(T.isalive ()) { the             Try { - T.join (); the}Catch(interruptedexception e) { the e.printstacktrace (); the             }94 t.interrupt (); the         } the         Super. Stop (); the     }98  About}

The main code in the Start method:

  New simpleevent ();                                 e.setbody ("Hello everyone". GetBytes ("GBK")
  Getchannelprocessor (). processevent (e);

  Super.start ();

A variety of custom configuration information can be obtained in the context of the Configure method. As in flume.conf, configure the following information:

 tier1.sources.testSources.type = Org.yunjume.source.MQSourcetier1.sources.testSources.MQ_userName  = Admintier1.sources.testSources.MQ_password  = Admin123tier1.sources.testSources.MQ_brokerURL  =tcp:// localhost:61616  tier1.sources.testsources.mq_queuekey= FirstQueuetier1.sources.testSources.MQ_reciveTimeout  =30000 Tier1.sources.testSources.heartbeat  =30000# to process MQ message Queue line and /span>return  new   line. Tier1.sources.testSources.handleClass  = Org.yunjume.handle.DefaultHandletier1.sources.testSources.channels  = Testchannels 

Get the Mq_username value code:

String mq_username = context.getstring ("Mq_username", Activemqconnection.default_user);

Stop is over.

The package jar is placed in the plug-in directory of the Flume home directory. Mine is /usr/lib/flume-ng/plugins.d .

If the name of the package is called Mqsource.jar, you should create a folder in Plugins.d Mqsource and put Mqsource.jar under Mqsource/lib.

The dependent jar is placed under Mqsource/libext. The directory structure is as follows

/usr/lib/flume-ng/plugins.d/mqsource/lib/Mqsource.jar

/usr/lib/flume-ng/plugins.d/mqsource/libext/-dependent jar packages

/usr/lib/flume-ng/plugins.d/mqsource/native local so file or DLL file

Flume Custom Source