Flume use in Windows environments

Flume is a highly available, highly reliable, distributed mass log capture, aggregation, and transmission system provided by Cloudera, Flume supports the customization of various data senders in the log system for data collection, while Flume provides simple processing of data The ability to write to various data-receiving parties (customizable). Currently belongs to a sub-project of Apache.

In general, Flume deployed to a server is installed in a unix/linux environment, but sometimes for testing and debugging purposes, we also have the requirements for installation on Windows systems. The latest version of the current flume, 1.6来 said, is relatively convenient to use on Windows because it comes with a set of execution environment shells.

This is a set of execution environments using PowerShell, starting the program in the Apache-flume-1.6.0-bin\bin directory, Flume-ng.cmd.

Open the command line and enter Flume-ng.cmd Help to see how the program is used.

As an example of a flume agent that collects an offline directory log source, you can run the agent with the following command:

Flume-ng.cmd Agent--conf. \conf--conf-file . \conf\t1.conf--name A1

T1.conf:

A1.sources =R1a1.channels=memorychannela1.sinks=SpoolSinka1.sources.r1.type=Spooldira1.sources.r1.spoolDir=Spool_dira1.sources.r1.fileHeader=trueA1.sources.r1.channels=Memorychannel#InterceptorsA1.sources.r1.interceptors =E1a1.sources.r1.interceptors.e1.type=Regex_extractora1.sources.r1.interceptors.e1.regex= ^ ([\\d]+[\\d]+[\\s]+[\\d\\:]+) \\s+ ([\\d\\-]+[\\s]+[\\d\\:]+[\\s]+[\\d\\:]+) \\s ([\\s\\-]+] \\s ([\\S\\/]+) \\s.[\ \w]+\\s ([\\d\\.] +) \\s\\s[\\w]+\\s ([\\w]+) \\S\\s ([\\d\\.] +\\s[\\d]+) \\S\\s ([\\d\\.] +\\s[\\d]+) \\s\\s[\\w]+ ([\\d\\.] +\\s[\\d]+) \\S\\s\\S ([\\d\\/]+[\\s]+[\\d\\:]+) \\s\\S\\s ([\\d\\/]+[\\s][\\d\\:]+] \\s\\s[\\w\\s]+\\s ([\\d]+) \\S[ \\w\\s]+\\s ([\\d]+) \\s\\s[\\w]+\\s ([\\d\\#]+) [\\w\\s]+\\s ([\\d\\.] +);$A1.sources.r1.interceptors.e1.serializers =S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 s12 S13 S14 s15a1.sources.r1.interceptors.e1.serializers.s1.name=Time1a1.sources.r1.interceptors.e1.serializers.s2.name=Time2a1.sources.r1.interceptors.e1.serializers.s3.name=Xya1.sources.r1.interceptors.e1.serializers.s4.name=Sessiona1.sources.r1.interceptors.e1.serializers.s5.name=Devipa1.sources.r1.interceptors.e1.serializers.s6.name=Protocola1.sources.r1.interceptors.e1.serializers.s7.name=Ip1a1.sources.r1.interceptors.e1.serializers.s8.name=Ip2a1.sources.r1.interceptors.e1.serializers.s9.name=Ip3a1.sources.r1.interceptors.e1.serializers.s10.name=Starttimea1.sources.r1.interceptors.e1.serializers.s11.name=Endtimea1.sources.r1.interceptors.e1.serializers.s12.name=Srcvpna1.sources.r1.interceptors.e1.serializers.s13.name=Desvpna1.sources.r1.interceptors.e1.serializers.s14.name=Statusa1.sources.r1.interceptors.e1.serializers.s15.name=username#ChannelsA1.channels.memoryChannel.type =memorya1.channels.memoryChannel.capacity= 300a1.channels.memoryChannel.transactionCapacity= 300#SinkA1.sinks.spoolSink.type =Com.hzfi.flume.PatternTestSinka1.sinks.spoolSink.channel= Memorychannel

In the example above, a interceptor regex_extractor is used to perform regular expression pattern recognition of the records in the log in the offline directory, and the records are divided into 15 sub-patterns, which are added to the header of the Flume event respectively.

Sink is a custom patterntestsink with the following code:

 PackageCom.hzfi.flume;ImportOrg.apache.flume.Channel;ImportOrg.apache.flume.Context;Importorg.apache.flume.Event;Importorg.apache.flume.EventDeliveryException;Importorg.apache.flume.Transaction;Importorg.apache.flume.conf.Configurable;ImportOrg.apache.flume.sink.AbstractSink; Public classPatterntestsinkextendsAbstractsinkImplementsConfigurable {@Override PublicStatus process ()throwseventdeliveryexception {Status result=Status.ready; Channel Channel=Getchannel (); Transaction Transaction=channel.gettransaction (); Event Event=NULL; Try{transaction.begin (); Event=Channel.take (); if(Event! =NULL) {String body=NewString (Event.getbody (), "Utf-8"); System.out.println ("----->event Headers ..."); System.out.println ("Header content:[" + event.getheaders (). toString () + "]"); System.out.println ("----->event body ..."); System.out.println ("Body content:[" + body + "]"); } Else{result=Status.backoff;        } transaction.commit (); } Catch(Exception ex) {transaction.rollback (); Throw NewEventdeliveryexception ("Failed to got pattern event:" +event, ex); } finally{transaction.close (); }        returnresult; } @Override Public voidConfigure (Context arg0) {//TODO auto-generated Method Stub    }}

Flume use in Windows environments