Flume instance one, listening directory log upload to other server

I. Introduction of FLUME-NG

Please refer to the official documentation: http://flume.apache.org/FlumeUserGuide.html

Second, examples

Requirements Description: A directory needs to be monitored and automatically uploaded to the server and encrypted during transmission.

Overall solution: N client-agent-->server-agent

Client-agent:

A1.sources =R1a1.channels=c1a1.sinks=K1#sourcea1.sources.r1.type=Spooldira1.sources.r1.channels=C1a1.sources.r1.basenameHeader=true#a1. Sources.r1.ignorePattern= .+\.log$a1.sources.r1.buffermaxlinelength= 1048576a1.sources.r1.batchSize= 5000#拦截器a1. Sources.r1.interceptors=i1 i2 I3a1.sources.r1.interceptors.i1.type=StaticA1.sources.r1.interceptors.i1.key=Ida1.sources.r1.interceptors.i2.type=StaticA1.sources.r1.interceptors.i2.key=Keya1.sources.r1.interceptors.i3.type=com.landray.behavior.interceptor.behaviorclientserurityhdfsinterceptor$builder#file channela1.channels=C1a1.channels.c1.type=Filea1.channels.c1.checkpointDir= ./Checkpointa1.channels.c1.dataDirs= ./Data#sinka1.sinks.k1.type=Avroa1.sinks.k1.channel=c1a1.sinks.k1.compression-type =deflatea1.sinks.k1.compression-leve = 9A1.sinks.k1.batch-size = 5000A1.sinks.k1.hostname= http://test.com.cnA1.sinks.k1.port = 5281a1.sinks.k1.request-timeout = 20000#user define# need to upload the log directory A1.sources.r1.spoolDir=d:/flume_tes/source# Customer Unique IDa1.sources.r1.interceptors.i1.value= 123456#秘钥a1. Sources.r1.interceptors.i2.value= key

Server-agent:

A2.sources =R2a2.channels=c2a2.sinks=K2#sourcea2.sources.r2.type=Avroa2.sources.r2.channels=c2a2.sources.r2.compression-type =Deflatea2.sources.r2.bind=Localhosta2.sources.r2.port= 5281a2.sources.r2.interceptors=I1a2.sources.r2.interceptors.i1.type=Com.landray.behavior.interceptor.behaviorserverserurityinterceptor$buildera2.channels=C2a2.channels.c2.type=Filea2.channels.c2.checkpointDir= ./Checkpointa2.channels.c2.dataDirs= ./dataa2.channels.c2.transactionCapacity= 20000#a2. Channels.c2.type=memory#a2.channels.c2.capacity= 1000000#a2. Channels.c2.transactionCapacity= 20000#a2. Channels.c2.byteCapacityBufferPercentage= 20#default80%#a2. Channels.c2.byteCapacity= 800000A2.sinks.k2.type=Com.landray.behavior.sink.BehaviorRollingFileSinka2.sinks.k2.channel=c2#no Checka2.sinks.k2.sink.rollInterval= 0a2.sinks.k2.sink.batchSize= 20000#user define#windowsa2.sinks.k2.sink.directory= d:/behavior/logs#linux#a2.sinks.k2.sink.directory=/home/nemo/behavior

Third, start the script

Directory:

Catalogue Description:

Checkpoint: Set the checkpoint in the flume

Conf is configured with: Log4j.properties

Data: It's flume's memory.

Libs:flume boot-up jar package

Source.conf: Configuration file

　

The official website gives the command start mode, the comparison is not flexible, here are the Linux and Windows version of two start mode

Linux:

#!/bin/shjava_opts=-xmx1024m$java_home/bin/java $JAVA _opts-dlog4j.configuration=file:./conf/ LOG4J.PROPERTIES-CP "./lib/*" Org.apache.flume.node.Application--conf-file./target.conf--name A2

Windows:

@echo onset flume_home=%~%~dp0rem Configure the JAVA environment variable set java_home=%flume_home%. \jdk1.6set PATH=%java_home%%java_home%-%java_opts%-dlog4j.configuration= File:./conf/log4j.properties-cp.;. /lib/*  org.apache.flume.node.Application

--conf-file./target_hdfs.conf--name A2pause

Flume instance one, listening directory log upload to other server