Apache Flume Agent Installation

1, Flume agent installation (using SPOOLDIR mode to obtain system, application and other log information)

Note: Install with Jyapp user

When a single virtual machine deploys multiple Java applications and needs to deploy multiple flume-agent for monitoring,

The following configuration files need to be adjusted:

The Spool_dir parameter in a flume-agent/conf/app.conf

Jmxremote.port parameters in B-flume-agent/conf/flume-env.sh

Filenamepattern parameters in C logback.xml

Dir parameter in D flume_spooldir.sh

1.1 Installation Environment

System:

CentOS Release 6.6

Software:

Flume-agent.tar.gz

flume-agent. tar.gz and flume_spooldir.sh file download path: ftp://172.21.100.61/pro/flume/flume-agent

1.2 Installation Steps1.2.1 Deploying flume Agent

Specific scripts (Jyapp users):

Cd/home/jyapp mkdir Flume_logs TAR-ZXVF flume-agent. tar.gz CD flume-agent #修改flume环境变量 VI conf/flume-env.sh Specific configuration analysis: (Note: Red needs to be modified) ------------------------------------------------------ Export java_opts= "-xms100m-xmx100m-dflume.root.logger=info,logfile-dflume.monitoring.type=http- dflume.monitoring.port=34545-dcom.sun.management.jmxremote-djava.rmi.server.hostname= native IP - Dcom.sun.management.jmxremote.port=9999-dcom.sun.management.jmxremote.authenticate=false- Dcom.sun.management.jmxremote.ssl=false " ------------------------------------------------------------------------------------- #修改agent脚本 VI conf/app.conf Specific configuration analysis: (Note: Red needs to be modified) ------------------------------------------------------ App.sources = R1 App.sinks = K1 K2 App.channels = C1 App.sources.r1.type = Spooldir App.sources.r1.spoolDir =/home/jyapp/flume_logs #定义监控的日志目录 App.sources.r1.deletePolicy = Immediate App.sources.r1.ignorePattern = ^ (.) *\\.tmp$ #忽略. tmp file App.sources.r1.deserializer = MULTILINE #使用多行反序列化器 App.sources.r1.deserializer.maxLineLength = 60000 App.sources.r1.interceptors = I1 i2 i3 App.sources.r1.interceptors.i1.type = Timestamp #设置时间戳 App.sources.r1.interceptors.i2.type = static App.sources.r1.interceptors.i2.key = Flume.client.ip App.sources.r1.interceptors.i2.value = Client IP address #设置IP地址 App.sources.r1.interceptors.i3.type = static App.sources.r1.interceptors.i3.key = Flume.client.system.flag App.sources.r1.interceptors.i3.value = Client system marked #设置系统标示, such as S001 App.channels.c1.type = Memory app.channels.c1.capacity = 1000 app.channels.c1.transactionCapacity = 1000 App.channels.c1.keep-alive = 3 App.channels.c1.byteCapacityBufferPercentage = 30 App.channels.c1.byteCapacity = 104857600 App.sinks.k1.type = Avro App.sinks.k1.hostname = 172.19.100.92 App.sinks.k1.port = 44466 App.sinks.k2.type = Avro App.sinks.k2.hostname = 172.19.100.102 App.sinks.k2.port = 44466 App.sources.r1.channels = C1 App.sinks.k1.channel = C1 App.sinks.k2.channel = C1 app.sinkgroups = G1 App.sinkgroups.g1.sinks = K1 K2 App.sinkgroups.g1.processor.type = Load_balance App.sinkgroups.g1.processor.backoff = True App.sinkgroups.g1.processor.selector = Round_robin ------------------------------------------------------------------------------------- Bulk Substitution Command Reference: #修改flume-env.sh Sed-i ' s/172.18.100.159/172.19.100.73/g;s/S000/S001/g '/home/jyapp/ flume-agent/conf/flume-env.sh #修改app. conf Sed-i ' s/172.18.100.159/172.19.100.73/g;s/S000/S001/g '/home/jyapp/ Flume-agent/conf/app.conf Background Boot Agent Cd/home/jyapp/flume-agent Bin/flume-ng agent--conf conf--conf-file conf/app.conf--name App & Stop agent, use the KILL command

Apache Flume agent installation