Apache Flume Agent Installation

Source: Internet
Author: User
Tags tmp file

1, Flume agent installation (using SPOOLDIR mode to obtain system, application and other log information)

Note: Install with Jyapp user

When a single virtual machine deploys multiple Java applications and needs to deploy multiple flume-agent for monitoring,

The following configuration files need to be adjusted:

The Spool_dir parameter in a flume-agent/conf/app.conf

Jmxremote.port parameters in B-flume-agent/conf/flume-env.sh

Filenamepattern parameters in C logback.xml

Dir parameter in D flume_spooldir.sh

1.1 Installation Environment

System:

CentOS Release 6.6

Software:

Flume-agent.tar.gz

flume-agent. tar.gz and flume_spooldir.sh file download path: ftp://172.21.100.61/pro/flume/flume-agent

1.2 Installation Steps1.2.1 Deploying flume Agent

Specific scripts (Jyapp users):

Cd/home/jyapp

mkdir Flume_logs

TAR-ZXVF flume-agent. tar.gz

CD flume-agent

#修改flume环境变量

VI conf/flume-env.sh

Specific configuration analysis: (Note: Red needs to be modified)

------------------------------------------------------

Export java_opts= "-xms100m-xmx100m-dflume.root.logger=info,logfile-dflume.monitoring.type=http- dflume.monitoring.port=34545-dcom.sun.management.jmxremote-djava.rmi.server.hostname= native IP - Dcom.sun.management.jmxremote.port=9999-dcom.sun.management.jmxremote.authenticate=false- Dcom.sun.management.jmxremote.ssl=false "

-------------------------------------------------------------------------------------

#修改agent脚本

VI conf/app.conf

Specific configuration analysis: (Note: Red needs to be modified)

------------------------------------------------------

App.sources = R1

App.sinks = K1 K2

App.channels = C1

App.sources.r1.type = Spooldir

App.sources.r1.spoolDir =/home/jyapp/flume_logs #定义监控的日志目录

App.sources.r1.deletePolicy = Immediate

App.sources.r1.ignorePattern = ^ (.) *\\.tmp$ #忽略. tmp file

App.sources.r1.deserializer = MULTILINE #使用多行反序列化器

App.sources.r1.deserializer.maxLineLength = 60000

App.sources.r1.interceptors = I1 i2 i3

App.sources.r1.interceptors.i1.type = Timestamp #设置时间戳

App.sources.r1.interceptors.i2.type = static

App.sources.r1.interceptors.i2.key = Flume.client.ip

App.sources.r1.interceptors.i2.value = Client IP address #设置IP地址

App.sources.r1.interceptors.i3.type = static

App.sources.r1.interceptors.i3.key = Flume.client.system.flag

App.sources.r1.interceptors.i3.value = Client system marked #设置系统标示, such as S001

App.channels.c1.type = Memory

app.channels.c1.capacity = 1000

app.channels.c1.transactionCapacity = 1000

App.channels.c1.keep-alive = 3

App.channels.c1.byteCapacityBufferPercentage = 30

App.channels.c1.byteCapacity = 104857600

App.sinks.k1.type = Avro
App.sinks.k1.hostname = 172.19.100.92
App.sinks.k1.port = 44466

App.sinks.k2.type = Avro
App.sinks.k2.hostname = 172.19.100.102
App.sinks.k2.port = 44466

App.sources.r1.channels = C1

App.sinks.k1.channel = C1

App.sinks.k2.channel = C1

app.sinkgroups = G1
App.sinkgroups.g1.sinks = K1 K2
App.sinkgroups.g1.processor.type = Load_balance
App.sinkgroups.g1.processor.backoff = True
App.sinkgroups.g1.processor.selector = Round_robin

-------------------------------------------------------------------------------------

Bulk Substitution Command Reference:

#修改flume-env.sh

Sed-i ' s/172.18.100.159/172.19.100.73/g;s/S000/S001/g '/home/jyapp/ flume-agent/conf/flume-env.sh

#修改app. conf

Sed-i ' s/172.18.100.159/172.19.100.73/g;s/S000/S001/g '/home/jyapp/ Flume-agent/conf/app.conf

Background Boot Agent

Cd/home/jyapp/flume-agent

Bin/flume-ng agent--conf conf--conf-file conf/app.conf--name App &

Stop agent, use the KILL command


Apache Flume agent installation

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.