Use flume-ng for log collection

I. installation environment
AGENT: 192.168.7.101
HDFS: 192.168.7.70 (namenode)
192.168.7.71 (datanode)
192.168.7.72 (datanode)
192.168.7.73 (datanode)
Operating System: centos 6.3 x86_64
Required software packages: jdk-1.7.0_65-fcs.x86_64 flume-ng-1.5.0 flume-ng-agent-1.5.0 hadoop-2.3.0 + cdh5.1.0

CAT/etc/hosts
192.168.7.70 cdh1
192.168.7.71 cdh2
192.168.7.72 cdh3
192.168.7.73 cdh4

2. Configure flume-ng-agent
1. involved log files:
/Home/logs/bizservice. Log
/Home/logs/base. Log
/Home/logs/agent. Log
/Home/logs/thirdser. Log

2. Install flume-ng-agent
Yum-y install flume-NG * jdk-1.7.0_65-fcs.x86_64

3. Configure/etc/profile. d/Java. SH and take effect
Echo 'java _ home =/usr/Java/latest '>/etc/profile. d/Java. Sh
Echo 'path = $ java_home/bin: $ path'>/etc/profile. d/Java. Sh
Echo 'export java_home path'>/etc/profile. d/Java. Sh
Source/etc/profile. d/Java. Sh

4. Use template. conf and flume. Sh to generate/etc/flume-ng/CONF/flume. conf
CAT/tmp/template. conf

# Name the components on this agent
AA. Sources = rr
AA. sinks = kk
AA. Channels = cc

# Describe/configure the source
AA. Sources. RR. type = Exec
AA. Sources. RR. Command = tail-F agent1
AA. Sources. RR. Channels = cc
AA. Sources. RR. Bind = 0.0.0.0
AA. Sources. RR. Port = 4141


# Describe the sink
AA. sinks. KK. type = HDFS
AA. sinks. KK. Channel = cc
AA. sinks. KK. HDFS. Path = HDFS: // cdh1: 8020/flume/agent2/% Y-% m-% d/% H % m/% s
AA. sinks. KK. HDFS. fileprefix = agent2 % {Host}
AA. sinks. KK. HDFS. Round = true
AA. sinks. KK. HDFS. roundvalue = 10
AA. sinks. KK. HDFS. roundunit = minute
AA. sinks. KK. HDFS. uselocaltimestamp = true

# Use a channel which buffers events in memory
AA. channels. CC. type = memory
AA. channels. CC. capacity = 1000
AA. channels. CC. transactioncapacity = 100
# Bind the source and sink to the Channel
AA. Sources. RR. Channels = cc
AA. sinks. KK. Channel = cc

Cat flume. Sh
#! /Bin/bash
Source/etc/profile;
CD/etc/flume-ng/CONF/
Object = 'Find/home/logs/-name *. log-type F | egrep "bizservice. log | base. log | agent. log | thirdser. log "| awk-F'/'' {print $ NF} '| SED's /. log // g''
Find/home/logs/-name *. log-type F | egrep "bizservice. log | base. log | agent. log | thirdser. log "| tr ''' \ n'>/tmp/temp
Rm-F flume. conf
For I in $ object
Do
\ CP-RF/tmp/template. conf "$ I". conf
Pathe = 'grep "$ I"/tmp/temp'
Sed-I's/AA/'$ I' _ A1/G' "$ I". conf
Sed-I's/RR/'$ I' _ R1/G' "$ I". conf
Sed-I's/KK/'$ I' _ k1/G' "$ I". conf
Sed-I's/CC/'$ I' _ C1/G' "$ I". conf
Sed-I's # agent1 # '$ path' # G' "$ I". conf
Sed-I's/agent2/'$ I'/G' "$ I". conf
Cat "$ I". conf> flume. conf
Rm-F "$ I". conf
Done
Rm-F temp

5. Modify/etc/init. d/flume-ng-agent
Default_flume_agent_name = 'cat/etc/flume-ng/CONF/flume. conf | grep A1 | awk-f. '{print $1}' | sort-u | tr' \ n ''''

For flume_agent in $ flume_agent_name
Do
/Bin/su-S/bin/bash-c "/bin/bash-C 'echo \$ \> >$ {flume_pid_file} & exec $ {exec_path} Agent -- Conf $ flume_conf_dir -- conf-File $ flume_conf_file -- name $ flume_agent >>$ {flume_log_dir}/flume. init. log 2> & 1' & "$ flume_user
Done

6. Start flume-ng-agent
/Etc/init. d/flume-ng-Agent start

7. view the log/var/log/flume-ng/flume. log, and the data written to HDFS is displayed.

03 September 20 14:09:39, 445 info [hdfs-bizservice_k1-call-runner-9] (Org. apache. flume. sink. HDFS. bucketwriter $8. call: 673)-Renaming HDFS: // cdh1: 8020/flume/bizservice/14-09-03/1400/00/bizservice.1409724409419.tmp to HDFS: // cdh1: 8020/flume/bizservice/14-09-03/1400/00/bizservice.1409724409419
03 August 14 14:09:39, 470 info [sinkrunner-pollingrunner-defaultsinkprocessor] (Org. apache. flume. sink. HDFS. bucketwriter. open: 261)-creating HDFS: // cdh1: 8020/flume/bizservice/14-09-03/1400/00/bizservice.1409724409420.tmp

This article is from the "past with the wind" blog and will not be reposted!

Use flume-ng for log collection