Flume Monitoring hive log files

• Flume Monitoring hive log files

One: Flume Monitor hive Log 1.1 case requirements:

1. 实时监控某个日志文件，将数据收集到存储hdfs 上面， 此案例使用exec source ，实时监控文件数据，使用Memory Channel 缓存数据，使用HDFS Sink 写入数据2. 此案例实时监控hive 日志文件，放到hdfs 目录当中。hive 的日志目录是hive.log.dir = /home/hadoop/yangyang/hive/logs

1.2 Create a collection directory above HDFs:

1.3 Copy the jar package required for flume

cd /home/hadoop/yangyang/hadoop/cp -p share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.6.jar /home/hadoop/yangyang/flume/lib/cp -p share/hadoop/common/hadoop-common-2.5.0-cdh5.3.6.jar /home/hadoop/yangyang/flume/lib/cp -p share/hadoop/tools/lib/commons-configuration-1.6.jar /home/hadoop/yangyang/flume/lib/cp -p share/hadoop/tools/lib/hadoop-auth-2.5.0-cdh5.3.6.jar /home/hadoop/yangyang/flume/lib/

1.4 Configure a new file for the Hive-test.properties file:

cp -p test-conf.properties hive-conf.properties

Vim Hive-conf.properties

# example.conf:a Single-node Flume configuration# Name The components in this agenta2.sources = R2a2.sinks = K2a2.channel s = c2# describe/configure the Sourcea2.sources.r2.type = Execa2.sources.r2.command = Tail-f/home/hadoop/yangyang/hive/ Logs/hive.loga2.sources.r2.bind = Namenode01.hadoop.coma2.sources.r2.shell =/bin/bash-c # Describe the Sinka2.sinks.k2.type = HDFs A2.sinks.k2.hdfs.path = hdfs://namenode01.hadoop.com:8020/flume/%y%m/% Da2.sinks.k2.hdfs.fileType = DataStreama2.sinks.k2.hdfs.writeFormat = Texta2.sinks.k2.hdfs.batchSize = 10# Set level two catalog by hour cut A2.sinks.k2.hdfs.round = Truea2.sinks.k2.hdfs.roundValue = 1a2.sinks.k2.hdfs.roundunit = hour# Set file rollback condition A2.sinks.k2.hdfs.rollInterval = 60a2.sinks.k2.hdfs.rollsize = 128000000a2.sinks.k2.hdfs.rollcount = 0a2.sinks.k2.hdfs.uselocaltimestamp = Truea2.sinks.k2.hdfs.minBlockReplicas = # Use a channel which buffers events in mem Orya2.channels.c2.type = memorya2.channels.c2.capacity = 1000a2.channels.c2.transactioncapacity = 100# Bind the Source and sink to the channela2.sources.r2.channels = C2a2.sinks.k2.channel = C2  

1.5 Running agent Processing

1.6 Log log file test written to hive:

cd /home/hadoop/yangyang/hive/logsecho "111" >> hive.log 每隔一段时间执行上面的命令测试

1.7 Go to HDFs to see:

Flume Monitoring hive log files