Flume-kafka Deployment Summary _flume

Deployment Readiness

Configure the Log collection system (FLUME+KAFKA), version:

apache-flume-1.8.0-bin.tar.gz
kafka_2.11-0.10.2.0.tgz

Suppose the Ubuntu system environment is deployed in three working nodes:

192.168.0.2
192.168.0.3
192.168.0.4

Flume Configuration Instructions

Suppose Flume's working directory is in/usr/local/flume,
Monitor a log file (such as/tmp/testflume/chklogs/chk.log),
Then new configuration:/usr/local/flume/conf/flume-kafka.conf

Flume2kafkaagent.sources=mysource
Flume2kafkaagent.channels=mychannel
flume2kafkaagent.sinks=mysink

flume2kafkaagent.sources.mysource.type=exec
Flume2kafkaagent.sources.mysource.channels=mychannel
flume2kafkaagent.sources.mysource.command=tail-f/tmp/ Testflume/chklogs/chk.log

Flume2kafkaagent.sinks.mysink.channel=mychannel
Flume2kafkaagent.sinks.mysink.type=org.apache.flume.sink.kafka.kafkasink
flume2kafkaagent.sinks.mysink.kafka.bootstrap.servers=192.168.0.2:9092,192.168.0.3:9092,192.168.0.4:9092
Flume2kafkaagent.sinks.mysink.kafka.topic=apilog
flume2kafkaagent.sinks.mysink.kafka.batchsize=20
Flume2kafkaagent.sinks.mysink.kafka.requiredacks=1

flume2kafkaagent.channels.mychannel.type=memory
flume2kafkaagent.channels.mychannel.capacity=30000
flume2kafkaagent.channels.mychannel.transactioncapacity=100

All three nodes execute the start command:

hadoop@1:/usr/local/flume$ bin/flume-ng agent-c conf-f conf/flume-kafka.conf-n flume2kafkaagent

Kafka Configuration Instructions

Suppose Kafaka's working directory is in/usr/local/kafka,
Modify/usr/local/kafka/config/server.properties The following:

Broker.id=0   # Broker ID, each node must be different, for example, three nodes are 0, 1, 2
port=9092
advertised.listeners=plaintext://your_ local_ip:9092 #当前节点ip
zookeeper.connect=192.168.0.2:2181,192.168.0.3:2181,192.168.0.4:2181

Each node starts Zookeeper:

hadoop@1:/usr/local/kafka$ bin/zookeeper-server-start.sh

Each node starts Kafka:

hadoop@1:/usr/local/kafka$ bin/kafka-server-start.sh

To view the Java process through JPS:

hadoop@1:/usr/local/kafka$ JPS
20225 JPS
13286 Kafka 15421 Quorumpeermain

Validate test simulation build log, write/tmp/testflume/chklogs/chk.log, such as:

#!/bin/bash

chkdir=/tmp/testflume/chklogs/
mkdir-p $chkdir for

((i=0; i<10000; i++));
Do
  dt=$ (date ' +%y%m%d%h:%m:%s ')
  echo "Current datetime: $dt" >> $chkdir/chk.log sleep
  0.5
Done

Start Kafka Consumer, view data receive record:

hadoop@1:/usr/local/kafka$ bin/kafka-console-consumer.sh--bootstrap-server 192.168.0.2:9092,192.168.0.3:2181,192.168.0.4:9092--from-beginning--topic apilog current
datetime:20171012 17:42:05 current
datetime:20171012 17:42:06 current
datetime:20171012 17:42:07 current
datetime: 20171012 17:42:08 Current
datetime:20171012 17:42:09 current
datetime:20171012 17:42:10 current
datetime : 20171012 17:42:11
datetime:20171012 17:42:12 current
datetime:20171012 17:42:13

This shows that Kafka has been able to continuously receive log data.