Flume Integrated Kafka

First, the demand

• Use flume to capture the file information under Linux and pass it into the Kafka cluster.
• Environment ready Zookeeper cluster and Kafka cluster are installed well.

Second, the configuration flume

• Download Flume website. The blogger himself is using flume1.6.0.
• Official Address http://flume.apache.org/download.html
• Unzip. TAR-ZXVF apache-flume-1.6.0-bin.tar.gz-c/usr/apps/
• Create a flume configuration file.
• Cd/usr/apps/flume/apache-flume-1.6.0-bin/conf
• Vim exec.conf flume configuration file as follows

1A1.sources = R12A1.channels = C13A1.sinks = K14A1.sources.r1.type = Exec5A1.sources.r1.command = Tail-f/usr/test/click_log/1.log6A1.sources.r1.channels = C17A1.channels.c1.type=memory8a1.channels.c1.capacity=100009a1.channels.c1.transactioncapacity=100Ten#固定的不需要修改 OneA1.sinks.k1.type = Org.apache.flume.sink.kafka.KafkaSink A#消息的主题 -A1.sinks.k1.topic = Logmq -#kafka Broker Cluster theA1.sinks.k1.brokerList = bhz136:9092,bhz137:9092,bhz139:9092 -A1.sinks.k1.requiredAcks = 1 -A1.sinks.k1.batchSize = 20 -A1.sinks.k1.channel = C1

Third, start flume connected to Kafka

• ./bin/flume-ng agent-n a1-c conf-f conf/exec.conf-dflume.root.logger=info,console

Iv. Start Kafka consumer consumption data

• kafka-console-consumer.sh--zookeeper bhz136:2181--from-beginning--topic logmq

V. Run the test procedure. The test results are as follows:

Note: The 1.log file is a test file generated by the following script

  1for ((i=0;i<=50000;i++));   2          do echo "message-" + $i >>/usr/test/click_log/1.log;   3 Done

Flume Integrated Kafka