Flume captures log data in real time and uploads it to Kafka __flume

Flume real-time crawl log data and upload to Kafka

1.Linux OK zookeeper is configured, start zookeeper first

sbin/zkserver.sh start

(sbin/zkserver.sh Status View startup state) JPS can check to see Le process as Quorumpeermain

2. Start Kafka,zookeeper need to start before Kafka

bin/kafka-server-start.sh Config/server.properties &

3. Start consumer receive log

bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic topic-004

(Before you can create topic; bin/kafka-topics.sh--create--zookeeper 127.0.0.1:2181--replication-factor 1--partitions 1--topic mytop Ic

4. View your deployment of the webserver log, when the page is operating, to see the log changes, to determine the changes in the log to be monitored log. For example, my own is:

Then I need to monitor the directory for Usr/local/nginx/access.log

5. Start Flume. Sbin/flume Start Source (multiple application after start up process)

Modify configuration file: Conf/flume-source-conf.properties the file path for monitoring is:

6. Operate the Web site, generate logs, and view Kafka under consumer to monitor the resulting log.