Flume Introduction and use (iii) Kafka installation of--kafka sink consumption data

The previous introduction of how to use thrift source production data, today describes how to use Kafka sink consumption data.

In fact, in the Flume configuration file has been set up with Kafka sink consumption data

Agent1.sinks.kafkaSink.type =Org.apache.flume.sink.kafka.KafkaSinkagent1.sinks.kafkaSink.topic=TRAFFIC_LOGagent1.sinks.kafkaSink.brokerList=10.208.129.3:9092,10.208.129.4:9092,10.208.129.5:9092agent1.sinks.kafkaSink.metadata.broker.list=10.208.129.3:9092,10.208.129.4:9092,10.208.129.5:9092Agent1.sinks.kafkaSink.producer.type=SyncAgent1.sinks.kafkaSink.serializer.class=Kafka.serializer.DefaultEncoderagent1.sinks.kafkaSink.channel= Memorychannel

Then when the flume channel receives the data, it will actively send the data event to the Kafka broker based on the configuration file, so as long as the Kafka is installed, the receipt can be consumed.

Step 1:download The Code

Download and unzip the installation package

> tar-xzf kafka_2.11-0.10.0.0.tgz

> CD kafka_2.11-0.10.0.0

Step 2:start the server

Kafka is based on Zookeeperl for distributed collaboration, so start the zookeeper first:

> %zookeeper_home%/bin /zkserver.sh start

In the configuration file server.properties, remove the previous comment from the following sentence and start the Kafka server

> #listeners =plaintext://:9092

> bin/kafka-server-start.sh config/server.properties

Next, start the other two brokers:

> CP config/server.properties Config/server-1.properties

> CP config/server.properties Config/server-2.properties

Modify configuration file, Broker.id cannot repeat

Config/server-1.properties:    broker.id=1
Config/server-2.properties:    broker.id=2

Step 3:create a topic

Create a broker with a traffic_log theme, with a replication factor of 3 (because there are 3 Kafka server clusters), the number of partitions is 1

> bin/kafka-topics.sh--create--zookeeper 10.208.129.4:2181--replication-factor 3--partitions 1--topic Traffic_log

Step 5:start a consumer > bin/kafka-console-consumer.sh--zookeeper 10.208.129.4: 2181--topic traffic_log --from-beginning

If you see the thrift source output before the terminal, then the whole flume+kafka is running through.

Flume and Kafka have been introduced here, and the next article will cover the differences between Kafka and Flume and Kafka.

Flume Introduction and use (iii) Kafka installation of--kafka sink consumption data