Flume Ng+kafka Integration of Flume Learning notes

Flume ng cluster +kafka cluster Integration:

Modify the Flume configuration file (flume-kafka-server.conf) and let Sink connect to the Kafka

HADOOP1:

#set Agent namea1.sources = r1a1.channels = C1a1.sinks = K1#set Channela1.channels.c1.type = memorya1.channels.c1.capacit y = 1000a1.channels.c1.transactioncapacity = 100# other Node,nna to Nnsa1.sources.r1.type = Avroa1.sources.r1.bind = Hadoo P1a1.sources.r1.port = 52020a1.sources.r1.interceptors = I1a1.sources.r1.interceptors.i1.type = Statica1.sources.r1.interceptors.i1.key = Collectora1.sources.r1.interceptors.i1.value = Hadoop1a1.sources.r1.channels = C1#set sink to HDFsa1.sinks.k1.type= Org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.topic = ScalaTopica1.sinks.k1.brokerList = Hadoop1 : 9092a1.sinks.k1.requiredacks = 1a1.sinks.k1.batchsize =a1.sinks.k1.channel=c1

HADOOP2:

#set Agent namea1.sources = r1a1.channels = C1a1.sinks = K1#set Channela1.channels.c1.type = memorya1.channels.c1.capacit y = 1000a1.channels.c1.transactioncapacity = 100# other Node,nna to Nnsa1.sources.r1.type = Avroa1.sources.r1.bind = Hadoo P2a1.sources.r1.port = 52020a1.sources.r1.interceptors = I1a1.sources.r1.interceptors.i1.type = Statica1.sources.r1.interceptors.i1.key = Collectora1.sources.r1.interceptors.i1.value = Hadoop2a1.sources.r1.channels = C1#set sink to HDFsa1.sinks.k1.type= Org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.topic = ScalaTopica1.sinks.k1.brokerList = Hadoop2 : 9092a1.sinks.k1.requiredacks = 1a1.sinks.k1.batchsize =a1.sinks.k1.channel=c1

Cluster testing:

1. Start Zookeeper (HADOOP1,HADOOP2,HADOOP3)
2. Start Kafka Server and Consumer (HADOOP1,HADOOP2)
3. Start Flume Server (HADOOP1,HADOOP2): Flume-ng agent--conf conf--conf-file/usr/local/flume/conf/flume-kafka-server.conf --name A1-dflume.root.logger=info,console
4. Start Flume Client (HADOOP3): Flume-ng agent--conf conf--conf-file/usr/local/flume/conf/flume-client.conf--name agent1- Dflume.root.logger=info,console
5. Append a log record on the HADOOP3
6. Kafka consumer receive the record, and the test is complete.

HADOOP3:

HADOOP1:

Test completed, so Flume+kafka integration, that is, Flume+kafka+spark streaming real-time log analysis system was born.

Flume Ng+kafka Integration of Flume Learning notes