Flume + Kafka Basic Configuration

apache-flume1.6 Sink Default Support Kafka

• [FLUME-2242]-FLUME Sink and Source for Apache Kafka

The official example is very intimate, you can directly run =,=, detailed configuration after a slow look.

A1.channels = Channel1
A1.sources = src-1
A1.sinks = K1

A1.sources.src-1.type = Spooldir
A1.sources.src-1.channels = Channel1
A1.sources.src-1.spooldir =/opt/flumespool/
A1.sources.src-1.fileheader = False

#a1. Sources.src-1.type = Exec
#a1. Sources.src-1.command = Tail-f/opt/flumespool/abc.log
#a1. sources.src-1.channels = Channel1

A1.sinks.k1.type = Org.apache.flume.sink.kafka.KafkaSink
A1.sinks.k1.topic = Testlog
A1.sinks.k1.brokerList = 1.1.1.11:9092,1.1.1.12:9092
A1.sinks.k1.requiredAcks = 1
A1.sinks.k1.batchSize = 20
A1.sinks.k1.channel = Channel1

A1.channels.channel1.type = Org.apache.flume.channel.kafka.KafkaChannel
A1.channels.channel1.capacity = 10000
a1.channels.channel1.transactionCapacity = 1000
a1.channels.channel1.brokerlist=1.1.1.11:9092,1.1.1.12:9092
A1.channels.channel1.topic=testlog
a1.channels.channel1.zookeeperconnect=1.1.1.13:2181

Flume + Kafka Basic Configuration