flume 整合kafka

標籤：led   time   conf   方式   kafka   require   red   cep   char   

 背景：系統的資料量越來越大，日誌不能再簡單的檔案的儲存，如此日誌將會越來越大，也不方便尋找與分析，綜合考慮下使用了flume來收集日誌，收集日誌後向kafka傳遞訊息，下面給出具體的配置

# The configuration file needs to define the sources,# the channels and the sinks.# Sources, channels and sinks are defined per agent,   # in this case called ‘agent‘agent.sources = r1agent.channels = c1agent.sinks = s1# For each one of the sources, the type is definedagent.sources.r1.type = netcatagent.sources.r1.bind = localhostagent.sources.r1.port = 10245agent.sources.r1.charset = UTF-8# The channel can be defined as follows.agent.sources.r1.channels = c1# Each sink‘s type must be definedagent.sinks.s1.type = org.apache.flume.sink.kafka.KafkaSinkagent.sinks.s1.topic = testagent.sinks.s1.brokerList = ｉｐ:9092agent.sinks.s1.requiredAcks = 1agent.sinks.s1.batchSize = 20agent.sinks.s1.channel = c1# Each channel‘s type is defined.agent.channels.c1.type = memory# Other config values specific to each type of channel(sink or source)# can be defined as well# In this case, it specifies the capacity of the memory channelagent.channels.c1.capacity = 100

啟動方式：

　　 bin/flume-ng agent --conf conf --conf-file conf/kafka.conf --name agent -Dflume.root.logger=INFO,console

再啟動之前一定要先啟動kafka，這裡可能會有一個錯誤

　　

Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired

這個是因為預設情況下kafka是廣播的localhost，所以如果不是同一個機器需要修改下配置

advertised.listeners=PLAINTEXT://ｉｐ:9092把預設的localhost替換成ＩＰ地址　重新啟動下就可以了．

flume 整合kafka