使用flume+kafka+storm構建即時日誌分析系統_PHP教程

使用flume+kafka+storm構建即時日誌分析系統

本文只會涉及flume和kafka的結合，kafka和storm的結合可以參考其他部落格
1. flume安裝使用
下載flume安裝包http://www.apache.org/dyn/closer.cgi/flume/1.5.2/apache-flume-1.5.2-bin.tar.gz
解壓$ tar -xzvf apache-flume-1.5.2-bin.tar.gz -C /opt/flume
flume設定檔放在conf檔案目錄下，執行檔案放在bin檔案目錄下。
1）配置flume
進入conf目錄將flume-conf.properties.template拷貝一份，並命名為自己需要的名字
$ cp flume-conf.properties.template flume.conf
修改flume.conf的內容，我們使用file sink來接收channel中的資料，channel採用memory channel，source採用exec source，設定檔如下：

 

  
agent.sources = seqGenSrc
  
agent.channels = memoryChannel
  
agent.sinks = loggerSink
  

  
# For each one of the sources, the type is defined
  
agent.sources.seqGenSrc.type = exec
  
agent.sources.seqGenSrc.command = tail -F /data/mongodata/mongo.log
  
#agent.sources.seqGenSrc.bind = 172.168.49.130
  

  
# The channel can be defined as follows.
  
agent.sources.seqGenSrc.channels = memoryChannel
  

  
# Each sink's type must be defined
  
agent.sinks.loggerSink.type = file_roll
  
agent.sinks.loggerSink.sink.directory = /data/flume
  

  
#Specify the channel the sink should use
  
agent.sinks.loggerSink.channel = memoryChannel
  

  
# Each channel's type is defined.
  
agent.channels.memoryChannel.type = memory
  

  
# Other config values specific to each type of channel(sink or source)
  
# can be defined as well
  
# In this case, it specifies the capacity of the memory channel
  
agent.channels.memoryChannel.capacity = 1000
  
agent.channels.memory4log.transactionCapacity = 100
 

2）運行flume agent
切換到bin目錄下，運行一下命令：
$ ./flume-ng agent --conf ../conf -f ../conf/flume.conf --n agent -Dflume.root.logger=INFO,console
在/data/flume目錄下可以看到產生的記錄檔。

2. 結合kafka
由於flume1.5.2沒有kafka sink，所以需要自己開發kafka sink
可以參考flume 1.6裡面的kafka sink，但是要注意使用的kafka版本，由於有些kafka api不相容的
這裡只提供核心代碼，process()內容。

 

  
Sink.Status status = Status.READY;
 
  


  
Channel ch = getChannel();

  
Transaction transaction = null;

  
Event event = null;

  
String eventTopic = null;

  
String eventKey = null;

  


  
try {

  
transaction = ch.getTransaction();

  
transaction.begin();

  
messageList.clear();

  


  
if (type.equals("sync")) {

  
event = ch.take();

  


  
 if (event != null) {

  
 byte[] tempBody = event.getBody();

  
 String eventBody = new String(tempBody,"UTF-8");

  
 Map headers = event.getHeaders();

  


  
 if ((eventTopic = headers.get(TOPIC_HDR)) == null) {

  
 eventTopic = topic;

  
 }

  


  
 eventKey = headers.get(KEY_HDR);

  


  
 if (logger.isDebugEnabled()) {

  
 logger.debug("{Event} " + eventTopic + " : " + eventKey + " : "

  
 + eventBody);

  
 }

  
 

  
 ProducerData data = new ProducerData

  
 (eventTopic, new Message(tempBody));

  
 

  
 long startTime = System.nanoTime();

  
 logger.debug(eventTopic+"++++"+eventBody);

  
 producer.send(data);

  
 long endTime = System.nanoTime(); 
  
 }

  
} else {

  
long processedEvents = 0;

  
for (; processedEvents < batchSize; processedEvents += 1) {

  
event = ch.take();

  


  
 if (event == null) {

  
 break;

  
 }

  


  
 byte[] tempBody = event.getBody();

  
 String eventBody = new String(tempBody,"UTF-8");

  
 Map headers = event.getHeaders();

  


  
 if ((eventTopic = headers.get(TOPIC_HDR)) == null) {

  
 eventTopic = topic;

  
 }

  


  
 eventKey = headers.get(KEY_HDR);

  


  
 if (logger.isDebugEnabled()) {

  
 logger.debug("{Event} " + eventTopic + " : " + eventKey + " : "

  
 + eventBody);

  
 logger.debug("event #{}", processedEvents);

  
 }

  


  
 // create a message and add to buffer

  
 ProducerData data = new ProducerData

  
 (eventTopic, eventBody);

  
 messageList.add(data);

  
}

  


  
// publish batch and commit.

  
 if (processedEvents > 0) {

  
 long startTime = System.nanoTime(); 
  
 long endTime = System.nanoTime(); 
  
 }

  
}

  


  
transaction.commit();

  
} catch (Exception ex) {

  
String errorMsg = "Failed to publish events";

  
logger.error("Failed to publish events", ex);

  
status = Status.BACKOFF;

  
if (transaction != null) {

  
try {

  
transaction.rollback(); 
  
} catch (Exception e) {

  
logger.error("Transaction rollback failed", e);

  
throw Throwables.propagate(e);

  
}

  
}

  
throw new EventDeliveryException(errorMsg, ex);

  
} finally {

  
if (transaction != null) {

  
transaction.close();

  
}

  
}

  


  
return status; 
 

下一步，修改flume設定檔，將其中sink部分的配置改成kafka sink，如：

 

  
producer.sinks.r.type = org.apache.flume.sink.kafka.KafkaSink
 
  
producer.sinks.r.brokerList = bigdata-node00:9092

  
producer.sinks.r.requiredAcks = 1

  
producer.sinks.r.batchSize = 100

  
#producer.sinks.r.kafka.producer.type=async

  
#producer.sinks.r.kafka.customer.encoding=UTF-8

  
producer.sinks.r.topic = testFlume1
 

type指向kafkasink所在的完整路徑
下面的參數都是kafka的一系列參數，最重要的是brokerList和topic參數

現在重新啟動flume，就可以在kafka的對應topic下查看到對應的日誌

http://www.bkjia.com/PHPjc/1109725.htmlwww.bkjia.comtruehttp://www.bkjia.com/PHPjc/1109725.htmlTechArticle使用flume+kafka+storm構建即時日誌分析系統 本文只會涉及flume和kafka的結合，kafka和storm的結合可以參考其他部落格 1. flume安裝使用 下載flume安裝...

