flume 日誌匯入elasticsearch

標籤：匯入   timestamp   ==   serialize   elastic   cal   啟動   amp   batch   

Flume配置

。

flume產生的資料結構

<span style="font-size:18px;">"_index" : "logstash-2013.01.07","_type" : "tms_jboss_syslog","_id" : "a_M9X_0YSpmE7A_bEzIFiw","_score" : 1.0, "_source" : {"@source":"file://localhost.localdomain/tmp/logstash_test.log","@tags":[],"@fields":{},"@timestamp":"2013-01-07T10:53:50.941Z","@source_host":"localhost.localdomain","@source_path":"/tmp/logstash_test.log","@message":"[2013-01-05 11:02:19,969] packBoxNumber eq 00004000000044043412 createdOffice eq VIP_BJ:;null","@type":"tms_jboss_syslog"}</span>

flume設定檔

agent.sources = tailagent.channels = memoryChannelagent.channels.memoryChannel.type = memoryagent.sources.tail.channels = memoryChannelagent.sources.tail.type = execagent.sources.tail.command = tail -F /home/hadoop/flume/conf/es_log/es_log.logagent.sources.tail.interceptors=i1 i2 i3agent.sources.tail.interceptors.i1.type=regex_extractoragent.sources.tail.interceptors.i1.regex = (\\w.*):(\\w.*):(\\w.*)\\sagent.sources.tail.interceptors.i1.serializers = s1 s2 s3agent.sources.tail.interceptors.i1.serializers.s1.name = sourceagent.sources.tail.interceptors.i1.serializers.s2.name = typeagent.sources.tail.interceptors.i1.serializers.s3.name = src_pathagent.sources.tail.interceptors.i2.type=org.apache.flume.interceptor.TimestampInterceptor$Builderagent.sources.tail.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builderagent.sources.tail.interceptors.i3.hostHeader = hostagent.sinks = elasticsearchagent.sinks.elasticsearch.channel = memoryChannelagent.sinks.elasticsearch.type=org.apache.flume.sink.elasticsearch.ElasticsearchSinkagent.sinks.elasticsearch.batchSize=100agent.sinks.elasticsearch.hostNames=127.0.0.1:9300agent.sinks.k1.indexType = bar_typeagent.sinks.elasticsearch.indexName=logstashagent.sinks.elasticsearch.clusterName=elasticsearchagent.sinks.elasticsearch.serializer=org.apache.flume.sink.elasticsearch.ElasticsearchLogStashEventSerializer

啟動：

 ../bin/flume-ng agent -c . -f es_log.conf -n agent  -Dflume.root.logger=INFO,console

測試資料

website:weblog:login_page weblog data1website:weblog:profile_page weblog data2website:weblog:transaction_page weblog data3website:weblog:docs_page weblog data4syslog:syslog:sysloggroup syslog data1syslog:syslog:sysloggroup syslog data2syslog:syslog:sysloggroup syslog data3syslog:syslog:sysloggroup syslog data4syslog:syslog:sysloggroup syslog data5syslog:syslog:sysloggroup syslog data6

之後就能夠在es叢集上看到通過flume匯入的資料了

這時候編輯log檔案時候會被flume讀入es叢集中並即時產生索引  例如以所看到的：

這樣就能夠通過es對日誌的即時檢索了

flume 日誌匯入elasticsearch