安裝logstash+kibana+elasticsearch+redis搭建集中式日誌分析平台

標籤：blog   http   java   os   檔案   io   for   2014   

本文是參考logstash官方文檔實踐的筆記，搭建環境和所需組件如下：

• Redhat 5.7 64bit / CentOS 5.x
• JDK 1.6.0_45
• logstash 1.3.2 (內帶kibana)
• elasticsearch 0.90.10
• redis 2.8.4

搭建的集中式日誌分析平台流程如下： 

 

elasticsearch

1、下載elasticsearch。

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.tar.gz

2、解壓後，進入bin目錄。執行如下命令，讓elasticsearch以前台方式啟動：

./elasticsearch -f

[2014-01-16 16:21:31,825][INFO ][node                     ] [Saint Elmo] version[0.90.10], pid[32269], build[0a5781f/2014-01-10T10:18:37Z][2014-01-16 16:21:31,826][INFO ][node                     ] [Saint Elmo] initializing ...[2014-01-16 16:21:31,836][INFO ][plugins                  ] [Saint Elmo] loaded [], sites [][2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] initialized[2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] starting ...[2014-01-16 16:21:35,578][INFO ][transport                ] [Saint Elmo] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.0.2.15:9300]}

Redis

1、其安裝方式可以參考我的另一篇文章Redis編譯安裝。

2、進入其bin目錄，執行如下命令，使之在控制台輸出debug資訊：

./redis-server --loglevel verbose

[32470] 16 Jan 16:45:57.330 * The server is now ready to accept connections on port 6379[32470] 16 Jan 16:45:57.330 - 0 clients connected (0 slaves), 283536 bytes in use

logstash日誌產生器（shipper）

1、建立一個設定檔：shipper.conf，其內容如下：

input {    stdin {        type => "example"    }}output {    stdout {        codec => rubydebug    }    redis {        host => "127.0.0.1"        port => 6379        data_type => "list"        key => "logstash"    }}

2、啟動shipper。執行如下命令：

java -jar logstash-1.3.2-flatjar.jar agent -f shipper.conf 

終端視窗將出現如下提示資訊：

Using milestone 2 output plugin ‘redis‘. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}

然後在終端視窗直接按斷行符號，將出現如下資訊：

{       "message" => "",      "@version" => "1",    "@timestamp" => "2014-01-16T08:15:19.400Z",          "type" => "example",          "host" => "redhat"}

這個json資訊將發送給redis， 同時redis的終端視窗將出現類似下面的提示資訊：

[32470] 16 Jan 17:09:23.604 - Accepted 127.0.0.1:44640[32470] 16 Jan 17:09:27.127 - DB 0: 1 keys (0 volatile) in 4 slots HT.[32470] 16 Jan 17:09:27.127 - 1 clients connected (0 slaves), 304752 bytes in use

logstash日誌索引器（indexer）

1、建立一個設定檔：indexer.conf，其內容如下：

input {  redis {    host => "127.0.0.1"    # these settings should match the output of the agent    data_type => "list"    key => "logstash"    # We use the ‘json‘ codec here because we expect to read    # json events from redis.    codec => json  }}output {  stdout { debug => true debug_format => "json"}  elasticsearch {    host => "127.0.0.1"  }}

2、開機記錄索引器。執行如下命令：

java -jar logstash-1.3.2-flatjar.jar agent -f indexer.conf 

終端視窗將出現如下提示資訊：

Using milestone 2 input plugin ‘redis‘. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}You are using a deprecated config setting "debug_format" set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future.  If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"debug_format", :plugin=>, :level=>:warn}

索引器從Redis接收到資訊，在終端視窗會顯示類似如下的資訊：

{"message":"","@version":"1","@timestamp":"2014-01-16T17:10:03.831+08:00","type":"example","host":"redhat"}{"message":"","@version":"1","@timestamp":"2014-01-16T17:13:20.545+08:00","type":"example","host":"redhat"}{

logstash WEB介面（kibana）

1、啟動kibana。執行如下命令：

java -jar logstash-1.3.2-flatjar.jar web

2、開啟瀏覽器（須支援HTML5），輸入地址：http://127.0.0.1:9292/index.html#/dashboard/file/logstash.json。介面效果如下： 

 

參考資料

• logstash-getting-started-centralized
• 訪談與書評：《LogStash，使日誌管理更簡單》

 

from:http://aofengblog.blog.163.com/blog/static/6317021201401664935685/