Install logstash + kibana + elasticsearch + redis to build a centralized Log Analysis Platform

Install logstash + kibana + elasticsearch + redis to build a centralized Log Analysis Platform

19:40:57 | category: logstash | label: logstash Kinana elasticsearch redis log analysis | report | font size subscription

This article is a reference to the practice of logstash official documentation. The environment and required components are as follows:

• RedHat 5.7 64bit/centos 5.x
• JDK 1.6.0 _ 45
• Logstash 1.3.2 (with kibana)
• Elasticsearch 0.90.10
• Redis 2.8.4

The process of building a centralized log analysis platform is as follows:

 

Elasticsearch

1. Download elasticsearch.

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.tar.gz

2. decompress the package and enter the bin directory. Run the following command to start elasticsearch in the previous mode:

./elasticsearch -f

[2014-01-16 16:21:31,825][INFO ][node                     ] [Saint Elmo] version[0.90.10], pid[32269], build[0a5781f/2014-01-10T10:18:37Z][2014-01-16 16:21:31,826][INFO ][node                     ] [Saint Elmo] initializing ...[2014-01-16 16:21:31,836][INFO ][plugins                  ] [Saint Elmo] loaded [], sites [][2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] initialized[2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] starting ...[2014-01-16 16:21:35,578][INFO ][transport                ] [Saint Elmo] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.0.2.15:9300]}

Redis

1. For the installation method, refer to my other article redis compilation and installation.

2. Go to the bin directory and run the following command to output the debug information on the console:

./redis-server --loglevel verbose

[32470] 16 Jan 16:45:57.330 * The server is now ready to accept connections on port 6379[32470] 16 Jan 16:45:57.330 - 0 clients connected (0 slaves), 283536 bytes in use

Logstash log generator (shipper)

1. Create a configuration file:shipper.confThe content is as follows:

input {    stdin {        type => "example"    }}output {    stdout {        codec => rubydebug    }    redis {        host => "127.0.0.1"        port => 6379        data_type => "list"        key => "logstash"    }}

2. Start shipper. Run the following command:

java -jar logstash-1.3.2-flatjar.jar agent -f shipper.conf 

The following message is displayed in the terminal window:

Using milestone 2 output plugin ‘redis‘. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}

Press enter in the terminal window to display the following information:

{       "message" => "",      "@version" => "1",    "@timestamp" => "2014-01-16T08:15:19.400Z",          "type" => "example",          "host" => "redhat"}

This JSON information will be sent to redis, and the following prompt will appear in the terminal window of redis:

[32470] 16 Jan 17:09:23.604 - Accepted 127.0.0.1:44640[32470] 16 Jan 17:09:27.127 - DB 0: 1 keys (0 volatile) in 4 slots HT.[32470] 16 Jan 17:09:27.127 - 1 clients connected (0 slaves), 304752 bytes in use

Logstash log Indexer)

1. Create a configuration file:indexer.confThe content is as follows:

input {  redis {    host => "127.0.0.1"    # these settings should match the output of the agent    data_type => "list"    key => "logstash"    # We use the ‘json‘ codec here because we expect to read    # json events from redis.    codec => json  }}output {  stdout { debug => true debug_format => "json"}  elasticsearch {    host => "127.0.0.1"  }}

2. Start the log indexer. Run the following command:

java -jar logstash-1.3.2-flatjar.jar agent -f indexer.conf 

The following message is displayed in the terminal window:

Using milestone 2 input plugin ‘redis‘. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}You are using a deprecated config setting "debug_format" set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future.  If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"debug_format", :plugin=>, :level=>:warn}

The indexer receives information from redis. The following information is displayed in the terminal window:

{"message":"","@version":"1","@timestamp":"2014-01-16T17:10:03.831+08:00","type":"example","host":"redhat"}{"message":"","@version":"1","@timestamp":"2014-01-16T17:13:20.545+08:00","type":"example","host":"redhat"}{

Logstash Web Interface (kibana)

1. Start kibana. Run the following command:

java -jar logstash-1.3.2-flatjar.jar web

2. Open a browser (HTML5 must be supported), Input address: http: // 127.0.0.1: 9292/index.html #/dashboard/file/logstash. JSON. The interface effect is as follows:

 

References

• Logstash-getting-started-centralized
• Interview and book reviews: "logstash to make log management simpler"

Install logstash + kibana + elasticsearch + redis to build a centralized Log Analysis Platform