Redis+logstash+elasticsearch+kibana Multi-redis log data stream

This article describes a simple case that explains how to configure multiple log streams in Logstash, receive them through the Redis cache, and then export to Elasticsearch multiple indexes, that is, a class of log data corresponding to a class of indexes.

Assuming that two sets of log data are written to the Redis cache by the log side, and that the two sets of log tokens are of type redis-data-a and Redis-data-b, the Logstash configuration file is written as follows

Input {

Redis {

Host = "127.0.0.1"

Type = "Redis-data-a"

data_type = "List"

Key = "ListA"

}

Redis {

Host = "127.0.0.1"

Type = "Redis-data-b"

data_type = "List"

Key = "Listb"

}

}

Output {

if [type] = = "Redis-data-a" {

Elasticsearch {

host = localhost

index = "logstash_event_a-%{+yyyy. MM.DD} "

}

stdout {codec = Rubydebug}

}

if [type] = = "Redis-data-b" {

Elasticsearch {

host = localhost

index = "logstash_event_b-%{+yyyy. MM.DD} "

}

stdout {codec = Rubydebug}

}

}

In the above Logstash configuration file, the focus is:

1. At input, set up two sets of redis input data, specifying the type of two sets of data by type

2, at the output, through if [type] = = "", set the conditional output. When the type = = Redis-data-a is satisfied , the data is exported to Elasticsearch, indexed in the format "logstash_event_a-%{+yyyy. MM.DD} ". if type = = Redis-data-b, the Export index format is "logstash_event_b-%{+yyyy." MM.DD} ". Note that the index format cannot have uppercase characters, otherwise elasticsearch will error.

Test, use the REDIS-CLI command to connect to the Redis service, and try to write data in two queues Lista and listb. Lista only enter English characters, LISTB only enter numbers. This is just to demonstrate that the actual development uses Java or Python to connect redis-server, and then write a JSON-formatted string. Logstash will disassemble the JSON string and write the data correctly to the Elasticsearch index. Very convenient.

Open Kibana, go to its homepage and click on the standard interface shown in the red box

Click the Gear button in the top right corner to configure Dashborad to read the index of the Redis-data-a log data, which is the index data starting with logstash_event_a-(the same method can be set logstash_ event_b-opening):

Click Save, Dashboard page refresh, you can see the message content displayed, as well as the type recorded in the redis-data-a, indicating that we have successfully obtained the data of this type of log, all English characters, and type is Redis-data-a.

To avoid closing the browser, next time you repeat the above configuration, click the Save button at the top right of the interface to save the configuration. Later, just click on the open icon to find this configuration.

Use the same method to view the log data redis-data-b, all numbers, and type redis-data-b:

Note: If Kibana appears, the data cannot be displayed. Please make the following configuration adjustments:

1, in Logstash directory Logstash-1.4.2/vendor/kibana, edit Config.js file, set Elasticsearch: "http://192.168.1.100:9200", This is the server address I'm experimenting with.

2. In elasticsearch Config directory, set elasticsearch.yml, add Http.cors.enabled:true and Http.cors.allow-origin on the last side: "*"

Redis+logstash+elasticsearch+kibana Multi-redis log data stream