Logstash Reading Redis Data

Redis server is the Logstash official recommended broker choice. The Broker role also means that both input and output plugins are present. Here we will first learn the input plugin.

Logstash::inputs::redis supports three types of data_type (in fact, Redis_type), and different data types lead to the actual use of different Redis command operations: List = Blpop Channel = SUBSCRIBE Pattern_channel = Psubscribe

Did you notice. There is no GET command in this.

Redis servers are often used as NoSQL databases, but Logstash are used only for Message Queuing. So don't worry that Redis in Logstash will burst your memory and disk. Configuration Example (the following configuration can not be less, otherwise Logstash will error)

Input {
    Redis {
        data_type = "Pattern_channel"
        key = "logstash-*"
        host = " 192.168.0.2 "
        port = 6379
        threads = 5
    }
}

How to use Basic Methods

First confirm that you set the host server already running the Redis-server service, and then open the terminal run Logstash process wait for input data, and then open another terminal, enter the REDIS-CLI command (the Redis package is installed first), Enter publish Logstash-demochan "Hello World" after the interactive prompt:

# redis-cli
127.0.0.1:6379> PUBLISH Logstash-demochan "Hello World"

You'll see in the first terminal that the Logstash process outputs something like this:

{
       "message" = "Hello World",
      "@version" = "1",
    "@timestamp" = "2014-08-08t16:26:29.399z"
}

Note: There is no host field in this event. (Maybe this is a bug ...) ) input JSON data

If you want to add more fields to the Logstash event via the Redis channel, you can simply publish the JSON string to the channel. Logstash::inputs::redis will convert the JSON directly into an event.

Continue to enter the following at the interactive prompt for the second terminal:

127.0.0.1:6379> PUBLISH Logstash-chan ' {"message": "Hello World", "@version": "1", "@timestamp": "2014-08-08t16 : 34:21.865z "," host ":" Raochenlindemacbook-air.local "," Key1 ":" Value1 "} '

You will see that the Logstash process in the first terminal also returns the new content as follows:

{
       "message" = "Hello World",
      "@version" = "1",
    "@timestamp" = "2014-08-09t00:34:21.865+08:00" ,
          "Host" = "raochenlindemacbook-air.local",
          "key1" = "value1"
}

Look, the new field appears. Now, you can ask the development engineer to send a message directly to your Redis channel, and everything will be done automatically. Tips

Here we recommend using Pattern_channel as the input plug-in's data_type setting value. Because in practice, your Redis channel may have many different keys, which are generally named Logstash-chan-%{type} in this form. This time Pattern_channel type will help you subscribe to all Logstash related channels at once. extension Mode

As mentioned in the previous paragraph "Tips", the previous two usage scenarios used the same configuration, that is, the data type is the channel to publish the subscription mode. In this way, when it is necessary to expand Logstash into a multi-node cluster, there is a problem: A message that is posted through the channel is received at the same time by all the Logstash processes subscribed to the channel, and then the duplicate content is output.

You can try to do the above experiment again, this time in two terminals simultaneously start the logstash-f redis-input.conf process, the result will be two terminals are output messages.

At this point, you need to use the list type. In this type, the data is entered into the Redis server staging, Logstash is connected to the Redis server take away (Blpop command, so long as the logstash does not clog, Redis server will not have data accumulation space) data. Configuration Examples

Input {
    Redis {
        batch_count = 1
        data_type + "list"
        key = "Logstash-list"
        host = " 192.168.0.2 "
        port = 6379
        threads = 5
    }
}

How to use

This time we run the Logstash-f redis-input-list.conf process at two terminals at the same time. Then start the REDIS-CLI command interaction in the third terminal:

$ redis-cli 
127.0.0.1:6379> rpush logstash-list "Hello World"
(integer) 1

As you can see, only one terminal outputs the result.

Successive Rpush several times, you can see two terminals almost each output half of the entries. Tips

Rpush support Batch mode, modify the Batch_count value in the Logstash configuration, as an example here only to 2, the actual application can be larger (in fact logstash::outputs::redis corresponding to this batch_event configuration default Value is 50).

After restarting the Logstash process, the REDIS-CLI command is changed to send as follows:

127.0.0.1:6379> rpush logstash-list "Hello World", "Hello World", "Hello World", "Hello World", "Hello World", "Hello World"
(integer) 3

As you can see, two terminals also output part of their results. And you only used one Rpush command. Recommended Reading

Http://redis.io reprinted from: http://udn.yyuap.com/doc/logstash-best-practice-cn/input/redis.html