Redis Server and collection terminal settings

Redis(logstash). conf Content # server configuration , Logstash crawl redis data, configure name pickup

Example One # read Data from Redis

Input {

Redis {

Host = "127.0.0.1"

Port = 6379

Type = "Redis-input"

data_type = "List"

Key = "Logstash:redis"

}

}

Output { # to ELA

StdOut {}

Elasticsearch {

cluster = "Elasticsearch"

codec = "JSON"

protocol = "HTTP"

}

}

example Two # read Data from Redis

Input {

Redis {

Host = ' 192.168.233.130 '

data_type = ' list '

Port = "6379"

Key = ' Logstash:redis '

Type = ' Redis-input '

}

}

Output {# to ELA

Elasticsearch {

Embedded = True

}

}

Logstash-kibama 9292

Logstash-redis 6379

Logstash-elasticsearch 9200

Kibana 5601

# vim Redis(logstash). conf # Log Collection side configuration , Logstash set is listening log file data, configuration name pickup

Input {#收集监控端日志文件

File {

Type = "Producer"

Path = "/soft/apache.log"

}

File {

Type = "Php-log"

Path = "/soft/php.log"

}

}

Filter {# Log contents will be filtered as long as there is a match for MySQL or get or error, sent to Logstash index

grep {

Match = ["@message", "mysql| Get|error "]

}

}

Output {#将收集的日志文件发送到redis

Redis {

Host = ' 192.168.233.130 '

data_type = ' list '

Key = ' Logstash:redis '

}

}

Test program Send data--Redis Message Queuing--Logstash-elasticsearch cluster

increase indexing speed through pipeline thinking

In order to solve the problem of Redis queue bottleneck and use the multi-pipeline mechanism to increase the throughput of the whole system, we have deployed multiple Redis instances and corresponding number of Logstash instances at the same time:

Test program Send data--Redis Message Queuing 1->LOGSTASH1-elasticsearch cluster

Test program Send data--Redis Message Queuing 2->LOGSTASH2-elasticsearch cluster

Test program Send data--Redis Message Queuing 3->LOGSTASH3-elasticsearch cluster

...

The advantage of using the pipeline mechanism is that extensibility is obvious

This article is from the cloud blog, so be sure to keep this source http://weimouren.blog.51cto.com/7299347/1732075

Redis Server and collection terminal settings