ELK + filebeat log analysis system deployment document

ELK + filebeat log analysis system deployment document

Environment Description
Architecture Description and architecture Diagram

Filebeat is deployed on the client to collect logs and send the collected logs to logstash.
Logstash sends the collected logs to elasticsearch.
Kibana extracts and displays data from elasticsearch.
The reason why filebeat is used for log collection is that filebeat does not use a large amount of resources like logstash, affecting the Service server.

Environment requirements
Java environment and redis

Yum install java
Yum install redis version
Java 1.8.0 _ 111

Redis 2.8.16

Filebeat 5.1.2

Logstash 5.1.2

Elasticsearch 5.1.1

Kibana 5.1.1

Install configurations
Filebeat installation and configuration
Install filebeat

Rpm -- import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
Cat>/etc/yum. repos. d/elk. repo <EOF
Elasticsearch-5.x
Name = Elasticsearch repository for 5.x packages
Base url = https://artifacts.elastic.co/packages/5.x/yum
Gpgcheck = 1
Gpgkey = https://artifacts.elastic.co/GPG-KEY-elasticsearch
Enabled = 1
Autorefresh = 1
Type = rpm-md
EOF
Yum clean all
Yum install filebeat-y

Configure filebeat

Vim/etc/filebeat. yml
Paths:
-/Var/log/nginx/access. log
Tags: ["nginx"]
Output. logstash:
Hosts: ["1.8.101.53: 5044"]

The path option is the path that filebeat sends to logstash. You can use multiple logs *. log configuration. file does not automatically recursive the subdirectories in the log directory. If you need recursive subdirectories, you can use a directory similar to/var/log /*/*. log structure. the tags option adds a label to the log. This label can be provided to logstash to differentiate logs of different services on different clients. output specifies the server to which the log is sent and the service to elasticsearch by default. in this example, logstash is used, so you need to comment out the configuration sent to elasticsearch and enable the configuration sent to logstash.

Start filebeat

/Usr/share/filebeat/bin/filebeat-c/etc/filebeat. yml-e & logstash

Installation and configuration
Install logstash

Rpm -- import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
Cat>/etc/yum. repos. d/elk. repo <EOF
Elasticsearch-5.x
Name = Elasticsearch repository for 5.x packages
Base url = https://artifacts.elastic.co/packages/5.x/yum
Gpgcheck = 1
Gpgkey = https://artifacts.elastic.co/GPG-KEY-elasticsearch
Enabled = 1
Autorefresh = 1
Type = rpm-md
EOF
Yum clean all
Yum install logstash
Ln-s/usr/share/logstash/bin/logstash/usr/bin/logstash configure logstash

Vim/etc/logstash/conf. d/nginx. conf
Input {
Beats {
Port = & gt; 5044
}
}
Filter {
If "nginx" in [tags] {
Grok {
Match => ["message", "% {COMBINEDAPACHELOG} + % {GREEDYDATA: extra_fields}"]
Overwrite => ["message"]
}
Mutate {
Convert => ["response", "integer"]
Convert => ["bytes", "integer"]
Convert => ["responsetime", "float"]
}
Geoip {
Source => "clientip"
Target => "geoip"
}
Date {
Match => ["timestamp", "dd/MMM/YYYY: HH: mm: ss Z"]
Remove_field => ["timestamp"]
}
Useragent {
Source => "agent"
}
}
}
Output {
If "nginx" = [tags] [0] {
Elasticsearch {
Hosts => ["1.8.101.53: 9200"]
Index => "access-% {+ YYYY. MM. dd }"
}
}
}

Start logstash

Logstash -- path. settings/etc/logstashelasticsearch installation and configuration
Install elasticsearch

Yum install elasticsearch start elasticsearch

Service elasticsearch startelasticsearch does not require too many configurations. Use the package management tool to start it.

Kibana installation and configuration

Install kibana

Yum intall kibana start kibana

/Usr/share/kibana/bin/kibana &

This article permanently updates link: https://www.bkjia.com/Linux/2018-02/150988.htm