Build an Elastic Stack Log Analysis System Under CentOS7

Build an Elastic Stack Log Analysis System Under CentOS7

This article introduces how to build a visual log analysis system using elasticsearch + logstash (beats) + kibana.
These software is also free open source, its official site: https://www.elastic.co/cn/products

1. Introduction to these software

Elasticsearch is an open-source distributed search engine that features: distributed, zero-configuration, automatic discovery, automatic index sharding, index copy mechanism, restful APIs, and multiple data sources, automatically search for loads.
Logstash is an open-source tool for collecting, analyzing, and storing logs.
Kibana is also an open-source and free tool. Kibana provides a log analysis friendly Web interface for Logstash and ElasticSearch to summarize, analyze, and search for important data logs.
Beats is an open-source agent for collecting system monitoring data from elasticsearch. It is a collectively referred to as a data collector running on the monitored server as a client, you can directly send data to Elasticsearch or Logstash to Elasticsearch for subsequent data analysis activities. Beats consists of the following components:
① Packetbeat: a network packet analyzer used to monitor and collect network traffic information. Packetbeat sniffers the traffic between servers, parses the application layer protocol, and associates it with message processing, it supports protocols such as ICMP (v4 and v6), DNS, HTTP, Mysql, PostgreSQL, Redis, MongoDB, and Memcache;
② Filebeat: used to monitor and collect server log files. It has replaced logstash forwarder;
③ Metricbeat: You can regularly obtain monitoring metrics of external systems. It can monitor and collect services such as Apache, HAProxy, MongoDB, MySQL, Nginx, PostgreSQL, Redis, System, and Zookeeper;
④ Winlogbeat: used to monitor and collect Windows system logs;
⑤ Create your own Beat: Custom beat. If the above indicators cannot meet your needs, elasticsarch encourages developers to use the go language to expand and implement custom beats. You only need to follow the template, monitor input, logs, and output.
Beats sends the  to Logstash. After being parsed and filtered by Logstash, the data is sent to Elasticsearch for storage and presented to the user by Kibana.
Beats, as a log collector, does not use Logstash as a log collector to consume resources. This solves the problem that Logstash occupies high system resources on each server node.

2. Deployment of elasticsearch + logstash + kibana

(1) approximate Network Topology

Note:
Blue Line path: file ----> logstash server ------> elasticsearch
Logstash implements its own filtering (input plugin), conversion (filter plugin), and output (output plugin) Mechanisms and then directly transmits them to the elasticsearch cluster.
Roadmap of the red line: The logstash agent transmits data to the logstash server and implements a unified output format on the servlet. At this moment, the agent inputs a file and outputs it to the server. On the logstash server, the input is the logstash agent and the output is the elasticsearch cluster.
Diagram of the dark color: Add a redis instance to buffer the queue. To reduce the pressure on the logstash server, this is also the mainstream configuration.
In the above three cases, we will simulate the third mainstream configuration. If you are interested, you can do it yourself.
(2) host configuration instructions

Host Name	Ip	Running Service
Master	172.16.5.4	Elasticsearch, filebeat
Server1	172.16.5.3	Logstash agent, redis
Server2	172.17.5.2	Logstash server, kibana

In this experiment, a large amount of memory is consumed, so no cluster is built. A server is used for testing.

(3) deploying elastic stach
① Environment deployment.
Disable iptables and selinux, and parse the time synchronization and hosts files.
Command: iptables-F firewall policy
Setenforce 0 temporarily disables selinux. If you want to disable selinux permanently, modify the configuration file.
② Install the java environment to solve the dependency and install it on all servers.
Yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel-y
③ Install the elasticsearch installation package.
Can download the official website: https://www.elastic.co/cn/products/elasticsearch
I used the downloaded installation package.
Rpm-ivh elasticsearch-5.4.2.rpm

④ Modify the configuration file
Vim/etc/elasticsearch/jvm. options
Modify memory:
-Xms1g
-Xmx1g [it is recommended that the production environment be 32 GB for optimal performance]
Vim/etc/elasticsearch. yml
Cluster. name: myels [cluster name]
Node. name: node1 [node name, host name]
Path. data:/data/els/data [index storage path]
Path. logs:/data/els/logs [Log path]
Network. host: 0.0.0.0 [Address used when adding a cluster, local address]
Http. port: 9200 [listening port]
Discovery. zen. ping. unicast. hosts: ["node1", "node2", "node3"] [determine whether it is in the same cluster, that is, whether it is listening to the same port]
Discovery. zen. minimum_master_nodes: 2 [There are several nodes, more than half of the minimum values]

⑤ Create the index and logs directory used by elasticsearch
Mkdir-pv/els/{date, logs} & chown-R elasticsearch: elasticsearch/els/* [Create index and log directory]

⑥ Start the service
Systemctl start elasticsearch. service

Test.

The above is successful.
(4) Deploy logstash agent and redis on Server 1
① Install the logstash and redis packages.

② View the logstash configuration file
/Etc/logstash/jvm. options [runtime environment, that is, the configuration of memory and other information; the server is recommended to be large, and the agent can be used by default]
/Etc/logstash/conf. d [configure the file used by the plug-in]
/Etc/logstash. yml [Master configuration file, how to run the configuration]
Config. reload. automatic: True [whether the file takes effect immediately. The default value is false and manual restart is required]
Config. reload. devel: 3 [How long will the configuration file be reloaded]
The configuration file does not need to be modified.
③ Add to path
Vim/etc/profied. d/logstash. sh
Export PATH = $ PATH:/usr/share/logstash/bin
④ Start the service
Systemctl start logstash

⑤ Modify the redis configuration file
We recommend that you modify this row to improve security.
Requirepass ilinux. io
⑥ Start the redis Service
Systemctl start redis
(5) Deploy the logstash server and kibana of server 2
① Install the logstash and kibana installation packages

② Modify the kibana configuration file
Vim/etc/kibana. yml
Server. host: "0.0.0.0"
Server. name: "host name"
Elasticsearch. uri: "http: // server: 9200"
③ Start logstash and kibana services
Systemctl start logstash. service
Systemctl start redis. service
④ Input http: // localhost: 5601 in the browser to configure the filebeat index (you only need to enter filebeat-).

(6) simulate the services that the input and output of each service point.
① Simulate the logstash agent to input the log file to redis.
Set the conversion mechanism of logstash.
The conversion rules are all stored in/etc/logstash/conf. d/Below.
Vim/etc/logstash/conf. d/redis. conf

Supplement: logstash has an independent variable function.
Rpm-ql logstash | grep patternt

To better simulate the production environment, we will install the httpd service here.
Yum install httpd
Start the httpd service
Systemctl start httpd. service
Create 20 pages
For I in {1 .. 20}; do echo "test $ {I}">/var/www/html/test $ {I}; done
Access 20 production logs
For I in {1 .. 20}; do j = $ [$ RANDOM % 20 + 1]; curl http: // 172.16.5.3/test1_1_j2.16.html; done
② The input end of the simulated logstash server is redis, and the output end is the elasticsearch cluster.
Vim/etc/logstash/conf. d/redis. conf

③ Send the content in the els cluster to kibana
In the first box, enter filebeat-* and wait for a moment. kibana will automatically recognize it. After OK, the buttons below will change from Gray to controllable "Create", as shown in. After you click this button, it will be displayed as shown in:

Go back and create a logstash index. Enter http: // server2: 5601 in the browser, click "Management" ==> on the Left bar, and then click "index Patterns" ==>

Click "Add New"

Click "Crete" to create a logstash index. After the index is created, it is displayed, as shown in:

Iii. beats lightweight data collector
Official station: https://www.elastic.co/cn/products/beats

Working Mode:
Beats --> redis --> logstash server --> els Cluster
Note:
In this case, beats is used instead of the logstash agent. Both logstash and filebeat have the log collection function. filebeat is lighter and consumes less resources. However, logstash has the filter function and can filter and analyze logs. Generally, filebeat collects logs and sends them to message queues, redis, and kafaka. Then logstash obtains the data, filters and analyzes data using the filter function, and stores the data in elasticsearch.
This experiment was basically completed in the previous experiment. This experiment is only used for the filebeat part.
① Installation package
Yum install httpd filebeat-y
② Modify the configuration file
Cd/etc/filebeat
Vim filebeat. yml
Filebeat. prospectors: [Where to load files, default]
Input_type: log [file type, log, default]
Paths:

• /Var/log/httpd/access_log

• /Var/log/httpd/error_log
  Hosts: [storage path]
  ③ Add data
  For I in {1 .. 20}; do echo "test $ I">/var/www/html/test?~ I =.html; done
  Start httpd service and filebeat Service
  Systemctl start httpd
  Randomly find a host for access, in order to get the log
  For I in {1 .. 20}; do j = $ [$ RANDOM % 20 + 1]; curl http: // 172.16.0.4/test1_1_j2.16.html
  ④ Output to elasticsearch
  Vim/etc/filebeat. yml
  Output. elasticsearch:
  Hosts: ["server1: 9200", "server2: 9200", "server3: 9200"]
  Directly output data to els Without Output plug-ins

  ⑤ Output to logstash
  Vim/etc/filebeat. yml
  Output. logstash:
  Hosts: ["Maid: 5044]
  Vim/etc/logstash/conf. d/Apachelog. conf
  Input {
  Beats {
  Port = & gt; 5044
  }
  Filter {
  Grok {
  Match => {
  "Message" => "% {HTTPD_COMBINEDLOG }"
  }
  }
  Date {
  Match => ["timestamp", "dd/MMM/YYYY: H: m: s Z"]
  }
  Mutate {
  Rename => {
  "Agent" => "user_agent"
  }
  }
  Geoip {
  Source => "clientip"
  Target => "geoip"
  Database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
  }
  Output {
  Elasticsearsh {
  Hosts => ["http: // server1: 9200", "http: // server2: 9200", "http: // master: 9200"]
  Index => "logstash-% {+ YYYY. MM. DD}
  Document_type => "http_access_logs"
  }
  }
  Start: logstash-f apachelog. conf
  ⑥ Output to redis
  Edit the filebeat configuration file
  Vim/etc/filebeat. yml
  Add:
  Output. redis:
  Hosts: ["redis server"]
  Password: "iliunx. io"
  Key: "httplog"
  Db: 0
  Timeout: 5
  Restart filebeat
  Systemctl restart filebeat
  Go to redis to view Data
  Redis-cli-a ilinux. io
  View data
  LLEN httplogs
  Configure the input mechanism on els server
  Vim/etc/elasticsearch/conf. d/redis2.conf
  Input {
  Redis {
  Batch_count => 1
  Data_type => "list"
  Key => "httpdlogs"
  Host => "192.168.0.2"
  Port = & gt; 6379
  Threads => 5
  Password => "ilinux. io"
  }
  }

Filter {
Grok {
Match => {
"Message" => "% {HTTPD_COMBINEDLOG }"
}
}
Date {
Match => ["timestamp", "dd/MMM/YYYY: H: m: s Z"]
}
Mutate {
Rename => {
"Agent" => "user_agent"
}
}
Geoip {
Source => "clientip"
Target => "geoip"
Database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
Output {
Elasticsearsh {
Hosts => ["http: // server1: 9200", "http: // server2: 9200", "http: // server3: 9200"]
Index => "logstash-% {+ YYYY. MM. DD}
Document_type => "http_access_logs"
}
}