Centos7 single-host ELK deployment and centos7 elk deployment

Centos7 single-host ELK deployment and centos7 elk deployment
I,Introduction1. 1Introduction

ELK is composed of three open-source tools:

Elasticsearch is an open-source distributed search engine that features: distributed, zero-configuration, automatic discovery, automatic index sharding, index copy mechanism, restful APIs, and multiple data sources, automatically search for loads.

Logstash is a fully open-source tool that collects, filters, and stores your logs for future use (such as searching ).

Kibana is also an open-source and free tool. It provides a user-friendly Web interface for log analysis provided by Logstash and ElasticSearch, helping you summarize, analyze, and search for important data logs.

1.2   Scenario Analysis

Logs mainly include system logs, application logs, and security logs. O & M personnel and developers can learn about the server software and hardware information through logs, and check the configuration errors and the causes of errors. Regular log analysis can understand the load and performance security of servers, so as to take timely measures to correct errors.

Generally, logs are stored separately on different devices. If you manage dozens of hundreds of servers, you are still using the traditional method of logging on to each server in sequence to view logs. Is this complicated and inefficient. It is imperative to use centralized log management, such as open-source syslog To collect and summarize logs on all servers.

After the log is managed centrally, log statistics and retrieval become troublesome. Generally, we can use Linux commands such as grep, awk, and wc to perform retrieval and statistics, however, this method is not suitable for queries, sorting, statistics, and a large number of machines.

Here using open source Real-time log analysis ELK platform can perfectly solve our above problems, of course, there are other platforms or tools can be used, here only discuss ELK, Official Website: https://www.elastic.co

 

Ii. Install Elas Ticsearch 2.1 install jdk

# java -versionjava version "1.8.0_121"Java(TM) SE Runtime Environment (build 1.8.0_121-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

 2.2 install ELasticsearch

# Tar-zxvf elasticsearch-5.6.3.tar.gz # mv elasticsearch-5.6.3/data/elasticsearch # cd elasticsearch/config/# backup configuration file # cp elasticsearch. yml elasticsearch. yml. bak

　　Edit configuration file

# cat elasticsearch.yml | grep -v ^#cluster.name: elk-applicationnode.name: node-1path.data: /data/elasticsearch/datapath.logs: /data/elasticsearch/logsnetwork.host: 172.16.220.248http.port: 9200discovery.zen.ping.unicast.hosts: ["node-1"]discovery.zen.minimum_master_nodes: 1

　　Add an elasticsearch user and cannot start with root.

# groupadd -g 1008 elasticsearch# useradd -g 1008 -u 1008 elasticsearch# chown -R elasticsearch:elasticsearch /data/elasticsearch/

　　Modify the sysctl. conf file

# vim /etc/sysctl.confvm.max_map_count = 262144# sysctl -p　　

　　Modify/Etc/security/limits. conf file, modify the file opening handle

*               soft    nofile          100000*               hard    nofile          100000*               soft    nproc           100000*               hard    nproc           100000

　　Add a hosts file

# vim /etc/hosts172.16.220.248 node-1

　　Start

# su -s elasticsearch# cd /data/elasticsearch/bin# ./elasticearch &

　　Check whether startup is enabled

　　

　　SimpleCurl Test

# curl http://172.16.220.248:9200

　　

 

3. Install Logstash and filebeat

　Filebeat is used to obtain data on each server, send the data to logstash, and then process the data by logstash.

3.1 install log Stash

# tar -zxvf logstash-5.6.3.tar.gz# mv logstash-5.6.3 /data/logstash

3.2 Install file Beat

　　Download and start filebeat, and use it to listen to the newly added content of the data source file after being processed by logstash and then upload it to es.

# tar -zxvf filebeat-5.6.3-linux-x86_64.tar.gz# mv filebeat-5.6.3-linux-x86_64 /data/filebeat# cd /data/filebeat# cp filebeat.yml filebeat.yml.bak

　　EditFilebeat. yml File

Filebeat. prospectors:-input_type: log paths:-/var/log/message-log # test a local log file output. logstash: hosts: ["172.16.220.248: 5044"]

　　StartFilebeat Service

# cd /data/filebeat# ./filebeat &

　　Check the startup. filebeat does not have a listening port. It mainly depends on logs and processes.

# tialf logs/filebeat# ps -ef | grep filebeat

　　

The file records monitored by filebeat are stored in/data/filebeat/data/registry.

Create a local file message-log, which can take the messages files of the local system.

3.3 start lo Gstash　　

Finally, create a logstash startup specified test. conf configuration file. The content is as follows:

Input {beats {port => "5044"} output {elasticsearch {hosts => "172.16.220.248: 9200 "} stdout {codec => rubydebug} # print the output on the screen and comment it out}

By default, Logstash has three regions: input, filter, and output. Generally, at least input and output need to be configured!

Choose not to modify the default logstash. yml configuration file of logstash!

In a simple test, logstash does not specify the configuration file to start.

# cd /data/filebeat/bin# ./logstash -e 'input { stdin {} } output {stdout {} }'

　　

We manually enter hello world, which also outputs hello world

　　Specify the configuration file to start logstash　

# ./logstash -f ../config/test.conf &

Check whether port 5044 and port 9600 are enabled.

　　

Wait for a while and the following output will appear, that is, the last line in test. conf will be output to the screen.

　　

4. Install kibana

# tar -zxvf kibana-5.6.3-linux-x86_64.tar.gz# mv kibana-5.6.3-linux-x86_64 /data/kinbana# cd /data/kinbana/config/# cp kibana.yml kibana.yml.bak

　　Edit the kibana. yml configuration file

# vim kibana.ymlserver.port: 5601server.host: "172.16.220.248"elasticsearch.url: "http://172.16.220.248:9200"

　　Start kinbana

# cd /data/kibana/bin# ./kibana &

　　View port

　　

　　Browser login View

Click the create button and then click the discover button above. Note that if there is no data, check the import time @ timestamp to compare with the current time, kibana only displays the data of the last 15 minutes by default. If the data exceeds 15 minutes, select the appropriate time. From kibana, 15 data entries in messages-log are imported normally. This completes the first effect of our implementation. But this is just to run the process, and there are more things we need to do next. Note that you can create an index in kibana only after importing data to elasticsearch.

　　

 

5. Obtain Nginx access logs

Nginx log format is not in the grok of logstash by default, We need to manually configure, you can use http://grokdebug.herokuapp.com/online tools to determine whether the configuration is correct.

5.1 install filebeat on the nginx Server

Server: 172.16.200.160

# tar -zxvf filebeat-5.6.3-linux-x86_64.tar.gz# mv filebeat-5.6.3-linux-x86_64 /data/filebeat# cd /data/filebeat# cp filebeat.yml filebeat.yml.bak

　　Modify the filebeat configuration file

# cat filebeat.yml | grep -v ^$ | grep -v ^# | grep -v "#"filebeat.prospectors:- input_type: log  paths:    - /data/nginx/logs/160_access.log   document_type: nginx_accessoutput.logstash:  hosts: ["172.16.220.248:5044"]

　　Start filebeat

# ./filebeat &　

5.2 reconfigure the logstash STARTUP configuration file

The nginx log format is modified according to business requirements, such as adding cookies and modifying access. log time format. This will be written in another blog and a link will be provided.

Add cookie information to nginx logs

Nginx changes the time format in access. log

Nginx log format　　

Log_format main '[$ time_local]-$ remote_addr: $ remote_port-$ upstream_addr $ upstream_status $ response-''" $ request "$ status $ bytes_sent $ request_time'' "$ http_referer"-"$ http_user_agent"-''" $ customerTag_cookie" -"$ ym_cookie"-"$ http_cookie" ''" $ http_x_forwarded_for "'; # Here is just our own format. You can add or delete as required.

Grok uses expressions

Maybe I don't understand it very well, and the writing is complicated. I will write the matching items one by one. You can understand them and configure them for your project.

%{SYSLOG5424SD} - %{IPV4:clientip}:%{NUMBER:clientport} - %{IPV4:hostip}:%{NUMBER:itemport} %{INT:upstream_status} %{NUMBER:response_time} - \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}\" %{INT:status} %{INT:body_bytes_sent} %{NUMBER:request_time} %{QS:url} - %{QS:user_agent} - %{QS:customerTag} - %{QS:ym_traffic_session_id} - %{QS:all_cookies} %{IPV4:realip}

　　Grok matching description

%{SYSLOG5424SD}	Time Format
%{IPV4:clientip}	Obtain the ip address. clientip is named by yourself.
%{NUMBER:clientport}	NUMBER matching NUMBER
%{INT:upstream_status}	INT integer
%{WORD:method}	WORD
%{URIPATHPARAM:request}	Request
%{QS:url}	QS can get a string

 

　　Modify the logstash STARTUP configuration file

Input {beats {port => "5044"} filter {if [type] = "nginx_access" {grok {match => {"message" => "% {SYSLOG5424SD} -% {IPV4: clientip }:% {NUMBER: clientport}-% {IPV4: hostip }:% {NUMBER: itemport }%{ INT: upstream_status }%{ NUMBER: response_time}-\ "% {WORD: method }%{ URIPATHPARAM: request} HTTP/% {NUMBER: http_version} \" % {INT: status} % {INT: body_bytes_sent }%{ NUMBER: request_time }%{ QS: url}-% {QS: user_agent}-% {QS: mermertag}-% {QS: ym_traffic_session_id}-% {QS: all_cookies }%{ IPV4: realip} "}} mutate {remove_field =>" message "# Remove the message field, which summarizes the information obtained from the previous item, in this case, the output {elasticsearch {hosts => "172.16.220.248: 9200"} # stdout {codec => rubydebug} is repeated }}

　　Test the configuration file.

./logstash -t -f ../config/logstash.conf

　　

Restart logstash

There will be data in elasticsearch and kibana.