Enterprise-level Log collection system--elkstack

Elkstack Introduction:

Elkstack is a combination of Elasticsearch, Logstash, and Kibana three open source software, forming a powerful real-time log collection display system.

The components function as follows:

Logstash: Log Collection tool can be used from local disk, network Service (own listening port, accept user log), message queue collects a variety of logs, then filter analysis, and output the log to Elasticsearch.

Elasticsearch: Log Distributed Storage/search tool, native support cluster function, can generate an index for a specified time log, speed up log query and access.

Kibana: A visual log web presence tool that displays the logs stored in the Elasticsearch and generates an illuminated dashboard.

The benefits of using Elkstack for operational operations:

1, the application of the log is mostly output in the server log files, most of them are developers to see, and then developed without logging on the server permissions, if developers need to view the log to the server to take the log, and then to the development; imagine a company with 10 development, A development every day to find operations to take a log, for the operation and maintenance personnel is a large amount of work, which greatly affect the operational efficiency, deployment of elkstack, the development of any can be directly logged into the Kibana log view, do not need to view the log through operations, This reduces the work of operation and maintenance.

2, a variety of logs, and scattered in different locations difficult to find: such as the LAMP/LNMP website access failure, this time may need to query the log to analyze the cause of the failure, if you need to view the Apache error log, you need to log on to Apache server to view, If you look at the database error log, you need to login to the database query, imagine if it is a clustered environment dozens of hosts it? At this time if the deployment of Elkstack can be logged to the Kibana page to view the log, to see the different types of logs only need to switch the index of the motorized mouse.

Elkstack Experimental Architecture diagram:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/83/64/wKiom1dyIIDycO6nAAAuHsm22zY193.png "title=" QQ picture 20160628145944.png "alt=" Wkiom1dyiidyco6naaauhsm22zy193.png "/>

Redis Message Queuing Role Description:

1. Prevent Logstash and ES from communicating properly, thus losing the log.

2, prevent the log volume too large cause ES can not withstand a large number of write operations to lose the log.

3, the application (Php,java) in the output log, you can directly output to the message queue, so as to complete the log collection.

Add: If the message queue used by Redis has an expansion bottleneck, a more powerful kafka,flume can be used instead.

Lab Environment Description:

[Email protected] ~]# cat/etc/redhat-release CentOS Linux release 7.2.1511 (Core) [[email protected] ~]# uname-rm3.10.0 -327.el7.x86_64 x86_64

Use the Software Description:

1. JDK-8U92 Official RPM Package

2, Elasticsearch 2.3.3 official RPM Package

3, Logstash 2.3.2 official RPM Package

4, Kibana 4.5.1 official RPM Package

5. Redis 3.2.1 Remi RPM Package

6, Nginx 1.10.0-1 official RPM Package

Deployment Sequence Description:

1. Elasticsearch Cluster configuration

2. Logstash Client configuration (write data directly to ES cluster, write system messages log)

3. Redis Message Queuing configuration (Logstash write data to Message Queuing)

4. Kibana Deployment

5. Nginx Load Balancer Kibana Request

6, mobile phone Nginx log

7, Kibana report function description

Configuration considerations:

1. Time must be synchronized

2. Turn off the firewall, SELinux

3, a problem, check the log

Elasticsearch Cluster Installation configuration

1. Configuring the Java Environment

[[email protected] ~]# yum-y install jdk1.8.0_92[[email protected] ~]# Java-versionjava version "1.8.0_92" Java (TM) SE Ru Ntime Environment (build 1.8.0_92-b14) Java HotSpot (TM) 64-bit Server VM (build 25.92-b14, Mixed mode)

2, install Elasticsearch, because I here yum source has been created, so can be installed directly

Official Document: Https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Official: Https://www.elastic.co/downloads/elasticsearch

[[email protected] ~]# yum-y install elasticstarch[[email protected] ~]# rpm-ql Elasticsearch/etc/elasticsearch/etc/ela Sticsearch/elasticsearch.yml #主配置文件/etc/elasticsearch/logging.yml/etc/elasticsearch/scripts/etc/init.d/ Elasticsearch/etc/sysconfig/elasticsearch/usr/lib/sysctl.d/usr/lib/sysctl.d/elasticsearch.conf/usr/lib/systemd /system/elasticsearch.service #启动脚本/usr/lib/tmpfiles.d/usr/lib/tmpfiles.d/elasticsearch.conf

3, modify the configuration file, here are some of the path to see personal habits

[Email protected] ~]# vim/etc/elasticsearch/elasticsearch.yml cluster.name: "Linux-es" Node.name:es1.bjwf.com33 Path.data:/elk/data37 path.logs:/elk/logs43 bootstrap.mlockall:true54 network.host:0.0.0.058 http.port:920068 Discovery.zen.ping.unicast.hosts: ["192.168.130.221", "192.168.130.222"]

4. Create the relevant directory and give permissions

[[email protected] ~]# mkdir-pv/elk/{data,logs}[[email protected] ~]# chown-r Elasticsearch.elasticsearch/elk[[email p Rotected] ~]# ll/elkdrwxr-xr-x. 2 Elasticsearch elasticsearch 6 June 03:51 Datadrwxr-xr-x. 2 Elasticsearch elasticsearch 6 June 03:51 logs

5. Start ES and check if 9200 and 9300 ports are listening

[[email protected] ~]# systemctl start elasticsearch.service[[email protected] ~]# netstat-tnlp|egrep "9200|9300" TCP6 0 0::: 9200:::* LISTEN 17535/java tcp6 0 0::: 9300:::* LISTEN 17535/java

6, install another machine, the same steps as the first one

[Email protected] ~]# vim/etc/elasticsearch/elasticsearch.yml node.name:es2.bjwf.com #主要修改主机名

7. View the status of two nodes

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/83/65/wKiom1dyMAnwyLcOAAB1gcxR8iY355.png "style=" float: none; "title=" QQ picture 20160628160524.png "alt=" Wkiom1dymanwylcoaab1gcxr8iy355.png "/>

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/83/64/wKioL1dyMAmQ2oHXAAB7jAcRfv8821.png "style=" float: none; "title=" QQ picture 20160628160538.png "alt=" Wkiol1dymamq2ohxaab7jacrfv8821.png "/>

Configuring the cluster Management plug-in (head, Kopf, etc.)

The official provides an ES cluster management plugin that allows very intuitive viewing of the ES cluster status and index data information

[[email protected] ~]#/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head[[email protected] ~]#/usr/ Share/elasticsearch/bin/plugin Install Lmenezes/elasticsearch-kopf

Access plugins:

http://192.168.130.222:9200/_plugin/head/

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M01/83/65/wKiom1dyMcnSDRznAACuYsH31pQ619.png "title=" QQ picture 20160628161246.png "alt=" Wkiom1dymcnsdrznaacuysh31pq619.png "/>

http://192.168.130.222:9200/_plugin/kopf/

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/83/64/wKioL1dyMdjgUk02AACX56gWJIk296.png "title=" QQ picture 20160628161318.png "alt=" Wkiol1dymdjguk02aacx56gwjik296.png "/>

The ES cluster configuration has been completed, the following can be configured Logstash to the ES cluster to write data

Logstash deployment

1, configure the Java environment, install Logstash

[[email protected] ~]# yum-y install jdk1.8.0_92[[email protected] ~]# yum-y install Logstash

2. Verify the input and output of the Logstash through the configuration file

[[email protected] ~]# Vim/etc/logstash/conf.d/stdout.confinput {stdin {}}output {stdout { codec = "Rubydebug"}}

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/83/65/wKiom1dyNJOwKWixAABMP79bwys941.png "title=" QQ picture 20160628162542.png "alt=" Wkiom1dynjowkwixaabmp79bwys941.png "/>

3. Define output to Elasticsearch

[[email protected] ~]# vim /etc/logstash/conf.d/ Logstash.confinput {        stdin {}}output {input  {        stdin {}}output {         elasticsearch {                 hosts => ["192.168.130.221:9200", "192.168.130.222:9200"]                 index =>   "Test"         }}[[email protected] ~]# /opt/ logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf settings: default  Hello, pipeline workers: 4pipeline main startedhello!. 

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/83/66/wKiom1dyPODgtfZwAAEEZ3deBEg557.png "title=" QQ picture 20160628170058.png "alt=" Wkiom1dypodgtfzwaaeez3debeg557.png "/> This time, Logstash pick up Elasticsearch is able to work properly, The following describes how to collect system logs

4, Logstash collection system log

Modify the Logstash configuration file as shown below, and start the Logstash service to see the messages log in the head as it is already written to Es, and create an index

[[email protected] ~]# vim /etc/logstash/conf.d/logstash.confinput {         file {           type =>  "MessageLog"           path =>   "/var/log/messages"           start_position =>   "Beginning"         }}output {         file {          path =>  "/tmp/123.txt"         }         elasticsearch {                 hosts => ["192.168.130.221:9200", "192.168.130.222:9200"]                 index =>  "system-messages-%{+yyyy. MM.DD} "        }} #检查配置文件语法:/etc/init.d/logstash configtest/opt/ logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --configtest# change start Logstash user:#  vim /etc/init.d/logstashls_user=rootls_group=root# starts with a configuration file [[email protected] ~]# /opt/ logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf &

Successful collection, auto-generated index of system-messages

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M00/83/66/wKioL1dyQy6Cl-n9AAHpHRtjtSM303.png "title=" QQ picture 20160628172531.png "alt=" Wkiol1dyqy6cl-n9aahphrtjtsm303.png "/>

Kibana deployment

Description: I am here to deploy Kibana in two ES nodes and use Nginx for load balancing, if there is no special need, you can only deploy a single node

 1, install Kibana, each ES node deploys a [[email protected] ~]# yum -y  INSTALL KIBANA2, configure Kibana, only need to specify ES address other configurations remain default [[email protected] ~]# vim /opt/ kibana/config/kibana.yml 15 elasticsearch.url:  "http://192.168.130.221:9200" [[email  Protected] ~]# systemctl start kibana.service[[email protected] ~]# netstat  -tnlp|grep 5601     #Kibana监听端口tcp          0      0 0.0.0.0:5601  0.0.0.0:*      listen      17880/node 

Look at the effect, this picture is pirated. I did this, I forgot.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/83/67/wKiom1dyTTnRBnjzAAFLdRHJKyo424.png "title=" QQ picture 20160628181042.png "alt=" Wkiom1dyttnrbnjzaafldrhjkyo424.png "/> After creation is complete

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/83/66/wKioL1dyTYfhtAZvAAFU22a4a40181.png "title=" QQ picture 20160628181208.png "alt=" Wkiol1dytyfhtazvaafu22a4a40181.png "/>

Write this first and continue tomorrow. The amount of work is not small ...

This article is from the "Ask Heaven" blog, please make sure to keep this source http://79076431.blog.51cto.com/8977042/1793682

Enterprise-level Log collection system--elkstack