CENTOS6.5 installation Log Analysis Elk Elasticsearch + logstash + Redis + Kibana

1. Workflow of Log Platform

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/71/5F/wKioL1XNWHGwPB_ZAAErAE7qZjQ757.jpg "title=" 1.png " alt= "Wkiol1xnwhgwpb_zaaerae7qzjq757.jpg"/>

• shipper means log collection, using Logstash to collect log data from various sources, such as system logs, files, Redis, MQ, and so on;

• broker as a buffer between the remote agent and the central agent, using Redis implementation, one can improve the performance of the system, the second is to improve the system's reliability, When the central agent fails to extract data, the data is kept in Redis, not lost;

• elasticsearch for storing final data and providing search functionality;

• kibana provides a simple, rich web interface with data from Elasticsearch to support a variety of queries, statistics and displays;

  
  

2, the deployment of the machine

192.168.1.140 # Redis Server, role broker
192.168.1.140 # Logstash role Indexer Server, integrated Elasticsearch, Kibana, must have installation Web service
192.168.1.132 # Nginx Server, role production server, Logstash need to collect its logs

The version of the software selected here:

logstash-1.4.2

elasticsearch-1.4.2

redis-2.6.16

Kibana is in the Logstash.

There is a compatibility issue between these software, please use other alternative version of the attention of the students.

2.1 Installing logstash-1.4.2

Yum-y Install JAVA-1.7.0-OPENJDK installation Logstash requires JDK support

wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz

Tar xzvf logstash-1.4.2.tar.gz-c/app/&& Mv/app/logstash-1.4.2/app/logstash

Mkdir-p/app/logstash/conf

Note: If Yum is having this problem:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/66/wKiom1XNmPGimkTjAAEhdUP90ls708.jpg "title=" 3.png " alt= "Wkiom1xnmpgimktjaaehdup90ls708.jpg"/>

Need to follow Epel

RPM-IVH http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

RPM--import/etc/pki/rpm-gpg/rpm-gpg-key-epel-6

Test installation

Cd/app/logstash/bin

./Logstash-e' input {stdin {}} ' output {stdout {}} '

Enter "Good job", if similar, indicating Logstash is working correctly

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/60/wKioL1XNY4WistO7AABZM1ME6tM652.jpg "title=" 2.png " alt= "Wkiol1xny4wisto7aabzm1me6tm652.jpg"/>

2.2 Installing elasticsearch-1.4.2

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.tar.gz

Tar xzvf elasticsearch-1.4.2.tar.gz-c/app/

Cd/app/elasticsearch-1.4.2/config

Modifying a configuration file elasticsearch.yml

Discovery.zen.ping.multicast.enabled:false #关闭广播, if the LAN has a machine open 9300 port, the service will not start

network.host:192.168.1.140 #指定主机地址, in fact, is optional, but it is best to specify that the HTTP connection error will be reported when the integration with Kibana is followed (the visual representation is that it is listening::: 9200 instead of 0.0.0.0:9 200)

Http.cors.allow-origin: "/.*/"

Http.cors.enabled:true #这2项都是解决跟kibana集成的问题, the error is that your elasticsearch version is too low

Start Elasticsearch

Cd/app/elasticsearch-1.4.2/bin

The./elasticsearch # configuration phase is recommended to start directly, the log output to the STDOUT,-D option means to start in daemon manner, and if no error occurs, the service starts normally

Test Logstash interacting with Elasticsearch data

/app/logstash/bin/logstash-e ' input {stdin {}} output {elasticsearch {host = 192.168.1.140}} '

Enter you know

Curl ' Http://192.168.1.140:9200/_search?pretty ' # if there is output and no error indicates successful server interaction

Note: May appear the following error message, I do not know what the reason, I did not control but do not delay behind the erection who knows how this is going to be or how to solve the message thank you!!

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/61/wKioL1XNisGzLrUOAAEiDE6asV0557.jpg "title=" 6.png " alt= "Wkiol1xnisgzlruoaaeide6asv0557.jpg"/>

2.3 Installing Kibana

Cd/app/logstash/vendor

Vim kibana/config.js #elasticsearch: "/http"/"+window.location.hostname+": 9200 ", modified to" http://192.168.1.140:9200 "

CP-RV configuration of kibana/var/www/html copy Web pages

/ETC/INIT.D/HTTPD Start/service httpd Start

Ability to access URLs http://192.168.1.140/kibana/index.html see such a screen

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/71/65/wKiom1XNiaXw_7aZAAUjnHG_bKg098.jpg "title=" 7.png " alt= "Wkiom1xniaxw_7azaaujnhg_bkg098.jpg"/>

2.4 Installing Redis

#tar XZVF redis-2.6.16.tar.gz-c/app

#cd/app/redis-2.6.16 && mkdir conf

#make target=linux26 (Error may be missing some compiled tools, follow the prompts, yum installation is OK

If the following error occurs:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/65/wKiom1XNir-wXt5zAAD5kpEPQzM970.jpg "title=" 1.png " alt= "Wkiom1xnir-wxt5zaad5kpepqzm970.jpg"/>

Can execute make MALLOC=LIBC

Detailed reasons can be directly Baidu out

Http://www.41443.com/HTML/DB2/20150525/373733.html

）

#./src/redis-server redis.conf # daemonize Yes using the default configuration file

2.5 Integrated Logstash Redis

Vim/app/logstash/conf/nginx_acces.conf writes the following:

Input {
Redis {
Host = ' 192.168.1.140 ' # I am convenient to test without specifying password, preferably specify password
data_type = ' list '
Port = "6379"
Key = ' Logstash:redis ' #自定义
Type = ' Redis-input ' #自定义
}
}
Output {
Elasticsearch {
Host = "192.168.1.140" # because Redis and Elasticsearch are on a single machine, IP is a

codec = "JSON"
protocol = "http" #版本1.0+ must specify protocol HTTP
}
}

Verifying the configuration file

Cd/app/logstash

#bin/logstash-f./conf/nginx_access.conf-t # error after start

#bin/logstash-f./conf/nginx_access.conf--verbose # to check for errors--debug

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/65/wKiom1XNjf6B3KS5AAPw4h7mTzs310.jpg "title=" 2.png " alt= "Wkiom1xnjf6b3ks5aapw4h7mtzs310.jpg"/>

Boot success with warning no tube

2.6 Log access to another machine installation The Logstash process is the same as above, but the nginx_access.conf is configured as follows

2. Input {
   

4. File {
   

6. type => "nginx_access"
   

8. Path = > path to the "/var/log/nginx/access.log" log
   

10. }
    

12. }

14. 
    

16. 
    

18. Output {
    

20. stdout {codec => rubydebug}
    

22. Redis {
    

24. Host = > ' 192.168.1.140 ' docking of the Redisip
    

26. data_type = > ' list '
    

28. Key = > ' Logstash:redis '
    

30. }
    

32. }

To this configuration, the following acceptance results

Refresh Nginx Log

Then the Logstash on 132 can see the following

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/62/wKioL1XNkrmQxXlcAAESfim6sK8002.jpg "title=" 1.png " alt= "Wkiol1xnkrmqxxlcaaesfim6sk8002.jpg"/>

Then on the 140 end you can see:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/62/wKioL1XNk4bxLPYhAAJ_CHFHCws783.jpg "title=" 1.png " alt= "Wkiol1xnk4bxlpyhaaj_chfhcws783.jpg"/>

The focus is to visit the final page of http://192.168.1.140/kibana/index.html acceptance

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/62/wKioL1XNk_DzfOBVAAUMo7CPb94248.jpg "title=" 1.png " alt= "Wkiol1xnk_dzfobvaaumo7cpb94248.jpg"/>

This is a simple format that has been designed and can certainly be defined by itself.

Method of Definition: http://blog.chinaunix.net/xmlrpc.php?r=blog/article&id=4938039&uid=24940078

The interface after the point in is this:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/62/wKioL1XNlOmTlnMVAAVLkn5PkRE466.jpg "title=" 1.png " alt= "Wkiol1xnlomtlnmvaavlkn5pkre466.jpg"/>

3 Installing the plug-in installation Bigdesk

To know the entire plugin list, please visit the http://www.elasticsearch.org/guide/reference/modules/plugins/plug-in or a lot of, personally think more worthy of attention to have the following several, other look at your needs, For example, if you want to import data, you have to focus on the river.

The plug-in can view the JVM information of the cluster, disk IO, index creation Delete information, etc., it is suitable to find the system bottleneck, monitor the cluster status and so on, can execute the following command to install, or access the project address: Https://github.com/lukas-vlcek/bigdesk

The first one:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/66/wKiom1XNlJ3S54apAAQxjqyDg34123.jpg "title=" 1.png " alt= "Wkiom1xnlj3s54apaaqxjqydg34123.jpg"/>

Here, I'm following Https://github.com/lukas-vlcek/bigdesk.

The second type of git clone above is done in the same way (the first is unsuccessful ...). ）

First make sure that Git is installed

Yum install-y git

You will then follow the above command:

git clone https://github.com/lukas-vlcek/bigdesk.git

CD bigdesk/

git tag

[... some tags left out for brevity ...]
v2.2.2
v2.2.3
v2.4.0

git checkout v2.4.0

Then copy the entire Bigdesk directory to HTTP

Cp-ar bigdesk/var/www/html/

You can then access the

Http://192.168.1.140/bigdesk

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/71/66/wKiom1XNlgzAotbkAAGnBUf5Pl4825.jpg "title=" 1.png " alt= "Wkiom1xnlgzaotbkaagnbuf5pl4825.jpg"/>

First modify the host and then connect and then will come out a small icon (in the results display) Click on the small icon will be able to display the monitoring options.

Disclaimer: This article refers to the following blogs, but I personally set up the whole process, the whole process of new control and optimization.

http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582

Http://www.mamicode.com/info-detail-475881.html

http://nkcoder.github.io/blog/20141031/elkr-log-platform-deploy/

This article is from the "New One" blog, please be sure to keep this source http://welcomeweb.blog.51cto.com/10487763/1684696

CENTOS6.5 installation Log Analysis Elk Elasticsearch + logstash + Redis + Kibana