1. Workflow of Log Platform
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/71/5F/wKioL1XNWHGwPB_ZAAErAE7qZjQ757.jpg "title=" 1.png " alt= "Wkiol1xnwhgwpb_zaaerae7qzjq757.jpg"/>
-
-
shipper means log collection, using Logstash to collect log data from various sources, such as system logs, files, Redis, MQ, and so on;
-
broker as a buffer between the remote agent and the central agent, using Redis implementation, one can improve the performance of the system, the second is to improve the system's reliability, When the central agent fails to extract data, the data is kept in Redis, not lost;
-
-
elasticsearch for storing final data and providing search functionality;
-
kibana provides a simple, rich web interface with data from Elasticsearch to support a variety of queries, statistics and displays;
2, the deployment of the machine
192.168.1.140 # Redis Server, role broker
192.168.1.140 # Logstash role Indexer Server, integrated Elasticsearch, Kibana, must have installation Web service
192.168.1.132 # Nginx Server, role production server, Logstash need to collect its logs
The version of the software selected here:
logstash-1.4.2
elasticsearch-1.4.2
redis-2.6.16
Kibana is in the Logstash.
There is a compatibility issue between these software, please use other alternative version of the attention of the students.
2.1 Installing logstash-1.4.2
Yum-y Install JAVA-1.7.0-OPENJDK installation Logstash requires JDK support
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
Tar xzvf logstash-1.4.2.tar.gz-c/app/&& Mv/app/logstash-1.4.2/app/logstash
Mkdir-p/app/logstash/conf
Note: If Yum is having this problem:
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/66/wKiom1XNmPGimkTjAAEhdUP90ls708.jpg "title=" 3.png " alt= "Wkiom1xnmpgimktjaaehdup90ls708.jpg"/>
Need to follow Epel
RPM-IVH http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
RPM--import/etc/pki/rpm-gpg/rpm-gpg-key-epel-6
Test installation
Cd/app/logstash/bin
./Logstash-e' input {stdin {}} ' output {stdout {}} '
Enter "Good job", if similar, indicating Logstash is working correctly
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/60/wKioL1XNY4WistO7AABZM1ME6tM652.jpg "title=" 2.png " alt= "Wkiol1xny4wisto7aabzm1me6tm652.jpg"/>
2.2 Installing elasticsearch-1.4.2
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.tar.gz
Tar xzvf elasticsearch-1.4.2.tar.gz-c/app/
Cd/app/elasticsearch-1.4.2/config
Modifying a configuration file elasticsearch.yml
Discovery.zen.ping.multicast.enabled:false #关闭广播, if the LAN has a machine open 9300 port, the service will not start
network.host:192.168.1.140 #指定主机地址, in fact, is optional, but it is best to specify that the HTTP connection error will be reported when the integration with Kibana is followed (the visual representation is that it is listening::: 9200 instead of 0.0.0.0:9 200)
Http.cors.allow-origin: "/.*/"
Http.cors.enabled:true #这2项都是解决跟kibana集成的问题, the error is that your elasticsearch version is too low
Start Elasticsearch
Cd/app/elasticsearch-1.4.2/bin
The./elasticsearch # configuration phase is recommended to start directly, the log output to the STDOUT,-D option means to start in daemon manner, and if no error occurs, the service starts normally
Test Logstash interacting with Elasticsearch data
/app/logstash/bin/logstash-e ' input {stdin {}} output {elasticsearch {host = 192.168.1.140}} '
Enter you know
Curl ' Http://192.168.1.140:9200/_search?pretty ' # if there is output and no error indicates successful server interaction
Note: May appear the following error message, I do not know what the reason, I did not control but do not delay behind the erection who knows how this is going to be or how to solve the message thank you!!
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/61/wKioL1XNisGzLrUOAAEiDE6asV0557.jpg "title=" 6.png " alt= "Wkiol1xnisgzlruoaaeide6asv0557.jpg"/>
2.3 Installing Kibana
Cd/app/logstash/vendor
Vim kibana/config.js #elasticsearch: "/http"/"+window.location.hostname+": 9200 ", modified to" http://192.168.1.140:9200 "
CP-RV configuration of kibana/var/www/html copy Web pages
/ETC/INIT.D/HTTPD Start/service httpd Start
Ability to access URLs http://192.168.1.140/kibana/index.html see such a screen
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/71/65/wKiom1XNiaXw_7aZAAUjnHG_bKg098.jpg "title=" 7.png " alt= "Wkiom1xniaxw_7azaaujnhg_bkg098.jpg"/>
2.4 Installing Redis
#tar XZVF redis-2.6.16.tar.gz-c/app
#cd/app/redis-2.6.16 && mkdir conf
#make target=linux26 (Error may be missing some compiled tools, follow the prompts, yum installation is OK
If the following error occurs:
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/65/wKiom1XNir-wXt5zAAD5kpEPQzM970.jpg "title=" 1.png " alt= "Wkiom1xnir-wxt5zaad5kpepqzm970.jpg"/>
Can execute make MALLOC=LIBC
Detailed reasons can be directly Baidu out
Http://www.41443.com/HTML/DB2/20150525/373733.html
)
#./src/redis-server redis.conf # daemonize Yes using the default configuration file
2.5 Integrated Logstash Redis
Vim/app/logstash/conf/nginx_acces.conf writes the following:
Input {
Redis {
Host = ' 192.168.1.140 ' # I am convenient to test without specifying password, preferably specify password
data_type = ' list '
Port = "6379"
Key = ' Logstash:redis ' #自定义
Type = ' Redis-input ' #自定义
}
}
Output {
Elasticsearch {
Host = "192.168.1.140" # because Redis and Elasticsearch are on a single machine, IP is a
codec = "JSON"
protocol = "http" #版本1.0+ must specify protocol HTTP
}
}
Verifying the configuration file
Cd/app/logstash
#bin/logstash-f./conf/nginx_access.conf-t # error after start
#bin/logstash-f./conf/nginx_access.conf--verbose # to check for errors--debug
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/65/wKiom1XNjf6B3KS5AAPw4h7mTzs310.jpg "title=" 2.png " alt= "Wkiom1xnjf6b3ks5aapw4h7mtzs310.jpg"/>
Boot success with warning no tube
2.6 Log access to another machine installation The Logstash process is the same as above, but the nginx_access.conf is configured as follows
Input {
File {
type => "nginx_access"
Path = > path to the "/var/log/nginx/access.log" log
}
}
Output {
stdout {codec => rubydebug}
Redis {
Host = > ' 192.168.1.140 ' docking of the Redisip
data_type = > ' list '
Key = > ' Logstash:redis '
}
}
To this configuration, the following acceptance results
Refresh Nginx Log
Then the Logstash on 132 can see the following
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/71/62/wKioL1XNkrmQxXlcAAESfim6sK8002.jpg "title=" 1.png " alt= "Wkiol1xnkrmqxxlcaaesfim6sk8002.jpg"/>
Then on the 140 end you can see:
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/62/wKioL1XNk4bxLPYhAAJ_CHFHCws783.jpg "title=" 1.png " alt= "Wkiol1xnk4bxlpyhaaj_chfhcws783.jpg"/>
The focus is to visit the final page of http://192.168.1.140/kibana/index.html acceptance
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/62/wKioL1XNk_DzfOBVAAUMo7CPb94248.jpg "title=" 1.png " alt= "Wkiol1xnk_dzfobvaaumo7cpb94248.jpg"/>
This is a simple format that has been designed and can certainly be defined by itself.
Method of Definition: http://blog.chinaunix.net/xmlrpc.php?r=blog/article&id=4938039&uid=24940078
The interface after the point in is this:
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/62/wKioL1XNlOmTlnMVAAVLkn5PkRE466.jpg "title=" 1.png " alt= "Wkiol1xnlomtlnmvaavlkn5pkre466.jpg"/>
3 Installing the plug-in installation Bigdesk
To know the entire plugin list, please visit the http://www.elasticsearch.org/guide/reference/modules/plugins/plug-in or a lot of, personally think more worthy of attention to have the following several, other look at your needs, For example, if you want to import data, you have to focus on the river.
The plug-in can view the JVM information of the cluster, disk IO, index creation Delete information, etc., it is suitable to find the system bottleneck, monitor the cluster status and so on, can execute the following command to install, or access the project address: Https://github.com/lukas-vlcek/bigdesk
The first one:
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/71/66/wKiom1XNlJ3S54apAAQxjqyDg34123.jpg "title=" 1.png " alt= "Wkiom1xnlj3s54apaaqxjqydg34123.jpg"/>
Here, I'm following Https://github.com/lukas-vlcek/bigdesk.
The second type of git clone above is done in the same way (the first is unsuccessful ...). )
First make sure that Git is installed
Yum install-y git
You will then follow the above command:
git clone https://github.com/lukas-vlcek/bigdesk.git
CD bigdesk/
git tag
[... some tags left out for brevity ...]
v2.2.2
v2.2.3
v2.4.0
git checkout v2.4.0
Then copy the entire Bigdesk directory to HTTP
Cp-ar bigdesk/var/www/html/
You can then access the
Http://192.168.1.140/bigdesk
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/71/66/wKiom1XNlgzAotbkAAGnBUf5Pl4825.jpg "title=" 1.png " alt= "Wkiom1xnlgzaotbkaagnbuf5pl4825.jpg"/>
First modify the host and then connect and then will come out a small icon (in the results display) Click on the small icon will be able to display the monitoring options.
Disclaimer: This article refers to the following blogs, but I personally set up the whole process, the whole process of new control and optimization.
http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582
Http://www.mamicode.com/info-detail-475881.html
http://nkcoder.github.io/blog/20141031/elkr-log-platform-deploy/
This article is from the "New One" blog, please be sure to keep this source http://welcomeweb.blog.51cto.com/10487763/1684696
CENTOS6.5 installation Log Analysis Elk Elasticsearch + logstash + Redis + Kibana