Open source Distributed search Platform Elk (elasticsearch+logstash+kibana) +redis+syslog-ng realize log real-time search

Turn from: http://blog.c1gstudio.com/archives/1765

Logstash + Elasticsearch + kibana+redis+syslog-ng

Elasticsearch is an open source, distributed, restful search engine built on Lucene. Designed for cloud computing, to achieve real-time search, stable, reliable, fast, easy to install and use. Supports the use of JSON for data indexing over HTTP.

Logstash is a platform for application log, event transmission, processing, management, and search. You can use it to unify the collection management of application logs, providing WEB interfaces for querying and counting. In fact, Logstash can be replaced by other, such as common fluented

Kibana is a WEB interface for log analysis provided for Logstash and Elasticsearch. It can be used to efficiently search, visualize, analyze and perform various operations on the log.

Redis is a high-performance memory Key-value database that is not required for installation to prevent data loss.

Reference:
http://www.logstash.net/
http://chenlinux.com/2012/10/21/elasticearch-simple-usage/
http://www.elasticsearch.cn
Http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz? Authparam=1408083909_3bf5b46169faab84d36cf74407132bba
http://curran.blog.51cto.com/2788306/1263416
http://storysky.blog.51cto.com/628458/1158707/
http://zhumeng8337797.blog.163.com/blog/static/10076891420142712316899/
http://enable.blog.51cto.com/747951/1049411
http://chenlinux.com/2014/06/11/nginx-access-log-to-elasticsearch/
http://www.w3c.com.cn/%E5%BC%80%E6%BA%90%E5%88%86%E5%B8%83%E5%BC%8F%E6%90%9C%E7%B4%A2%E5%B9%B3%E5%8F% B0elkelasticsearchlogstashkibana%e5%85%a5%e9%97%a8%e5%ad%a6%e4%b9%a0%e8%b5%84%e6%ba%90%e7%b4%a2%e5%bc%95
Http://woodygsd.blogspot.com/2014/06/an-adventure-with-elk-or-how-to-replace.html
http://www.ricardomartins.com.br/enviando-dados-externos-para-a-stack-elk/
Http://tinytub.github.io/logstash-install.html

http://jamesmcfadden.co.uk/securing-elasticsearch-with-nginx/
Https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns
Http://zhaoyanblog.com/archives/319.html
http://www.vpsee.com/2014/05/install-and-play-with-elasticsearch/

IP description
118.X.X.X/16 for Client IP
192.168.0.39 and 61.x.x.x for Elk Intranet and extranet IP installation JDK

Http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html Tar zxvf Jdk-7u67-linux-x64.tar.gz\? authparam\=1408083909_3bf5b46169faab84d36cf74407132b MV Jdk1.7.0_67/usr/local/cd/usr/local/ln-s jdk1.7.0_67 JDK Chown-r Root:root jdk/

Configure Environment variables
Vi/etc/profile Export JAVA_HOME=/USR/LOCAL/JDK export jre_home= $JAVA _home/jre export classpath=.: $JAVA _home/lib/ Dt.jar: $JAVA _home/lib/tools.jar: $JRE _home/lib: $CLASSPATH export path= $JAVA _home/bin: $PATH export redis_home=/usr/ Local/redis Export Es_home=/usr/local/elasticsearch export es_classpath= $ES _home/config

Variable takes effect
Source/etc/profile

Verify version
Java-version java Version "1.7.0_67" Java (tm) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot (tm) 64-bit Server VM (build 24.65-b04, Mixed mode)

If you have previously installed Java, you can first uninstall
Rpm-qa |grep Java
Java-1.6.0-openjdk-1.6.0.0-1.24.1.10.4.el5
Java-1.6.0-openjdk-devel-1.6.0.0-1.24.1.10.4.el5

RPM-E java-1.6.0-openjdk-1.6.0.0-1.24.1.10.4.el5 java-1.6.0-openjdk-devel-1.6.0.0-1.24.1.10.4.el5 installation Redis

Http://redis.io/wget http://download.redis.io/releases/redis-2.6.17.tar.gz tar zxvf redis-2.6.17.tar.gz MV Redis-2.6.17/usr/local/cd/usr/local ln-s redis-2.6.17 redis cd/usr/local/redis make make install

CD Utils
./install_server.sh Please select the Redis port for this instance: [6379] Selecting default:6379 Please select the Redis Config file name [/etc/redis/6379.conf] Selected Default-/etc/redis/6379.conf Please select the Redis log file name [/V Ar/log/redis_6379.log] Selected Default-/var/log/redis_6379.log, select the data directory for this instance [/var /LIB/REDIS/6379] Selected Default-/var/lib/redis/6379 Please select the Redis executable path [/usr/local/bin/redis-ser Ver

Edit configuration file
vi/etc/redis/6379.conf daemonize Yes Port 6379 Timeout tcp-keepalive 60

Start
/etc/init.d/redis_6379 start

exists, process is already running or crashed
If you report this mistake, you need to edit the/etc/init.d/redis_6379 to remove \ n on the head.

Join Auto Start
Chkconfig–add redis_6379 Installation Elasticsearch

http://www.elasticsearch.org/
http://www.elasticsearch.cn
cluster installation as long as the node in the same network segment, set a consistent cluster.name, Elasticsearch can be started to detect each other, forming a cluster wget https://download.elasticsearch.org/ Elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz tar zxvf elasticsearch-1.3.2.tar.gz mv elasticsearch-1.3.2/ Usr/local/cd/usr/local/ln-s elasticsearch-1.3.2 elasticsearch elasticsearch/bin/elasticsearch-f [2014-08-20 13:19:05,710][info][node                     ] [Jackpot] Version[1 .3.2], pid[19320], build[dee175d/2014-08-13t14:29:30z] [2014-08-20 13:19:05,727][info][node                     ] [Jackpot] initializing ... [2014-08-20 13:19:05,735] [INFO] [plugins                  [Jackpot] loaded [], sites [] [2014-08-20 13:19:10,722][info][node                     ] [jackpot] initialized [2014-08-20 13:19:10,723][info][node          &NBSP          ] [Jackpot] starting ... [2014-08-20 13:19:10,934] [INFO] [transport                [Jackpot] bound_address {inet[/0.0.0.0:9301]}, publish_address {inet[/61.x.x.x:9301]} [2014-08-20 13:19:10,958][INFO ][discovery               ] [Jackpot] ELASTICSEARCH/5HUOX-2ES82S_0ZVI9BUDG [2014 -08-20 13:19:14,011][info][cluster.service         ] [jackpot] New_master [jackpot][ 5HUOX-2ES82S_0ZVI9BUDG][IMPALA][INET[/61.X.X.X:9301]], Reason:zen-disco-join (elected_as_master)