開源分布式搜尋平台ELK(Elasticsearch+Logstash+Kibana)+Redis+Syslog-ng實現日誌即時搜尋

轉自：http://blog.c1gstudio.com/archives/1765

logstash + elasticsearch + Kibana+Redis+Syslog-ng

Elasticsearch是一個基於Lucene構建的開源，分布式，RESTful搜尋引擎。設計用於雲端運算中，能夠達到即時搜尋，穩定，可靠，快速，安裝使用方便。支援通過HTTP使用JSON進行資料索引。

logstash是一個應用程式記錄檔、事件的傳輸、處理、管理和搜尋的平台。你可以用它來統一對應用程式記錄檔進行收集管理，提供 Web 介面用於查詢和統計。其實logstash是可以被別的替換，比如常見的fluented

Kibana是一個為 Logstash 和 Elasticsearch 提供的日誌分析的 Web 介面。可使用它對日誌進行高效的搜尋、可視化、分析等各種操作。

Redis是一個高效能的記憶體key-value資料庫,非必需安裝,可以防止資料丟失.

參考:
http://www.logstash.net/
http://chenlinux.com/2012/10/21/elasticearch-simple-usage/
http://www.elasticsearch.cn
http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz?AuthParam=1408083909_3bf5b46169faab84d36cf74407132bba
http://curran.blog.51cto.com/2788306/1263416
http://storysky.blog.51cto.com/628458/1158707/
http://zhumeng8337797.blog.163.com/blog/static/10076891420142712316899/
http://enable.blog.51cto.com/747951/1049411
http://chenlinux.com/2014/06/11/nginx-access-log-to-elasticsearch/
http://www.w3c.com.cn/%E5%BC%80%E6%BA%90%E5%88%86%E5%B8%83%E5%BC%8F%E6%90%9C%E7%B4%A2%E5%B9%B3%E5%8F%B0elkelasticsearchlogstashkibana%E5%85%A5%E9%97%A8%E5%AD%A6%E4%B9%A0%E8%B5%84%E6%BA%90%E7%B4%A2%E5%BC%95
http://woodygsd.blogspot.com/2014/06/an-adventure-with-elk-or-how-to-replace.html
http://www.ricardomartins.com.br/enviando-dados-externos-para-a-stack-elk/
http://tinytub.github.io/logstash-install.html

http://jamesmcfadden.co.uk/securing-elasticsearch-with-nginx/
https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns
http://zhaoyanblog.com/archives/319.html
http://www.vpsee.com/2014/05/install-and-play-with-elasticsearch/

ip說明
118.x.x.x/16 為用戶端ip
192.168.0.39和61.x.x.x為ELK的內網和外網ip 安裝JDK

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html tar zxvf jdk-7u67-linux-x64.tar.gz\?AuthParam\=1408083909_3bf5b46169faab84d36cf74407132b mv jdk1.7.0_67 /usr/local/ cd /usr/local/ ln -s jdk1.7.0_67 jdk chown -R root:root jdk/

配置環境變數
vi /etc/profile export JAVA_HOME=/usr/local/jdk    export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$PATH export REDIS_HOME=/usr/local/redis export ES_HOME=/usr/local/elasticsearch export ES_CLASSPATH=$ES_HOME/config

變數生效
source /etc/profile

驗證版本
java -version java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

如果之前安裝過java,可以先卸載
rpm -qa |grep java
java-1.6.0-openjdk-1.6.0.0-1.24.1.10.4.el5
java-1.6.0-openjdk-devel-1.6.0.0-1.24.1.10.4.el5

rpm -e java-1.6.0-openjdk-1.6.0.0-1.24.1.10.4.el5 java-1.6.0-openjdk-devel-1.6.0.0-1.24.1.10.4.el5 安裝redis

http://redis.io/ wget http://download.redis.io/releases/redis-2.6.17.tar.gz tar zxvf redis-2.6.17.tar.gz mv redis-2.6.17 /usr/local/ cd /usr/local ln -s redis-2.6.17 redis cd /usr/local/redis make make install

cd utils
./install_server.sh Please select the redis port for this instance: [6379] Selecting default: 6379 Please select the redis config file name [/etc/redis/6379.conf] Selected default - /etc/redis/6379.conf Please select the redis log file name [/var/log/redis_6379.log] Selected default - /var/log/redis_6379.log Please select the data directory for this instance [/var/lib/redis/6379] Selected default - /var/lib/redis/6379 Please select the redis executable path [/usr/local/bin/redis-server]

編輯設定檔
vi /etc/redis/6379.conf daemonize yes port 6379 timeout 300 tcp-keepalive 60

啟動
/etc/init.d/redis_6379 start

exists, process is already running or crashed
如報這個錯,需要編輯下/etc/init.d/redis_6379,去除頭上的\n

加入自動啟動
chkconfig –add redis_6379 安裝Elasticsearch

http://www.elasticsearch.org/
http://www.elasticsearch.cn
叢集安裝只要節點在同一網段下，設定一致的cluster.name，啟動的Elasticsearch即可相互檢測到對方，組成叢集 wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz tar zxvf elasticsearch-1.3.2.tar.gz mv elasticsearch-1.3.2 /usr/local/ cd /usr/local/ ln -s elasticsearch-1.3.2 elasticsearch elasticsearch/bin/elasticsearch -f [2014-08-20 13:19:05,710][INFO ][node                     ] [Jackpot] version[1.3.2], pid[19320], build[dee175d/2014-08-13T14:29:30Z] [2014-08-20 13:19:05,727][INFO ][node                     ] [Jackpot] initializing ... [2014-08-20 13:19:05,735][INFO ][plugins                  ] [Jackpot] loaded [], sites [] [2014-08-20 13:19:10,722][INFO ][node                     ] [Jackpot] initialized [2014-08-20 13:19:10,723][INFO ][node                     ] [Jackpot] starting ... [2014-08-20 13:19:10,934][INFO ][transport                ] [Jackpot] bound_address {inet[/0.0.0.0:9301]}, publish_address {inet[/61.x.x.x:9301]} [2014-08-20 13:19:10,958][INFO ][discovery                ] [Jackpot] elasticsearch/5hUOX-2ES82s_0zvI9BUdg [2014-08-20 13:19:14,011][INFO ][cluster.service          ] [Jackpot] new_master [Jackpot][5hUOX-2ES82s_0zvI9BUdg][Impala][inet[/61.x.x.x:9301]], reason: zen-disco-join (elected_as_master)