The purpose of building this platform is to facilitate the operation of the research and development of the log query. Kibana a free web shell; Logstash integrates various collection log plug-ins, or is a good regular cutting log tool; Elasticsearch an open-source search engine framework that supports the cluster architecture approach.
1 Installation Requirements 1.1 theoretical topology
1.2 Installation Environment 1.2.1 hardware environment
192.168.50.62 (HP DL 385 G7, ram:12g, Cpu:amd 6128, Disk:sas 146*4)
192.168.50.98 (HP DL 385 G7, ram:12g, Cpu:amd 6128, Disk:sas 146*6)
192.168.10.42 (Xen virtual machine, ram:8g, cpu:x4, disk:100g)
1.2.2 Operating System
CentOS 5.6 X64
1.2.3 Web-server Basic Environment
nginx+php (Installation process skipped)
1.2.4 Software List
JDK 1.6.0_25
Logstash-1.1.0-monolithic.jar
Elasticsearch-0.18.7.zip
Redis-2.4.12.tar.gz
Kibana
1.3 Get Method 1.3.1 JDK Fetch path
Http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u25-download-346242.html
1.3.2 Logstash Get Path
Http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
1.3.3 Elasticsearch Get Path
Https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.zip
1.3.4 Kibana Get Path
Http://github.com/rashidkpc/Kibana/tarball/master
2 Installation Steps 2.1 JDK download and installation
Basic Installation
wget Http://download.oracle.com/otn-pub/java/jdk/6u25-b06/jdk-6u25-linux-x64.bin
SH jdk-6u25-linux-x64.bin
Mkdir-p/usr/java
mv./jdk1.6.0_25/usr/java
Ln–s/usr/java/jdk1.6.0_25/usr/java/default
Edit the/etc/profile file and add the following line content
Export Java_home=/usr/java/default
Export path= $JAVA _home/bin: $PATH
Export classpath=.: $JAVA _home/lib/tools.jar: $JAVA _home/lib/dt.jar: $CLASSPATH
Refresh Environment Variables
Source/etc/profile
2.2 Redis Download and installation
wget http://redis.googlecode.com/files/redis-2.4.14.tar.gz
Make–j24
Make install
Mkdir-p/data/redis
cd/data/redis/
mkdir {DB,LOG,ETC}
2.3 Elasticsearch Download and installation
cd/data/
Mkdir–p elasticsearch && CD Elasticsearch
wget--no-check-certificate Https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.zip
Unzip Elasticsearch-0.18.7.zip
2.4 Logstash Download and installation
mkdir–p/data/logstash/&& Cd/data/logstash
wget Http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
2.5 Kibana Download and installation
wget Http://github.com/rashidkpc/Kibana/tarball/master--no-check-certificate
Tar zxvf Master
3 related configuration and startup 3.1 Redis configuration and Startup 3.1.1 Profile
Vim/data/redis/etc/redis.conf
#----------------------------------------------------
#this is the config file for Redis
Pidfile/var/run/redis.pid
Port 6379
Timeout 0
LogLevel verbose
Logfile/data/redis/log/redis.log
Databases 16
Save 900 1
Save 300 10
Save 60 10000
Rdbcompression Yes
Dbfilename Dump.rdb
dir/data/redis/db/
Slave-serve-stale-data Yes
AppendOnly No
Appendfsync everysec
No-appendfsync-on-rewrite No
Auto-aof-rewrite-percentage 100
Auto-aof-rewrite-min-size 64MB
Slowlog-log-slower-than 10000
Slowlog-max-len 128
Vm-enabled No
Vm-swap-file/tmp/redis.swap
Vm-max-memory 0
Vm-page-size 32
Vm-pages 134217728
Vm-max-threads 4
Hash-max-zipmap-entries 512
Hash-max-zipmap-value 64
List-max-ziplist-entries 512
List-max-ziplist-value 64
Set-max-intset-entries 512
Zset-max-ziplist-entries 128
Zset-max-ziplist-value 64
activerehashing Yes
3.1.2 Redis Boot
[Email protected]_2 redis]# redis-server/data/redis/etc/redis.conf &
3.2 Elasticsearch Configuration and startup 3.2.1 Elasticsearch Boot
[Email protected]_2 redis]#/data/elasticsearch/elasticsearch-0.18.7/bin/elasticsearch–p. /esearch.pid &
3.2.2 Elasticsearch Cluster configuration
Curl 127.0.0.1:9200/_cluster/nodes/192.168.50.62
3.3 Logstash Configuration and startup 3.3.1 Logstash configuration file
Input {
Redis {
Host = "192.168.50.98"
data_type = "List"
Key = "Logstash:redis"
Type = "Redis-input"
}
}
Filter {
Grok {
Type = "Linux-syslog"
Pattern = "%{syslogline}"
}
Grok {
Type = "Nginx-access"
Pattern = "%{nginxaccesslog}"
}
}
Output {
Elasticsearch {
Host = "192.168.50.62"
}
}
3.3.2 Logstash start as Index
Java-jar Logstash.jar agent-f my.conf &
3.3.3 Logstash start as Agent
Configuration file
Input {
file{
Type = "Linux-syslog"
Path = ["/var/log/*.log", "/var/log/messages", "/var/log/syslog"]
}
File {
Type = "Nginx-access"
Path = "/usr/local/nginx/logs/access.log"
}
File {
Type = "Nginx-error"
Path = "/usr/local/nginx/logs/error.log"
}
}
Output {
Redis {
Host = "192.168.50.98"
data_type = "List"
Key = "Logstash:redis"
}
}
Agent Startup
Java-jar Logstash-1.1.0-monolithic.jar agent-f shipper.conf &
3.3.4 Kibana Configuration
First add site configuration in Nginx
server {
Listen 80;
server_name logstash.test.com;
Index index.php;
root/usr/local/nginx/html;
#charset Koi8-r;
#access_log Logs/host.access.log Main;
Location ~. *\. (PHP|PHP5) $
{
#fastcgi_pass Unix:/tmp/php-cgi.sock;
Fastcgi_pass 127.0.0.1:9000;
Fastcgi_index index.php;
Include fastcgi.conf;
}
}
4 Performance Tuning 4.1 Elasticsearch tuning 4.1.1 JVM Tuning
Edit Elasticsearch.in.sh File
Es_classpath= $ES _classpath: $ES _home/lib/*: $ES _home/lib/sigar/*
If ["X$es_min_mem" = "x"]; Then
es_min_mem=4g
Fi
If ["X$es_max_mem" = "x"]; Then
es_max_mem=4g
Fi
4.1.2 Elasticsearch Index Compression
Vim index_elastic.sh
#!/bin/bash
#comperssion the data for Elasticsearch now
Date= ' Date +%y.%m.%d '
# compression the new index;
/usr/bin/curl-xput http://localhost:9200/logstash-$date/nginx-access/_mapping-d ' {"nginx-access": {"_source": {" Compress ": true}} '
echo ""
/usr/bin/curl-xput http://localhost:9200/logstash-$date/nginx-error/_mapping-d ' {"Nginx-error": {"_source": {" Compress ": true}} '
echo ""
/usr/bin/curl-xput http://localhost:9200/logstash-$date/linux-syslog/_mapping-d ' {"Linux-syslog": {"_source": {" Compress ": true}} '
echo ""
Save the script and execute
SH index_elastic.sh
5 using 5.1 logstash query page
Access http://logstash.test.com using Firefox or Google Chrome
Kibana+logstash+elasticsearch Log Query system