Elasticsearch+logstash+kinaba+redis Log Analysis System

First, Introduction

1. Composition

Elk consists of three parts: Elasticsearch, Logstash and Kibana.

Elasticsearch is an open source distributed search engine, it features: distributed, 0 configuration, automatic discovery, Index auto-shard, index copy mechanism, RESTful style interface, multi-data source, automatic search load, etc.

Logstash is a fully open source tool that collects, analyzes, and stores your logs for later use

Kibana is an open source and free tool that provides log analytics friendly Web interface for Logstash and ElasticSearch to help you summarize, analyze, and search for important data logs.

2. Components

Logstash:logstash server side used to collect logs;

Elasticsearch: Store all kinds of logs;

Kibana:web interfaces are used as search and visualization logs;

Logstash Forwarder:logstash client is used to send logs to Logstash server via Lumberjack Network protocol;

3. Work Flow

Deploy Logstash on all services that need to collect logs, as logstash_agent (Logstash shipper) to monitor and filter the collection logs, send filtered content to Redis, and Logstash_server Collect the logs together to the full-text search service Elasticsearch, you can use Elasticsearch to customize the search by Kibana to combine custom search for page presentation.

4. Service distribution

Host A 192.168.0.100 Elasticsearch + Logstash-server + Kinaba + Redis
Host B 192.168.0.101 Logstash-agent

Ii. start of Deployment Services

On Host B above 192.168.0.101

Deploying the Java Environment

#下载软件包, unzip, set environment variables

wget http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.tar.gz
tar -xf jdk-8u111-linux-x64.tar.gz -C /usr/local
mv /usr/local/jdk-8u111-linux-x64 /usr/local/java
echo "export PATH=\$PATH:/usr/local/java/bin" > /etc/profile.d/java.sh
. /etc/profile

2. Deploying Logstash-agent

wget https://download.elastic.co/logstash/logstash/logstash-2.2.0.tar.gz
tar -xf logstash-2.2.0.tar.gz -C /usr/local
echo "export PATH=\$PATH:/usr/local/logstash-2.2.0/bin" > /etc/profile.d/logstash.sh
. /etc/profile

3, Logstash Common parameters

-e: specifies the configuration information of logstash, which can be used for quick testing;
  -f: specify the configuration file of logstash; can be used in production environment;

4. Start Logstash

4.1 Specify the configuration information of the Logstash with the-e parameter for quick testing and direct output to the screen.

#logstash -e "input {stdin {}} output {stdout {}}"
             my name is zhengyansheng. // Enter after entering manually,
             After 10 seconds, the result will be returned. Logstash startup completed 2015-10-08T13: 55: 50.660Z 0.0.0.0
             my name is zhengyansheng. This output is directly returned intact ...

4.2 Specify the configuration information of the Logstash with the-e parameter for quick testing and output to the screen in JSON format.

5.1 Logstash output information is stored in the Redis database

Save the output information of the Logstash to the Redis database, as follows

if (192.168.0.100) has a Redis database, the next step is to install the Redis database.

# cat logstash_agent.conf
input {stdin {}}
output {
     stdout {codec => rubydebug}
     redis {
         host => ‘192.168.0.100’
          port => ‘6379’
         password => ‘12345678’
 
         data_type => ‘list’
         key => ‘logstash: redis’
     }
}
 
If Failed to send event to Redis is displayed, it means that the connection to Redis fails or is not installed, please check ...

6. View Logstash Process

# logstash agent -f logstash_agent.conf --verbose
#  ps -ef|grep logstash

192.168.0.100 on Host A

Deploying the Java Environment (IBID.)

Deploying Redis

wget http://download.redis.io/releases/redis-2.8.19.tar.gz
yum install tcl -y
tar zxf redis-2.8.19.tar.gz
cd redis-2.8.19
make MALLOC = libc
make test // This step will take longer ...
make install
 
cd utils /
./install_server.sh // After the script is executed, all options are subject to the default parameters.
Welcome to the redis service installer
This script will help you easily set up a running redis server
 
Please select the redis port for this instance: [6379]
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf]
Selected default-/etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log]
Selected default-/var/log/redis_6379.log
Please select the data directory for this instance [/ var / lib / redis / 6379]
Selected default-/ var / lib / redis / 6379
Please select the redis executable path [/ usr / local / bin / redis-server]
Selected config:
Port: 6379
Config file: /etc/redis/6379.conf
Log file: /var/log/redis_6379.log
Data dir: / var / lib / redis / 6379
Executable: / usr / local / bin / redis-server
Cli Executable: / usr / local / bin / redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service ...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server ...
Installation successful!

View the monitoring ports for Redis

# netstat -tnlp |grep redis
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      3843/redis-server * 
tcp        0      0 127.0.0.1:21365             0.0.0.0:*                   LISTEN      2290/src/redis-serv 
tcp        0      0 :::6379                     :::*                        LISTEN      3843/redi

Test if Redis works

# cd redis-2.8.19 / src /
# ./redis-cli -h 192.168.0. 100 -p 6379 // Connect redis
192.168.0.100:6379> ping
PONG
192.168.0.100:6379> set name zhengyansheng
OK
192.168.0.100:6379> get name
"zhengyansheng"
192.168.0.100:6379> quit
   
Start redis
/ usr / local / redis / bin / redis-server /usr/local/redis/conf/redis.conf

Boot Logstash based on ingress Redis (boot logstash-agent on Host B)

# cat logstash_agent.conf
input { stdin { } }
output {
    stdout { codec => rubydebug }
    redis {
        host => ‘192.168.0.100‘
        data_type => ‘list‘
        key => ‘logstash:redis‘
    }
}
# logstash agent -f logstash_agent.conf --verbose
Pipeline started {:level=>:info}
Logstash startup completed
dajihao linux
{
       "message" => "dajihao linux",
      "@version" => "1",
    "@timestamp" => "2015-10-08T14:42:07.550Z",
          "host" => "0.0.0.0"
}

Install Elasticsearch on Host a

1, installation Elasticsearch

# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-2.2.0.tar.gz
# tar zxf elasticsearch-2.2.0.tar.gz -C /usr/local/

2. Modify the Elasticsearch configuration file elasticsearch.yml and make the following modifications.

# vim /usr/local/elasticsearch-2.2.0/config/elasticsearch.yml
discovery.zen.ping.multicast.enabled: false #Turn off the broadcast. If the LAN machine has port 9300, the service will not start.
network.host: 192.168.0.100 #Specifying the host address is actually optional, but it is best to specify that it will report an http connection error when it is integrated with kibana (visually it seems to be listening ::: 9200 instead of 0.0.0.0 : 9200)
http.port: 9200

3. Start Elasticsearch Service

Nohup/usr/local/elasticsearch-2.2.0/bin/elasticsearch >/usr/local/elasticsearch-2.2.0/nohub &

If this method fails to start, create a normal user es boot

groupadd elk
useradd es -g elk
chown -R es.elk /usr/local/elasticsearch-2.2.0
su - es
nohup  /usr/local/elasticsearch-2.2.0/bin/elasticsearch >/usr/local/elasticsearch-2.2.0/nohub &

4. View Elasticsearch's Listening port

# netstat -tnlp |grep java
tcp        0      0 :::9200                     :::*                        LISTEN      7407/java           
tcp        0      0 :::9300                     :::*                        LISTEN      7407/java

 

	
  


 

Install Logstash-server on Host A (IBID.) Note that the configuration file is different

cat logstash_server.conf input {redis {host => '192.168.0.100' port => '6379' password => '12345678' data_type => 'list' key => 'logstash: redis' type => "redis-input" }} output {elasticsearch {hosts => "192.168.0.100" index => "logstash-% {+ YYYY.MM.dd}"}}

Start logstash-server (get data from redis and transfer to es)

logstash agent -f logstash_server.conf --verbose

Installing the Elasticsearch Plugin

#Elasticsearch-kopf plug-in can query the data in Elasticsearch, install Elasticsearch-kopf, just execute the following command in the directory where you install Elasticsearch:

# cd /usr/local/elasticsearch-2.2.0/bin/
# ./plugin install lmenezes / elasticsearch-kopf
-> Installing lmenezes / elasticsearch-kopf ...
Trying https: //github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading ............................. ............................
Installed lmenezes / elasticsearch-kopf into /usr/local/elasticsearch-2.2.0/plugins/kopf
 
After executing the plug-in installation, it will prompt failure, most likely it is the network, etc ...
-> Installing lmenezes / elasticsearch-kopf ...
Trying https: //github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Failed to install lmenezes / elasticsearch-kopf, reason: failed to download out of all possible locations ..., use --verbose to get detailed information
 
The solution is to download the software manually, without installing commands via the plugin ...
cd /usr/local/elasticsearch-2.2.0/plugins
wget https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip
unzip master.zip
mv elasticsearch-kopf-master kopf
The above operation is exactly equivalent to the installation command of the plugin

Browser Access Kopf page access to Elasticsearch saved data

Install Kibana on Host a

1, installation Kinaba

# wget https://download.elastic.co/kibana/kibana/kibana-4.4.0-linux-x64.tar.gz# Tar zxf Kibana-4.4.0-linux-x64.tar.gz-c/usr/local

2. Modify the Kinaba configuration file kinaba.yml

# Vim/usr/local/kibana-4.4.0-linux-x64/config/kibana.ymlelasticsearch_url: "http://192.168.0.100:9200"

3. Start Kinaba

Nohup/usr/local/kibana-4.4.0-linux-x64/bin/kibana >/usr/local/kibana-4.4.0-linux-x64/nohub.out &

Output the following information to indicate Kinaba success.

{"Name": "Kibana", "hostname": "Localhost.localdomain", "pid": 1943, "level": +, "msg": "No existing Kibana index found", " Time ":" 2015-10-08t00:39:21.617z "," V ": 0}

{"Name": "Kibana", "hostname": "Localhost.localdomain", "pid": 1943, "level": +, "msg": "Listening on 0.0.0.0:5601", " Time ":" 2015-10-08t00:39:21.637z "," V ": 0}

Kinaba default listener on local port 5601

4. Browser Access Kinaba

4.1 Use the default logstash-* index name, and it is time-based, click "Create".

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/74/25/wKioL1YWFTbx56DuAAOecFQkxcA301.jpg "style=" float: none; "Title=" 01 (Home). png "alt=" wkiol1ywftbx56duaaoecfqkxca301.jpg "width=" 650 "/>

4.2 See the following interface to illustrate the completion of index creation.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/74/27/wKiom1YWFR-hbgxOAAQZ85RcxMg067.jpg "style=" float: none; "title=" (create). png "alt=" wkiom1ywfr-hbgxoaaqz85rcxmg067.jpg "width=" 650 "/>

4.3 Click "Discover" to search and browse the data in Elasticsearch.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/74/25/wKioL1YWFTaRWGZmAAQ81v02YE8823.jpg "style=" float: none; "Title=" (Discover). png "alt=" wkiol1ywftarwgzmaaq81v02ye8823.jpg "width=" 650 "/>

>>> End <<<

1. Elk Default port number elasticsearch:9200 9300logstash:9301kinaba:5601

2. Summary of Errors

(1) Java version is too low

[2015-10-07 18:39:18.071] WARN--Concurrent: [DEPRECATED] Java 7 is DEPRECATED and use Java 8.

(2) Kibana tip elasticsearch version too low ...

This version of Kibana requires Elasticsearch 2.0.0 or higher on all nodes. I found the following incompatible nodes in your cluster:

Elasticsearch v1.7.2 @ inet[/192.168.1.104:9200] (127.0.0.1)

Workaround:

Reference: http://467754239.blog.51cto.com/4878013/1700828

Elasticsearch+logstash+kinaba+redis Log Analysis System