Elasticsearch + Logstash + Kibana

Elasticsearch + Logstash + Kibana

ELK stack is a combination of three open source software: Elasticsearch, Logstash, and Kibana. Currently, all are under the Elastic. co company name. ELK is a common open-source log monitoring and analysis system, including a distributed index and search service Elasticsearch,

Logstash, a tool for managing logs and events, and Kibana

Logstash_1.5.3 collects, processes, and stores logs.

Elasticsearch-1.7.2 is responsible for Log retrieval and analysis

Kibana-4.1.2-linux-x64.tar.gz is responsible for log Visualization

Redis-2.4.14 DB and log transmission channels for processing

Use a graph to represent the relationship between them.

This article uses two servers as an example to deploy

Server A: 192.168.0.1 java elasticsearch redis kibana logstash (agent indexer)

Server B: 192.168.0.2 java logstash (agent)

First, install the software for server.

1. Install basic software

Yum-y install curl wget lrzsz axel

Ii. install and configure the redis Service

1. Install tcl8.6.1

A) tar-xf tcl8.6.1-src.tar.gz -- strip-components = 1
B) cd tcl8.6.1/unix
C)./configure -- prefix =/usr/local
D) make
E) make test
F) make install
G) make install-private-headers
H) ln-v-sf tclsh8.6/usr/bin/tclsh
I) chmod-v 755/usr/lib/libtcl8.6.so (optional, and it does not matter if the file cannot be found)

2. Install redis-3.0.2

Wget http://download.redis.io/releases/redis-3.0.2.tar.gz
Tar xzf redis-3.0.2.tar.gz/usr/local/redis
Cd redis-3.0.2
Make MALLOC = libc
Make test
Make install

2. Configure redis

A) mkdir/etc/redis
B) mkdir/var/redis
C) cp utils/redis_init_script/etc/init. d/redis
D) vim/etc/init. d/redis
Add a header:
# Chkconfig: 345 60 60
#! /Bin/bash
E. mkdir/var/redis/6379
F) cp redis. conf/etc/redis/6379. conf
G) vim/etc/redis/6379. conf
# Set daemonize to yes
# Set pidfile to/var/run/redis_62.16.pid
# Set loglevel
# Set logfile to/var/log/redis_62.16.log
# Set dir to/var/redis/6379
H) sysctl vm. overcommit_memory = 1
I) chkconfig -- add redis
J) chkconfig redis on

3. Restart the service

Service redis start/stop

4. View processes and ports
1) view Processes

Ps-ef | grep redis
Root 31927 25099 0 00:00:00 pts/0 vi/etc/init. d/redis

2) view the port

Netstat-tupnl | grep redis
Tcp 0 0 0.0.0.0: 6379 0.0.0.0: * LISTEN 31966/redis-server
Tcp 0 0: 6379: * LISTEN 31966/redis-server

3. Install the java environment

1. Installation Package

Yum-y list java *
Yum-y install openjdk-7-jdk

2. view the version

Java-version
Java version "1.7.0 _ 91"
OpenJDK Runtime Environment (rhel-2.6.2.2.el6_7-x86_64 u91-b00)
OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)

 

4. Install elasticsearch

1. Download elasticsearch

Wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.noarch.rpm

2. Install elasticsearch

Rpm-ivh elasticsearch-1.7.2.noarch.rpm

3. Configuration
1) backup configuration

Cp/etc/elasticsearch. yml/etc/elasticsearch. yml. bak

2) modify the configuration

Echo "network. bind_host: 192.168.0.1">/etc/elasticsearch. yml

4. Start the elasticsearch Service

/Etc/init. d/elasticsearch start
/Etc/init. d/elasticsearch stop

5. View processes and ports

1) view Processes

Ps-ef | grep java

2) view the port

Netstat-tupnl | grep java

6. Test

Curl-x get http: // 192.168.54.147: 9200
{
"Status": 200,
"Name": "Miguel O 'Hara ",
"Cluster_name": "elasticsearch ",
"Version ":{
"Number": "1.7.2 ",
"Build_hash": "e43676b1385b8125d647f593f7202acbd816e8ec ",
"Build_timestamp": "2015-09-14T09: 49: 53Z ",
"Build_snapshot": false,
"Inclue_version": "4.10.4"
},
"Tagline": "You Know, for Search"
}

7. Add to start

Update-rc.d elasticsearch defaults
Update-rc.d: using dependency based boot sequencing

5. Install logstash

1. Download logstash

Wget https://download.elastic.co/logstash/logstash/packages/CentOS/logstash-1.5.4-1.noarch.rpm

2. Install logstash

Rpm-ivh logstash-1.5.4-1.noarch.rpm

3. configuration (this configuration file is not available by default)

Vim/etc/logstash/conf. d/logstash_indexer.conf
Input {
Redis {
Host => "192.168.0.1"
Data_type => "list"
Key => "logstash: redis"
Type => "redis-input"
Port => "6379"
}
}
Output {
Elasticsearch {
Host => "192.168.0.1"
}
}

4. Start the service

/Etc/init. d/logstash start

5. Use jps-mlv or ps-ef to view the process

Ps-ef | grep logst

6. Set startup

Update-rc.d: logstash ults
Update-rc.d: using dependency based boot sequencing

6. Install kibana (front-end web)

1. Download

Wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

2. decompress the package to the specified directory.

Tar zxvf kibana-4.1.2-linux-x64.tar.gz-C/opt

3. Create a log directory

Mkdir-p/opt/kibanalog

4. Configuration
1) backup configuration

Cp/opt/kibana-4.1.2-linux-x64/config/kibana. yml/opt/kibana-4.1.2-linux-x64/config/kibana. yml. bak

2) modify the configuration

Sed-I's! ^ Elasticsearch_url :.*! Elasticsearch_url: "http: // 192.168.0.1: 9200 "! G'/opt/kibana-4.1.2-linux-x64/config/kibana. yml
Sed-I's! ^ Host :.*! Host: "192.168.0.1 "! G'/opt/kibana-4.1.2-linux-x64/config/kibana. yml

5. Start the service

Cd/opt/kibanalog & nohup/opt/kibana-4.1.2-linux-x64/bin/kibana &

6. View processes and ports
1) view Processes

Ps aux | grep kibana

2) view the port

Netstat-tupnl | grep 5601

7. Access on Windows

Http: // 192.168.0.1: 5601

8. Set startup

Echo "cd/opt/kibanalog & nohup/opt/kibana-4.1.2-linux-x64/bin/kibana &">/etc/rc. local

All software related to server A has been installed;

Now you can install the log collection program (server B), which can contain multiple

Install and configure logstash (agent) in clientB)

1. Install the java environment

Yum-y list java *
Yum-y install openjdk-7-jdk

1. Download logstash

Wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm

2. Install logstash

Rpm-ivh logstash-1.5.4-1.noarch.rpm

3. configuration (this configuration file is not available by default)
1) Configure logstash_agent

Vim/etc/logstash/conf. d/logstash_agent.conf

Input {
File {
Path => "/tmp/*. log"
Start_position => beginning
}
}

Output {
Redis {
Host => "192.168.0.1"
Data_type => "list"
Key => "logstash: redis"
}
}

5. Start the service
 
/Etc/init. d/logstash start
Logstash started.

6. Use jps-mlv or ps-ef to view the process

Ps-ef | grep logst

7. Set startup

Update-rc.d: logstash ults
Update-rc.d: using dependency based boot sequencing

So far, server B has been installed and configured. You can configure N service B according to the needs of the log collection end.

How to view logs:

1. View redis logs

Cat/var/log/redis/redis-server.log.

2. View elasticsearch logs

Cat/var/log/elasticsearch. log
Tail-300f/var/log/elasticsearch. log

3. View logstash logs

Cat/var/log/logstash. errtail-30f/var/log/logstash. err

4. View kibana logs

Cat/opt/kibanalog/nohup. out
Tail-30f/opt/kibanalog/nohup. out

Error Handling

1) unable to fetch mapping, do you have indices matching the pattem?

Kibana reports this error because it does not come from logstash. Generally, it checks data transmission.