ELK Log Analysis System

ELK Log Analysis System

ELK refers to the combination of Elasticsearch, Logstash, and Kibana three open source software.

Logstash responsible for the collection, processing and storage of logs

Elasticsearch responsible for log retrieval and analysis

Kibana responsible for the visualization of logs

First, the environment

1. CentOS Linux release 7.1.1503 (Core)

Server-172.16.32.31

2. Installing the Base software

Yum-y Install Curl wget lrzsz Axel

3. Installing Redis

wget https://github.com/antirez/redis/archive/2.8.23.tar.gz

Tar zxvf 2.8.23.tar.gz

CD redis-2.8.23

Make

Make install

CD Utils

./install_server.sh Initialize configuration information and modify the configuration path.

Modifying a Redis configuration file

VI redis.conf

The following configuration requires attention:

---------------------------------------------------------------------

# Modify the configuration ports inside and configure the configuration needs to be modified as follows:

Daemonize Yes #后台模式运行

Pidfile/opt/local/redis/redis_6379.pid #pid File

Port 6379 #运行端口

Timeout #请求超时时间, default 0

LogFile "/opt/local/codis_server/logs/codis_6379.log" #日志文件

Save 1 #打开保存快照的条件 (the first * indicates how long, and the third * indicates how many write operations are performed)

Save 300 10

Save 60 10000

Dbfilename 6379.rdb #数据快照保存的名字

Dir/opt/local/codis_server/data #数据快照的保存目录

Appendfilename "6379_appendonly.aof" #Redis更加高效的数据库备份及灾难恢复方式.

Appendfsync everysec # (always:always indicates that each write operation is synchronized. Everysec: Indicates that the write operation is cumulative, synchronized once per second)

-----------------------------------------------------------------------------------

Start Redis

Service Redis Start

Second, install the Java environment

1. wget http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm? authparam=1445478596_a41d759b5cc27a6510ed83c701ee5676

RPM-IVH jdk-8u65-linux-x64.rpm

Third, installation Elasticsearch

1. Download Elasticsearch

wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.3.noarch.rpm

RPM-IVH elasticsearch-1.7.3.noarch.rpm

2. Modify the configuration

Cp/etc/elasticsearch/elasticsearch.yml/etc/elasticsearch/elasticsearch.yml-bak

echo "Cluster.name:logssearch" >>/etc/elasticsearch/elasticsearch.yml #必须修改名字, otherwise it will automatically query the same network segment with the same name ELA

echo "network.bind_host:172.16.32.31" >>/etc/elasticsearch/elasticsearch.yml

VI elasticsearch.yml

In the end, add the following configuration: Modify ES spanning support with ES cache type soft

Http.json.enable:true

Http.cors.allow-origin: "/.*/"

Http.cors.enabled:true

Index.cache.field.type:soft

3. Start Elasticsearch Service

Service Elasticsearch Start

4. Add to boot

Chkconfig Elasticsearch on

5. Install head Plug-in

Execute the following command:

/usr/share/elasticsearch/bin/plugin-install Mobz/elasticsearch-head

6. Visit Http://172.16.32.31:9200/_plugin/head to see if the success

Iv. installation of Logstash

1. Download Logstash

wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm

2, Installation Logstash

RPM-IVH logstash-1.5.4-1.noarch.rpm

3, configuration logstash_indexer (default does not have this configuration file)

Add this profile to the server:

Vi/etc/logstash/conf.d/logstash_indexer.conf

--------------------------------------------------------------------------------------------------------------

Input {

Redis {

Host = "172.16.32.31"

data_type = "List"

Key = "Logstash:redis"

Type = "Redis-input"

Port = "6379"

}

}

Output {

Elasticsearch {

Embedded = False

protocol = "HTTP"

Host = "172.16.32.31"

}

}

--------------------------------------------------------------------------------------------------------------

4, the client configuration to increase this configuration file (default does not have this profile)

Vi/etc/logstash/conf.d/logstash_agent.conf

----------------------------------------------------------------------------------------------------------

Input {

File {

Type = "Nginx_access"

Path = ["/usr/share/nginx/logs/test.access.log"]

}

}

Output {

Redis {

Host = "172.16.32.31"

data_type = "List"

Key = "Logstash:redis"

}

}

------------------------------------------------------------------------------------------------------------

5. Start Logstash Service

Service Logstash Start

Chkconfig Logstash on

V. Installation of Kibana (front-end web)

1. Download Kibana

wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

Tar zxvf kibana-4.1.2-linux-x64.tar.gz

MV Kibana-4.1.2-linux-x64/opt/local/kibana

Mkdir/opt/local/kibana/logs

Cd/opt/local/kibana

2. Modify the configuration

Cp/opt/local/kibana/config/kibana.yml/opt/local/kibana/config/kibana.yml.bak

Sed-i ' S!^elasticsearch_url:. *!elasticsearch_url: "http://172.16.32.31:9200"!g '/opt/local/kibana/config/ Kibana.yml

Sed-i ' S!^host:. *!host: "172.16.32.31"!g '/opt/local/kibana/config/kibana.yml

3. Start the Kibana service

Cd/opt/local/kibana/logs && Nohup/opt/local/kibana/bin/kibana &

4. View the boot port

Netstat-tupnl|grep 5601

5. Visit http://172.16.32.31:5601

Appears Elasticsearch is still initializing the Kibana index ...  Trying again in 2.5 second. Error

Delete Index {"acknowledged": true} using Curl-xdelete Http://172.16.32.31:9200/.kibana

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/74/CB/wKioL1YprbGzJWtDAAMTnZHGS5A947.jpg "title=" QQ picture 20151023114610.png "alt=" Wkiol1yprbgzjwtdaamtnzhgs5a947.jpg "/>

ELK Log Analysis System