"Reprint" using Logstash+elasticsearch+kibana to quickly build a log platform

Original link: http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html

Log analysis and monitoring in the system development occupies a very important position, the more complex the system, log analysis and monitoring is the more important, the common needs are:

• Query log details by keyword
• Monitor the health of the system
• Statistical analysis, such as the number of calls to the interface, execution time, success rate, etc.
• Auto-trigger message notification for exception data
• Log-based data mining

Some of the issues that many teams may encounter with logging are:

• Developers cannot log on to online servers to view detailed logs, which are time-consuming and laborious to transport
• Log data scattered across multiple systems, difficult to find
• Large log data size, slow query speed
• One call involves multiple systems, which makes it difficult to locate data quickly in the logs of these systems
• Insufficient real-time data

Some of the most common, heavy-source trace systems are

• Facebook scribe
• Cloudera Flume
• Twitter Zipkin
• Storm

These projects are powerful, but are too complex for many teams to configure and deploy, and recommend lightweight download-ready scenarios, such as the Logstash+elasticsearch+kibana (LEK) combination, before the system is large enough to a certain extent.

For the log, the most common need is to collect, query, display, is corresponding to Logstash, Elasticsearch, Kibana function.

Logstash

Logstash Home

Logstash deployment is simple, download a jar can be used, the processing logic of the log is very simple, is a pipeline process

inputs >> codecs >> filters >> outputs

The corresponding plugin has

From the above can be seen Logstash support common log types, and other monitoring system integration is also very convenient, you can export data to Zabbix, Nagios, email and so on.

Redis is recommended as the input buffer queue.

You can also output the data to graphite to achieve a visual display of statistical data.

Metrics Demo
Statsd
Graphite

Reference documents

• Cookbook
• Doc
• Demo

Elasticsearch

Elasticsearch Home

Elasticsearch is an open source search engine based on Lucene, which has developed rapidly in recent years, and its main features are

• Real Time
• Distributed
• High Availability
• Document oriented
• Schema free
• RESTful API

After the detailed introduction of Elasticsearch, some common resources are as follows

Chinese

SMARTCN, es default Chinese participle
Https://github.com/elasticsearch/elasticsearch-analysis-smartcn

Mmseg
Https://github.com/medcl/elasticsearch-analysis-mmseg

Ik
Https://github.com/medcl/elasticsearch-analysis-ik

Pinyin, pinyin word breaker, can be used to enter phonetic hints in Chinese
Https://github.com/medcl/elasticsearch-analysis-pinyin

Stconvert, Chinese Simplified traditional interchange
Https://github.com/medcl/elasticsearch-analysis-stconvert

Common plugins

Elasticsearch-servicewrapper, a package of Elasticsearch made with the Java Service Wrapper
Https://github.com/elasticsearch/elasticsearch-servicewrapper

Monitoring tools for Elastic Hq,elasticsearch
http://www.elastichq.org

ELASTICSEARCH-RTF, integrated with relevant plugins for Chinese (RTF = ready to Fly)
Https://github.com/medcl/elasticsearch-rtf
Author homepage

Kibana

Kibana Home

Kibana is a powerful Elasticsearch data display client, Logstash has built-in Kibana, you can also deploy Kibana alone, the latest version of Kibana3 is a pure HTML+JS client, can be easily deployed to Apache, Nginx and other HTTP servers.

Address of Kibana3: Https://github.com/elasticsearch/kibana
Address of Kibana2: Https://github.com/rashidkpc/Kibana
KIBANA3 Demo Address: http://demo.kibana.org

From the demo you can first look at some of the basic functions of Kibana

Chart

Data table, you can customize which columns display and display the order

You can see the actual execution of the query statement

Add a new row

New panel, you can see the supported panel types

Add a pie chart

Display statistical results in a map

According to HTTP response code to count

Rich query syntax

Installation deployment

Below is a list of steps to build a simple LEK experience environment

Installing JDK 1.7

Oracle Java home Page

Omit installation process, recommended 1.7+ version

java -version

Set the environment variables for Java, such as

sudo vim ~/.bashrc>>export JAVA_HOME=/usr/lib/jvm/java-7-oracleexport JRE_HOME=${JAVA_HOME}/jre  export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  export PATH=${JAVA_HOME}/bin:$PATH  >>source ~/.bashrc

Installing Redis

Redis Home

cd ~/srcwget http://download.redis.io/releases/redis-2.6.16.tar.gztar -zxf redis-2.6.16.tar.gzcd redis-2.6.16makesudo make install

Simplified configuration through the Utils/install_server script in the Redis source code

Install_server.sh after asking you a few questions will be installed Redis as a boot-up service, you can start/stop the service by the following command line

Start the Redis client to verify the installation

redis-cli> keys *

Installing Elasticsearch

Elasticsearch Home

cd /searchsudo mkdir elasticsearchcd elasticsearchsudo wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.zipsudo unzip elasticsearch-0.90.5.zip

Elasticsearch decompression can be used very convenient, next we look at the effect, first start the ES service, switch to the Elasticsearch directory, run under the bin Elasticsearch

cd /search/elasticsearch/elasticsearch-0.90.5 bin/elasticsearch -f

Access the default 9200 port

curl -X GET http://localhost:9200

Installing Logstash

Logstash Home

cd /searchsudo mkdir logstashcd logstashsudo wget http://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar

Logstash download can be used, command line parameters can refer to Logstash flags, mainly

agent   #运行Agent模式-f CONFIGFILE #指定配置文件web     #自动Web服务-p PORT #指定端口，默认9292

Installing Kibana

The latest version of Logstash has built-in Kibana, and you can deploy Kibana separately. Kibana3 is a purely javascript+html client, so it can be deployed to any HTTP server.

cd /searchsudo mkdir kibanasudo wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zipsudo unzip kibana-latest.zipsudo cp -r  kibana-latest /var/www/html

You can modify the Config.js to configure the address and index of the Elasticsearch.

Try http://127.0.0.1/html/kibana-latest/index.html with browser access

Integration

Integrate the above system.

First, start the Redis and the Elasticsearch.

Create a new configuration file for Logstash

cd /search/logstashsudo vi redis.conf

The configuration file contents are as follows

input {  redis {    host => "127.0.0.1"    port => "6379"     key => "logstash:demo"    data_type => "list"    codec  => "json"    type => "logstash-redis-demo"    tags => ["logstashdemo"]  }}output {  elasticsearch {    host => "127.0.0.1"  }}

Start the Logstash agent with this configuration file

java -jar /search/logstash/logstash-1.2.1-flatjar.jar agent -f /search/logstash/redis.conf &

Launch Logstash built-in web

java -jar /search/logstash/logstash-1.2.1-flatjar.jar web &

View the web, there should be no data

http://127.0.0.1:9292

Add a piece of data to Redis

RPUSH logstash:demo "{\"time\": \"2013-01-01T01:23:55\", \"message\": \"logstash demo message\"}"

Look at the index status in Elasticsearch

curl 127.0.0.1:9200/_search?pretty=true curl -s http://127.0.0.1:9200/_status?pretty=true | grep logstash

And then look through the Logstash Web query.

http://127.0.0.1:9292

View through a separate Kibana interface

http://127.0.0.1/html/kibana-latest/index.html#/dashboard/file/logstash.json

Data cleanup

Logstash by default, the ES index is created by the day, so the benefit is that deleting the historical data directly deletes the entire index, which is convenient and fast.

Elasticsearch can also set the TTL for each document (time to live), which is equivalent to setting the document's expiration date, but consumes more IO operations than deleting the entire index.

Index

Elasticsearch By default, the field is split according to the delimiter, there are some fields do not participle, such as the URL, you can set the Not_analyzed property for such fields.

Set the Multi-field-type property to map fields to other types. Multi-field-type.

A large number of logs are imported in bulk mode.

For log queries, filter does not perform scoring in a faster filter than query and can be automatically cached. Query-dsl.

Elasticsearch default an index operation returns after all shards have finished indexing the document, and you can set replication to asynchronous to expedite the import of bulk logs.

Elasticsearch optimization

Optimizing the JVM
Optimize the system to open the maximum number of file descriptors
Appropriately increase the interval for index refreshes

Best practices

• First of all, your program is going to write logs
• Log logs to help you analyze the problem, logging only "parameter errors" such as the log is not helpful to solve the problem
• Don't rely on exceptions, exceptions only deal with places you don't think about.
• To record key parameters such as time of occurrence, execution time, log source, input parameter, output parameter, error code, exception stack information, etc.
• Keep track of key parameters such as SessionID, Transitionid, UserID, etc. that can help you quickly locate and concatenate the logs of each system
• Recommended Plain Text +json format
• Using queues

Other Log Accessibility tools

• Rsyslog
• Syslog-ng
• Graylog
• Fluentd
• Nxlog

"Reprint" using Logstash+elasticsearch+kibana to quickly build a log platform