Original link: http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html
Log analysis and monitoring in the system development occupies a very important position, the more complex the system, log analysis and monitoring is the more important, the common needs are:
- Query log details by keyword
- Monitor the health of the system
- Statistical analysis, such as the number of calls to the interface, execution time, success rate, etc.
- Auto-trigger message notification for exception data
- Log-based data mining
Some of the issues that many teams may encounter with logging are:
- Developers cannot log on to online servers to view detailed logs, which are time-consuming and laborious to transport
- Log data scattered across multiple systems, difficult to find
- Large log data size, slow query speed
- One call involves multiple systems, which makes it difficult to locate data quickly in the logs of these systems
- Insufficient real-time data
Some of the most common, heavy-source trace systems are
- Facebook scribe
- Cloudera Flume
- Twitter Zipkin
- Storm
These projects are powerful, but are too complex for many teams to configure and deploy, and recommend lightweight download-ready scenarios, such as the Logstash+elasticsearch+kibana (LEK) combination, before the system is large enough to a certain extent.
For the log, the most common need is to collect, query, display, is corresponding to Logstash, Elasticsearch, Kibana function.
Logstash
Logstash Home
Logstash deployment is simple, download a jar can be used, the processing logic of the log is very simple, is a pipeline process
inputs >> codecs >> filters >> outputs
The corresponding plugin has
From the above can be seen Logstash support common log types, and other monitoring system integration is also very convenient, you can export data to Zabbix, Nagios, email and so on.
Redis is recommended as the input buffer queue.
You can also output the data to graphite to achieve a visual display of statistical data.
Metrics Demo
Statsd
Graphite
Reference documents
Elasticsearch
Elasticsearch Home
Elasticsearch is an open source search engine based on Lucene, which has developed rapidly in recent years, and its main features are
- Real Time
- Distributed
- High Availability
- Document oriented
- Schema free
- RESTful API
After the detailed introduction of Elasticsearch, some common resources are as follows
Chinese
SMARTCN, es default Chinese participle
Https://github.com/elasticsearch/elasticsearch-analysis-smartcn
Mmseg
Https://github.com/medcl/elasticsearch-analysis-mmseg
Ik
Https://github.com/medcl/elasticsearch-analysis-ik
Pinyin, pinyin word breaker, can be used to enter phonetic hints in Chinese
Https://github.com/medcl/elasticsearch-analysis-pinyin
Stconvert, Chinese Simplified traditional interchange
Https://github.com/medcl/elasticsearch-analysis-stconvert
Common plugins
Elasticsearch-servicewrapper, a package of Elasticsearch made with the Java Service Wrapper
Https://github.com/elasticsearch/elasticsearch-servicewrapper
Monitoring tools for Elastic Hq,elasticsearch
http://www.elastichq.org
ELASTICSEARCH-RTF, integrated with relevant plugins for Chinese (RTF = ready to Fly)
Https://github.com/medcl/elasticsearch-rtf
Author homepage
Kibana
Kibana Home
Kibana is a powerful Elasticsearch data display client, Logstash has built-in Kibana, you can also deploy Kibana alone, the latest version of Kibana3 is a pure HTML+JS client, can be easily deployed to Apache, Nginx and other HTTP servers.
Address of Kibana3: Https://github.com/elasticsearch/kibana
Address of Kibana2: Https://github.com/rashidkpc/Kibana
KIBANA3 Demo Address: http://demo.kibana.org
From the demo you can first look at some of the basic functions of Kibana
Chart
Data table, you can customize which columns display and display the order
You can see the actual execution of the query statement
Add a new row
New panel, you can see the supported panel types
Add a pie chart
Display statistical results in a map
According to HTTP response code to count
Rich query syntax
Installation deployment
Below is a list of steps to build a simple LEK experience environment
Installing JDK 1.7
Oracle Java home Page
Omit installation process, recommended 1.7+ version
java -version
Set the environment variables for Java, such as
sudo vim ~/.bashrc>>export JAVA_HOME=/usr/lib/jvm/java-7-oracleexport JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH >>source ~/.bashrc
Installing Redis
Redis Home
cd ~/srcwget http://download.redis.io/releases/redis-2.6.16.tar.gztar -zxf redis-2.6.16.tar.gzcd redis-2.6.16makesudo make install
Simplified configuration through the Utils/install_server script in the Redis source code
Install_server.sh after asking you a few questions will be installed Redis as a boot-up service, you can start/stop the service by the following command line
Start the Redis client to verify the installation
redis-cli> keys *
Installing Elasticsearch
Elasticsearch Home
cd /searchsudo mkdir elasticsearchcd elasticsearchsudo wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.zipsudo unzip elasticsearch-0.90.5.zip
Elasticsearch decompression can be used very convenient, next we look at the effect, first start the ES service, switch to the Elasticsearch directory, run under the bin Elasticsearch
cd /search/elasticsearch/elasticsearch-0.90.5 bin/elasticsearch -f
Access the default 9200 port
curl -X GET http://localhost:9200
Installing Logstash
Logstash Home
cd /searchsudo mkdir logstashcd logstashsudo wget http://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar
Logstash download can be used, command line parameters can refer to Logstash flags, mainly
agent #运行Agent模式-f CONFIGFILE #指定配置文件web #自动Web服务-p PORT #指定端口,默认9292
Installing Kibana
The latest version of Logstash has built-in Kibana, and you can deploy Kibana separately. Kibana3 is a purely javascript+html client, so it can be deployed to any HTTP server.
cd /searchsudo mkdir kibanasudo wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zipsudo unzip kibana-latest.zipsudo cp -r kibana-latest /var/www/html
You can modify the Config.js to configure the address and index of the Elasticsearch.
Try http://127.0.0.1/html/kibana-latest/index.html with browser access
Integration
Integrate the above system.
First, start the Redis and the Elasticsearch.
Create a new configuration file for Logstash
cd /search/logstashsudo vi redis.conf
The configuration file contents are as follows
input { redis { host => "127.0.0.1" port => "6379" key => "logstash:demo" data_type => "list" codec => "json" type => "logstash-redis-demo" tags => ["logstashdemo"] }}output { elasticsearch { host => "127.0.0.1" }}
Start the Logstash agent with this configuration file
java -jar /search/logstash/logstash-1.2.1-flatjar.jar agent -f /search/logstash/redis.conf &
Launch Logstash built-in web
java -jar /search/logstash/logstash-1.2.1-flatjar.jar web &
View the web, there should be no data
http://127.0.0.1:9292
Add a piece of data to Redis
RPUSH logstash:demo "{\"time\": \"2013-01-01T01:23:55\", \"message\": \"logstash demo message\"}"
Look at the index status in Elasticsearch
curl 127.0.0.1:9200/_search?pretty=true curl -s http://127.0.0.1:9200/_status?pretty=true | grep logstash
And then look through the Logstash Web query.
http://127.0.0.1:9292
View through a separate Kibana interface
http://127.0.0.1/html/kibana-latest/index.html#/dashboard/file/logstash.json
Data cleanup
Logstash by default, the ES index is created by the day, so the benefit is that deleting the historical data directly deletes the entire index, which is convenient and fast.
Elasticsearch can also set the TTL for each document (time to live), which is equivalent to setting the document's expiration date, but consumes more IO operations than deleting the entire index.
Index
Elasticsearch By default, the field is split according to the delimiter, there are some fields do not participle, such as the URL, you can set the Not_analyzed property for such fields.
Set the Multi-field-type property to map fields to other types. Multi-field-type.
A large number of logs are imported in bulk mode.
For log queries, filter does not perform scoring in a faster filter than query and can be automatically cached. Query-dsl.
Elasticsearch default an index operation returns after all shards have finished indexing the document, and you can set replication to asynchronous to expedite the import of bulk logs.
Elasticsearch optimization
Optimizing the JVM
Optimize the system to open the maximum number of file descriptors
Appropriately increase the interval for index refreshes
Best practices
- First of all, your program is going to write logs
- Log logs to help you analyze the problem, logging only "parameter errors" such as the log is not helpful to solve the problem
- Don't rely on exceptions, exceptions only deal with places you don't think about.
- To record key parameters such as time of occurrence, execution time, log source, input parameter, output parameter, error code, exception stack information, etc.
- Keep track of key parameters such as SessionID, Transitionid, UserID, etc. that can help you quickly locate and concatenate the logs of each system
- Recommended Plain Text +json format
- Using queues
Other Log Accessibility tools
- Rsyslog
- Syslog-ng
- Graylog
- Fluentd
- Nxlog
"Reprint" using Logstash+elasticsearch+kibana to quickly build a log platform