Build distributed log system with open source architecture Elk

This article describes how to use the Mature classic architecture elk (i.e. elastic search,logstash and Kibana) to build distributed log monitoring system, many companies use this architecture to build distributed log system, including Sina Weibo, Freewheel, Chang Jie and so on.

Background
Log, for each system, is very important, and easily overlooked part. The log records key information about the execution of the program, error and warning information, and so on. We can do a lot of things according to the log, do data analysis, system monitoring, troubleshooting problems and so on. However, no one medium-sized system can be a single server, log files scattered on dozens of or even thousands of servers, which is a log-based data analysis, troubleshooting is a great difficulty.

Centralized Logging

ELK
The concept of centralized log storage (centralized Logging) emerges as a result of a unified collection of logs on each server into a single log platform. Elk is dedicated to doing this thing. Elk is the abbreviation for Elasticsearch (hereinafter abbreviated as ES) +logstash+kibana. ES is responsible for data storage and indexing, Logstash is responsible for data adjustment, filtering, transmission, Kibana responsible for data display.

Ealsticsearch
First of all, ES, which is a good lucene-based storage engine, can also be seen as a petabyte-level NoSQL DB. Support CRUD, multi-field query, Aggreagation, full-text search, can be said to cover a variety of query requirements. It comes with features like replication and sharding that can be duplicated and fragmented with a simple configuration. Also, ES has made some optimizations in the NRT (near real time), which makes the application very good in real terms.

Logstash
Logstash is a very flexible data processing and transmission tool. We can no longer need to write a bunch of wget, SCP, socket code, to transfer our data. Logstash Unified Implementation of each link of data transmission, we only need to focus on the logic of data processing. We no longer need to learn all kinds of APIs to find out how to get data (such as tail-f) and how to write data (such as writing es or MongoDB). The Logstash pre-provisioned a bunch of input & output plugin to allow real-time incremental data outflow and inflow through configuration. For example, both the file input and the ES output we used are already available.

Kibana
Kibana is a powerful data presentation tool. In most cases, you don't need to develop any code to get a dashboard.
What needs to be done is:
(1) Put the data in ES
(2) The report template is configured on the Kibana page, or a set of report templates is written according to the rules.

Architecture

Freewheel-ui-log Platform Architecture

Is the architecture diagram of the log system. Our data source is a four-module log file that imports data into MQ in real time via Logstash, and then imports the data into ES in real time from MQ.
The purpose of using MQ is to:
(1) Reduce peak pressure
(2) The real-time data processing, to avoid this processing on the application server, resulting in the burden on the application.
At the same time, the corresponding alert is triggered by making a rule match based on the data in MQ. If the error information is found, it will be stored in the Jira for problem recording, dispatch and tracking. There are some strategies to control how error is stored in Jira, including error weighing, bug fixes and alert again, and so on. The hourly, day-level alert interacts directly with ES through a background-scheduled task framework.
ES combines Hadoop Plugin with HADOOP2 to make map reduce calculations.

Freewheel-ui-dashboard of Throughput & Response time
(Hidden y-axis data and query criteria)
Finally, the Kibana, by reading the ES data, to provide report presentation. We are now kibana3&4 mixed, mainly using KIBANA3, and have developed some plugin on the basis of KIBANA3. For example, menu, optional time zone datepicker, and so on. Kibana4 just released, not very well, so we haven't completely switched to 4. At present, a report is developed using KIBANA4, using the new features provided by 4, such as percentile aggregation, multilevel aggregation, etc.

Log Platform Application

Freewheel-application on Log Platform
In front of us also mentioned some, with this log platform data, we can do a lot of things.
(1) Log query, troubleshooting, on-line inspection
(2) Server monitoring, application monitoring, error alarm, bug management
(3) Performance analysis, user behavior analysis

Summarize
Log platform after the launch, solve a lot of original problems, it is more and more important, the future of our plans are as follows:
(1) More flexible reporting. Kibana solves the underlying reporting requirements, but does not implement a logically complex report. Consider extending the Kibana to support custom charts +kibana mixed dashboard.
(2) more powerful alarms. Refer to or combine Watcher,zabbix to achieve a more powerful and easier to use alarm configuration.

Build distributed log system with open source architecture Elk