Centralized log system ELK protocol stack detailed

Reprinted from http://www.ibm.com/developerworks/cn/opensource/os-cn-elk/

Company these days on a set, look at the article of the Great God under the study:), the following is reproduced content:

In our daily life, we often need to look back at some of the things that happened before, or, when there are some problems, we can find out the reasons from some places and look for the traces of the problems. There is no need to avoid the use of text, images and other forms of records. In computer terms, it is log, or log.

Log, which is an important part of any system. In a computer system, more so. However, because the computer systems are more complex now, many systems are not in one place, even cross-border, even in a local system, there are different sources, such as operating systems, application services, business logic and so on. They are constantly producing all kinds of log data. With incomplete statistics, we have about 2EB (1018) of data per day around the world.

In the face of such a huge amount of data, but also distributed in different places, if we need to find some important information, it is still using the traditional method, to log on to a machine to view. It seems that the traditional tools and methods have become very clumsy and inefficient. As a result, some smart people put forward a centralized approach to integrating data from different sources into one place.

A complete centralized log system is inseparable from the following key features. Collection-capable of capturing log data from multiple sources-can reliably transfer logs to a central system store-how to store log analysis-can support UI analysis warnings-can provide error reporting, monitoring mechanisms

Back to top products on the market

Based on the above ideas, many products or solutions have emerged. For example, simple rsyslog,syslog-ng, commercial Splunk, open-source Scribe,apache Chukwa,linkedin Kafak,cloudera Fluentd,elk of the FaceBook company, etc.

In the above products, Splunk is a very good product, but it is a commercial product, expensive, so many people deterred.

Until the advent of ELK, so that we have a choice. Relative to several other open source software, this article focuses on the introduction of ELK.

Back to top ELK protocol stack introduction and architecture

ELK is not a software, but a complete set of solutions, is the first acronym for three software products, Elasticsearch,logstash and Kibana. These three software are open-source software, usually with the use, but also attributed to elastic.co company name, it is referred to as ELK protocol stack, see figure 1. Figure 1.ELK Protocol stack Elasticsearch

Elasticsearch is a real-time distributed search and analysis engine that can be used for full-text search, structured search, and analytics. It is a search engine based on the full text search engine Apache Lucene, written in the Java language. Currently, the latest version is 2.1.0.

Key features real-time analysis of distributed real-time file storage, and each of the fields are indexed document-oriented, all of the objects are document high availability, easy to expand, support clusters (Cluster), sharding and Replication (shards and replicas). See Figure 2 and Figure 3 interface friendly, support JSON Figure 2. Cluster diagram 3. sharding and Replication Logstash

Logstash is a data collection engine with real-time channel capabilities. Written using the JRuby language. The author is the world-renowned operations engineer Jordancise (Jordansissel). The latest version is now 2.1.1.

Key features almost any data access can be combined with a variety of external applications to support elastic scaling

It consists of three main parts, see figure 4:shipper-Send log data broker-collect data, default built-in Redis indexer-data write Figure 4.Logstash Basic composition Kibana

Kibana is a WEB platform based on the Apache Open source protocol that is written in JavaScript and provides analysis and visualization for Elasticsearch. It can find, interact with data in the Elasticsearch index, and generate a table diagram of various dimensions. Currently the latest version is 4.3, referred to as Kibana 4. ELK protocol Stack Architecture

The complete ELK protocol stack architecture is shown in Figure 5. The basic process is that Shipper is responsible for collecting data from a variety of data sources and then sending it to Broker,indexer to write the data that is stored in the Broker elasticsearch,elasticsearch the data to be indexed, and then by Kibana Perform various analyses and display them in a graphical form. Figure 5.ELK Protocol stack Architecture

ELK three software is used in conjunction with each other, perfect convergence, efficient to meet the application of many occasions, and is adopted by many users, such as Reuters, Facebook, StackOverflow and so on.

ELK Installation and configuration of the top- back Page

In this section, I'll describe how to install the Configuration ELK protocol stack.

Selected experimental platform for Ubuntu 14.04 Centos 7.1

The software used is the following Elasticsearch use of the 2.1.0 Logstash use is 2.1.1 Kibana use is 4.3.0 In addition, also need to use the software, Nginx,logstash-forwarder and JDK.

The architecture of the experimental system is shown in Figure 6. Figure 6: Experimental system architecture

It is worth noting that in our experiment, the use of Nginx reverse proxy, so that users can access from the external to the Kibana, and it itself has the role of load balancing, but also to improve performance.

In particular, mention is made of logstash-forwarder, a log Transfer tool written using the Go language. Since Logstash is run in Java, we do not use Logstash to transfer data over the client system in order to improve the efficiency of the network transmission. This software is the original lumberjack, in the future will slowly be elastic.co company's other product Beat absorption merger into the filebeat, this beyond the scope of this article, it is not discussed in detail here. The specific installation process is as follows Step 1, install JDK Step 2, install Elasticsearch Step 3, install Kibana step 4, install Nginx Step 5, install Logstash Step 6, configure Logstash step 7, Ann Install Logstash-forwarder Step 8, Final verify pre-installation two 64-bit virtual machines, operating system is Ubuntu 14.04,2 cpu,4g memory, 30G HDD Two 64-bit virtual machines, operating system is CentOS 7 . cpu,4g memory, 30G HDD Create user Elk and group Elk, all of the following installations are operated by this user and are granted sudo permissions if it is CentOS, you also need to configure the official YUM source to access the CentOS package

Note: All of the following operations are done on two platforms. Step 1, install the JDK

Elasticsearch requires at least Java 7. Oracle JDK 1.8 or OpenJDK 1.8 is generally recommended. We use OpenJDK 1.8 here.

Ubuntu 14.04

Join the Java Software source (Repository)

$ sudo add-apt-repository Ppa:openjdk-r/ppa

Update the system and install the JDK

$ sudo apt-get update 
$ sudo apt-get install OPENJDK-8-JDK

Validating Java

$ java-version
openjdk version "1.8.0_45-internal"
openjdk Runtime Environment (build 1.8.0_45-internal-b14) C2/>OPENJDK 64-bit Server VM (build 25.45-b02, Mixed mode)

CentOS 7.1

Configuring the YUM Source

$ cd/etc/yum.repos.d
$ sudo vi centos.repo

Add the following content

[Base]
name=centos-$releasever-base
mirrorlist=http://mirrorlist.centos.org/?release= $releasever &arch=$ Basearch&repo=os&infra= $infra
#baseurl =http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/rpm-gpg-key-centos-7
#released updates 
[updates]
name=centos-$releasever-updates
mirrorlist=http://mirrorlist.centos.org/?release= $releasever &arch = $basearch &repo=updates&infra= $infra
#baseurl =http://mirror.centos.org/centos/$releasever/updates/ $basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/rpm-gpg-key-centos-7

Installing the JDK

$ sudo yum install JAVA-1.8.0-OPENJDK

Validating Java

$ java-version
openjdk version "1.8.0_65"
openjdk Runtime Environment (build 1.8.0_65-b17)
openjdk 64-bit Server VM (build 25.65-b01, Mixed mode)

Step 2, install Elasticsearch

Ubuntu 14.04

Download Elasticsearch software

$ wget Https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch /2.1.0/elasticsearch-2.1.0.tar.gz

Extract

$ tar xzvf elasticsearch-2.1.0.tar.gz

The file directory structure is as follows: