A tentative study on "elk"-centos6.7 under construction

I. Architecture at a glance:

The so-called elk, respectively refers to the Elasticsearch, Logstash, Kibana; Official website: https://www.elastic.co/products;

Three roles clear: Elasticsearch is responsible for indexing (create INDEX, search data), equivalent to the database; Logstash is responsible for uploading the log, in the process of uploading the log, the log can be structured, the regular log into the Elasticsearch Kibana is responsible for visualizing the data in the Elasticsearch. In the actual application, if the real-time upload log volume is too large, the index speed can not keep up with the upload speed, you need to Logstash to elasticsearch in the middle of a cache layer, usually we use Redis as a pure cache queue.

Deployment situation: Four machines A B C D

A Deploying Redis + Logstash +elasticsearch +kibana

B C D Deployment Elasticsearch

A B C D composition Elasticsearch Cluster

Need to collect logs of four machines N1, N2, N3,N4 for four load-bearing nginx machine, respectively, on the 4 machines deployed Logstash, filtered regular nginx log uploaded to a machine redis. Then the Logstah from the a machine gets the data output from the Redis to the Elasticsearch cluster, and the Kibana shows the data in the Elasticsearch cluster.

two. Version selection:

Java Environment: JDK1.8.0_65

Log collection, upload: Logstash1.5.4 download Address: https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz

Elasticsearch cluster: Elasticsearch1.7.3 Download Address: https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.3.tar.gz

Log Show: kibana4.1.2 Download Address: https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

Queue cache: redis3.0.4 Download address: http://download.redis.io/releases/redis-3.0.4.tar.gz

three. Concrete Construction

Machine Environment: Centos6.7

1. Install the Java environment:

The latest Java environment is recommended, and the JDK version of Elasticsearch cluster is consistent, here we choose jdk1.8.0_65.

In the native environment, the default is actually installed OPENJDK, and the environment of each machine is disorderly seven or eight slots, so before installing the Java environment I put the default OPENJDK environment to kill.

1.1 Uninstalling OpenJDK

#java-version found the OPENJDK environment.

#rpm-qa|grep java or #Rpm-qa | grep jdk Find the Java possible output of RPM installation as follows:

Tzdata-java-2015g-2.el6.noarch

Java-1.6.0-openjdk-1.6.0.37-1.13.9.4.el6_7.x86_64 java-1.7.0-openjdk-1.7.0.91-2.6.2.2.el6_7.x86_64

Uninstall the above Java in turn:

#yum-y Remove java-1.7.0-openjdk-1.7.0.91-2.6.2.2.el6_7.x86_64 #yum-y-Remove java-1.6.0-openjdk-1.6.0.37-1.13.9.4.e l6_7.x86_64 #yum-y-Remove Tzdata-java-2015g-2.el6.noarch

1.2 Installing JDK1.8.0_65

To http://www.oracle.com/technetwork/java/javase/downloads/find the Java version you need wget down

Unzip: #tar zxvf jdk-8u65-linux-x64.tar.gz-c/usr/local/java

Setting environment variables

#vi/etc/profile at the end of the document

Export java_home=/usr/local/java/jdk1.8.6_65

Export JRE_HOME=${JAVA_HOME}/JRE

Export classpath=.: ${java_home}/lib: ${jre_home}/lib

Export Path=${java_home}/bin: $PATH

Save, immediate effect environment variable

#source/etc/profile

Verify that the environment is OK:

#java-version appears jdk1.8.0_65 description Java environment OK

2.redis Installation

#wget http://download.redis.io/releases/redis-3.0.4.tar.gz

Unzip: TAR-XZVF redis-3.0.4.tar.gz to the directory you need

#cd redis-3.0.4

#make

#make Install

In the make process can be error, general installation prompts Yum installation can be referred to as follows:

#yum install vim unix2dos wget Curl Curl-devel expect Expect-devel

#yum Install gcc glibc glib2 libgcc glibc-devel glib2-devel

#yum Install M4 automake autoconf cmake CPP

#yum Install gcc-c++ libstdc++ libstdc++-devel libstdc++-docs

If the following error occurs:

Zmalloc.h:55:2: Error: #error ' newer version of Jemalloc required '

Can be performed: Make MALLOC=LIBC

Modify Daemonize Yes

To run Redis:

#./src/redis-server redis.conf

#ps-ef | grep redis View Process

or #redis -3.0.4/src/redis-cli .

>set a B

>get A

#b

Redis OK.

3. Installing the Elasticsearch cluster

Elasticsearch cluster installation configuration is very simple, as long as in the same LAN, the cluster name is consistent, ES will automatically discover other machines, and join the cluster, but in the actual environment, due to network environment problems, often occur node drop, or long time to join the cluster situation, We need to change the configuration, optimize the relevant parameters, has reached the actual requirements, this article is closely based on the default configuration for example, subsequent optimizations will be shared in another article.

# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.3.tar.gz

# Unzip TAR-XZVF elasticsearch-1.7.3-c to the directory you need

Download the installation separately on a B C D four Machine

# elasticsearch-1.7.3/bin/elasticsearch-d Boot

Access to the 9200 ports of each machine, return 200 instructions OK, if return 503, you can wait a little longer, in this case, mostly because the node is trying to join the cluster, I have encountered a wait for 10 minutes, the state is 503 into 200. If the status is not correct, do not-D start, direct start, will play a startup log to see if the error.

Elasticsearch related plug-ins have a lot, I mainly installed two: Elasticsearch-head and Kopf, head of the interface slightly rough point, Kopf interface style more geek, related other plug-ins can refer to:/http Www.cnblogs.com/huangfox/p/3541300.html

4. Installing Logstash

# wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz

#解压tar–xvzf logstash-1.5.4.tar.gz to your directory

5. Installing Kibana4

# wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

#解压 TAR-XZVF kibana-4.1.2 to your directory

Now that the KIBANA4 has its own web capability, the default port is 5601 and can be started directly at Kibana/bin/kibana Kibana4

Four. Related configurations

1. Configure Logstash

Configuration of the Logstash on the N1~N4 machine:

1.1 mkdir two folders under Logstash: Conf and logs store Logstash configuration files and log files respectively

1.2#cd conf

#vi nginx_access.conf

The contents are as follows:

Input {file {type = "XXX" path = "/xxx/nginx/access.log"} }
Output {redis {
Host = "xxx.xxx.xx.xx" port = xx data_type = ' list ' key = ' Logstash:redis '} Description: Read log file from text, output to a Redis on the machine. 1.3# CD logs/ New Logstash.log 1.4#nohup./logstash-f./conf/nginx_access.conf-l./logs/logstash.log &Run, can be run before the actual./logstash-f./conf/nginx_access.conf-tCheck if the configuration file is correct
Logstash configuration on  a machine:  input {   redis {     host = "xxx.xx.xxx.xx"     &N Bsp;port = 6379      data_type = ' list '      key = ' Logstash:redis '   &nbsp ;  type = ' redis-input '    }  }  filter { }  output {   elasticsearch {      host = "xxx.xx.xxx.xx"      port = 9200      CODEC = "JSON"      protocol = "http"  } Description: Get data from Redis on a machine, output to Elasticsearch cluster