Construction of log analysis platform Elk in Big Data era

A, first of all say elk is what, elk is Elasticsearch, Logstash and Kiabana three open source tools. Logstash is the data source, Elasticsearch is the analysis of the data, Kiabana is to display the data

B, start doing

1, install Logstash dependent package JDK

     wget http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
　　  If there is no wget can yum-y install wget installed wget, some machines are not installed

The JDK is easy to install, just unzip the downloaded package to the appropriate directory.

　　  Mkdir/usr/local/java   tar-zxf jdk-8u45-linux-x64.tar.gz-c/usr/local/java/

Set the environment variables for the JDK as follows:

 tail-3 ~/.bash_profile export java_ Home=/usr/local/java/jdk1. 8.0_45 export path= $PATH :  $JAVA _home/bin exportclasspath=. : $JAVA _home/lib/tools. Jar: $JAVA _home/lib/dt. $CLASSPATH 
 Then you can try Java or javac the JDK is installed successfully 
 (1) Install Logstash 
 Download 
 wget https://do Wnload.elastic.co/logstash/logstash/logstash- 1.5.2.tar.gz     
 Extract 
                

TAR-ZXF logstash-1.5. 2.tar.gz-c/usr/local/
Start,-e is the description to start as a parameter
　　/usr/local/logstash-1.5.2/bin/logstash-e ' input {stdin {}} ' output {stdout {}} '

2, install elastic search
Download, unzip to complete the installation

TAR-ZXF elasticsearch-1.6.0.tar.gz-c/usr/local/
Next we create a test file for testing Logstash using Elasticsearch as the Logstash back end in the Logstash installation directory logstash-es-simple.conf,
The file defines stdout and Elasticsearch as output, so that the "multiple outputs" are guaranteed to be displayed on the screen and also output to Elastisearch.
 　　 Cat logstash-es-simple.conf
Input {stdin {}}} output {elasticsearch {hosts = "localhost"} stdout {codec=> Rubydebug}}
Be sure to pay attention to the above is the hosts and not the host domestic are dead host and then I got a half a day after Google only found a little s
Start Logstash
　　/usr/local/logstash-1.5.2/bin/logstash agent-f logstash-es-simple.conf  
You will then see the system print
Logstash Startup completed
Hello Logstash {"Message" = "Hello logstash", "@version" and " 1", "@timestamp" and "=" 2015-07-15t18 : 12:00.450z ", " host "= " noc.vfast.com "}       
You can use the Curl command to see if ES has received dataCurl ' Http://localhost:9200/_search?pretty '
3, install Kibana
Unzip to the corresponding folder after downloading
　　TAR-ZXF kibana-4.1.1-linux-x64.tar.gz-c/usr/local/
Start

　　/usr/local/kibana-4.1.1-linux-x64/bin/kibana
With http://kibanaServerIP:5601 access to Kibana, after logging in, first configure an index, by default, Kibana data is pointed to Elasticsearch, uses the default logstash-* index name, and is based on the time , click "Create" to
And then it shows that the basic has been deployed.
However, if you need to display some system information or collect the log information of the system, you need a lot of configuration
For your funeral, please continue to follow

Construction of log analysis platform Elk in Big Data era