Fluentd combined with Kibana, elasticsearch real-time search to analyze Hadoop cluster logs

Fluentd is an open source collection event and log system that currently offers 150 + extensions that let you store big data for log searches, data analysis and storage.

Official address http://fluentd.org/plugin address http://fluentd.org/plugin/

Kibana is a Web UI tool that provides log analysis for ElasticSearch, and it can be used to efficiently search, visualize, analyze, and perform various operations on logs. Official Address http://www.elasticsearch.org/overview/kibana/

Elasticsearch is an open source (Apache2 protocol), distributed, restful, search engine built on Apache Lucene.

Official address http://www.elasticsearch.org/overview/Chinese address http://es-cn.medcl.net/

The specific workflow is to use FLUENTD to monitor and filter the system logs of the Hadoop cluster, send the filtered log content to the full-text search service Elasticsearch, and then use the Elasticsearch kibana to customize the Search Web page presentation.

The deployment methods and procedures are explained below. The following installation steps are in the CentOS 5 64-bit test pass

One, Elasticsearch installation deployment

Elasticsearch official offers several installation packages for Windows ZIP compression package for Unix/linux tar.gz compression package, RPM package for CentOS system and Ubuntu Deb package. You can choose your own installation to use.

Because the Elasticsearch requires a Java environment to run, the JDK needs to be installed first, and the installation steps are omitted.

To install a deployment using the. tar.gz compression pack, download the ZIP package first

# wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.tar.gz

# tar ZXVF elasticsearch-0.90.5.tar.gz

If it is a stand-alone deployment

# CD elasticsearch-0.90.5

# elasticsearch-0.90.5/bin/elasticsearch-f

You can start the search service and see if Port 9200 is turned on if the instructions are turned on correctly.

If you are deploying a cluster, you need to configure

For example, in 192.168.0.1 192.168.0.2 two server deployment, both servers are installed JDK, download Elasticsearch unzip, and then edit the configuration file

192.168.0.1 Server Edit File

VI elasticsearch-0.90.5/config/elasticsearch.yml

Delete Cluster.name previous comment, modify cluster name

Cluster.name:es_cluster

Delete the pre-node.name comment, modify the name of the node, and do not modify it, the system will generate the node name immediately after startup.

Node.name: "Elastic_inst1"

Node.master:true set the node as the primary node

192.168.0.2 Editing files

VI elasticsearch-0.90.5/config/elasticsearch.yml

Delete Cluster.name previous comment, modify cluster name

Cluster.name:es_cluster

Delete the pre-node.name comment, modify the name of the node, and do not modify it, the system will generate the node name immediately after startup.

Node.name: "Elastic_inst2"

Node.master:false set the node as the primary node

After you start the service for two servers separately, you'll see in the 192.168.0.2 log

[Elastic_inst2] detected_master [elastic_inst1] log information. Indicates that the cluster connection was successful.

II. Installation and Deployment Fluentd

Install FLUENTD in a Hadoop cluster node that requires monitoring analysis, and the installation steps are simple

Curl-l http://toolbelt.treasure-data.com/sh/install-redhat.sh | Sh

After the installation is complete, edit the configuration file

# vim/etc/td-agent/td-agent.conf

<source>
  Type Tail  # # # # Tail way to collect logs
  path/var/log/hadoop/mapred/ Hadoop-mapred-tasktracker-node-128-70.log   # # # Hadoop log path
  Pos_file/var/log/td-agent/task-access.log.pos
  tag task.mapred
  format/^ (? <message>.+ (warn| Error). +) $/   # # # # # # # Collect error or warn log.
</source>

<match task.**>
  host 192.168.0.1  #####  elasticsearch server address
  Type Elasticsearch
  logstash_format true
  flush_interval 5s
  include_tag_key true
  tag_key mapred
</match>

Start the FLUENTD service

# service Td-agent Start

III. installation and Deployment Kibana 3

Kibana 3 is a Web UI front-end tool developed using HTML and JavaScript.

Download wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip

Decompression Unzip Kibana-latest.zip

Install Apache yum-y Install httpd

Cp-r kibana-latest/var/www/html

Because I installed KIBANA3 in the same server as Elasticsearch, so I don't have to modify the configuration file

Start Apache service httpd start

Open browser Http://ip/kibana to see Kibana interface

Initial use of Kibana requires you to define your own module