Fluentd is an open source collection event and log system that currently offers 150 + extensions that let you store big data for log searches, data analysis and storage.
Official address http://fluentd.org/plugin address http://fluentd.org/plugin/
Kibana is a Web UI tool that provides log analysis for ElasticSearch, and it can be used to efficiently search, visualize, analyze, and perform various operations on logs. Official Address http://www.elasticsearch.org/overview/kibana/
Elasticsearch is an open source (Apache2 protocol), distributed, restful, search engine built on Apache Lucene.
Official address http://www.elasticsearch.org/overview/Chinese address http://es-cn.medcl.net/
The specific workflow is to use FLUENTD to monitor and filter the system logs of the Hadoop cluster, send the filtered log content to the full-text search service Elasticsearch, and then use the Elasticsearch kibana to customize the Search Web page presentation.
The deployment methods and procedures are explained below. The following installation steps are in the CentOS 5 64-bit test pass
One, Elasticsearch installation deployment
Elasticsearch official offers several installation packages for Windows ZIP compression package for Unix/linux tar.gz compression package, RPM package for CentOS system and Ubuntu Deb package. You can choose your own installation to use.
Because the Elasticsearch requires a Java environment to run, the JDK needs to be installed first, and the installation steps are omitted.
To install a deployment using the. tar.gz compression pack, download the ZIP package first
# wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.tar.gz
# tar ZXVF elasticsearch-0.90.5.tar.gz
If it is a stand-alone deployment
# CD elasticsearch-0.90.5
# elasticsearch-0.90.5/bin/elasticsearch-f
You can start the search service and see if Port 9200 is turned on if the instructions are turned on correctly.
If you are deploying a cluster, you need to configure
For example, in 192.168.0.1 192.168.0.2 two server deployment, both servers are installed JDK, download Elasticsearch unzip, and then edit the configuration file
192.168.0.1 Server Edit File
VI elasticsearch-0.90.5/config/elasticsearch.yml
Delete Cluster.name previous comment, modify cluster name
Cluster.name:es_cluster
Delete the pre-node.name comment, modify the name of the node, and do not modify it, the system will generate the node name immediately after startup.
Node.name: "Elastic_inst1"
Node.master:true set the node as the primary node
192.168.0.2 Editing files
VI elasticsearch-0.90.5/config/elasticsearch.yml
Delete Cluster.name previous comment, modify cluster name
Cluster.name:es_cluster
Delete the pre-node.name comment, modify the name of the node, and do not modify it, the system will generate the node name immediately after startup.
Node.name: "Elastic_inst2"
Node.master:false set the node as the primary node
After you start the service for two servers separately, you'll see in the 192.168.0.2 log
[Elastic_inst2] detected_master [elastic_inst1] log information. Indicates that the cluster connection was successful.
II. Installation and Deployment Fluentd
Install FLUENTD in a Hadoop cluster node that requires monitoring analysis, and the installation steps are simple
Curl-l http://toolbelt.treasure-data.com/sh/install-redhat.sh | Sh
After the installation is complete, edit the configuration file
# vim/etc/td-agent/td-agent.conf
<source>
Type Tail # # # # Tail way to collect logs
path/var/log/hadoop/mapred/ Hadoop-mapred-tasktracker-node-128-70.log # # # Hadoop log path
Pos_file/var/log/td-agent/task-access.log.pos
tag task.mapred
format/^ (? <message>.+ (warn| Error). +) $/ # # # # # # # Collect error or warn log.
</source>
<match task.**>
host 192.168.0.1 ##### elasticsearch server address
Type Elasticsearch
logstash_format true
flush_interval 5s
include_tag_key true
tag_key mapred
</match>
Start the FLUENTD service
# service Td-agent Start
III. installation and Deployment Kibana 3
Kibana 3 is a Web UI front-end tool developed using HTML and JavaScript.
Download wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip
Decompression Unzip Kibana-latest.zip
Install Apache yum-y Install httpd
Cp-r kibana-latest/var/www/html
Because I installed KIBANA3 in the same server as Elasticsearch, so I don't have to modify the configuration file
Start Apache service httpd start
Open browser Http://ip/kibana to see Kibana interface
Initial use of Kibana requires you to define your own module